Repository navigation
Identity-only SSSOM refresh corrupts writer provenance and YAML description metadata #1237
Description
Identity-only SSSOM refresh mismatches tool provenance and corrupts YAML descriptions
Related: #1236 (bounded Potato material-scope hold). This is a preexisting
metadata defect exposed when preparing its separate identity-policy candidate;
it is not evidence that the underlying mapping rows were generated incorrectly.
Reproduction 1: inconsistent tool/version pair
scripts/consolidate_chemical_mappings.py:3063 replaces
mapping_tool_version with its own script SHA-256, but copies the predecessor's
mapping_tool unchanged. A valid one-row water SSSOM with the actual historical
reconstruction metadata produces:
| Field | Before | After identity-only refresh |
|---|---|---|
| mapping_tool | kg-microbe/scripts/mim_conservative_refresh.py | kg-microbe/scripts/mim_conservative_refresh.py |
| mapping_tool_version | sha256:58ec62b9b01a28f4f3b6b47ff319a11396617d4a07ada409acc2742bcb77ae6a | sha256:0d651d97ed0881c335f361f49486be2d9fc8902878487e3588569b4e4db98912 |
The latter hash identifies consolidate_chemical_mappings.py, not the script
named by mapping_tool. This reproducibly publishes an internally inconsistent
provenance pair, even though the scientific row is unchanged. The actual local
SSSOM validator passes both outputs; a second cycle is byte-identical and
therefore does not detect or repair the mismatch.
Reproduction 2: a literal quote is added to the description
The same actual output independently exposes another metadata corruption at
scripts/consolidate_chemical_mappings.py:3067: the code assumes the description
is a double-quoted, one-line YAML scalar, removes an optional closing quote and
unconditionally adds a quote after its fingerprint. The valid plain YAML scalar
in the tiny fixture changes semantically from:
Tiny independent provenance reproduction.
to this parsed string (the final double quote is part of the value):
Tiny independent provenance reproduction. Ingredient identity policy sha256:61786a46effa48de9901f77713b172db16a7d6797f35711d5c15c01b0e0ea926."
The second cycle preserves the corrupted value; deterministic output is not
proof that the source description was preserved. The current immutable unified
baseline also uses a plain, not quoted/folded scalar:
mapping_set_description: Conservative MIM candidate; reviewed manifest sha256:9bb29d5605d93dea351be9624d99c5d8ada57d4b9831b22764957b784bd685af.Therefore the real identity-only candidate route has the same affected input
shape. Its baseline bytes remain unchanged. Single-quoted and multiline YAML
descriptions are adjacent unsupported forms that need regression coverage;
this report does not claim they were present in the current baseline.
Saved reproduction (tiny owned files only, no production reads/writes):
reproduce_identity_refresh_provenance.pyprovenance-repro-01/result.jsonprovenance-repro-01/tiny-baseline.tsvprovenance-repro-01/first.tsv.gzandsecond.tsv.gz
Both cycles report one row read, zero removals, and zero relabelings. Their
identical compressed SHA-256 is
1f9b9efe4de192d70cf1200a5defb6bd069b4e9114c5c24c5e5c8729c08f49c2.
The baseline SHA-256 is
51d29c1258fe37dfc5a6ebf2f5e6f404c28a0e8ebd240c3e03150f28bcfa59b1.
Minimal correction and acceptance
Parse and edit the semantics of only the three affected top-level YAML
fields (mapping_tool, mapping_tool_version, and
mapping_set_description), not line suffixes that assume a particular scalar
style. Preserve unrelated metadata, all assertion dates, the literal TSV
header and unrelated serialized data rows. Do not mutate the whole metadata
mapping as a side effect of reformatting it. Missing tool/version fields must
gain the truthful pair; malformed or ambiguous metadata must fail safely.
When this entry point performs identity-only refresh, set mapping_tool to
kg-microbe/scripts/consolidate_chemical_mappings.py alongside the actual script
fingerprint in mapping_tool_version. Do not misidentify the current writer in
order to retain predecessor provenance; bind the immutable baseline identity
and original metadata in the separate candidate/delta report instead.
Add hermetic tests using a different predecessor tool/version that require:
- The refreshed tool names the actual identity-refresh script and its version
is that script's content hash. - Plain, single-quoted, double-quoted, folded (
>), and literal (|)
descriptions retain their original parsed meaning before the one appended
current policy fingerprint, including intended quotes and newlines. - A missing tool/version pair is inserted accurately; unrelated metadata,
assertion dates, and complete serialized rows remain unchanged except for
existing documented policy effects. - Repeating the refresh is byte-identical and there is exactly one current
policy fingerprint, without a stray quote or lost description content.
Regenerate only a separate reviewed candidate after that fix. The current
candidate is diagnostic, not promotable. Keep supported MIM rows, release pin,
and historical fixtures unchanged. Update the native artifact-pair expectations
only after independently verifying the replacement candidate and exact delta.
Activity
realmarcin commentedon Sep 29, 2026
Resolved in PR #1239,
merge commit
c7dae95a046801e4dac171889df4af7204022c6e. The merged tree exactlymatches the independently reviewed/tested head
03dd4bc918566ceffd43692ef1ed865419a441aa.Full pytest and tox's pytest each passed 4,952 tests (15 skipped); all six tox
environments, lock/config checks and Python 3.10/3.11/3.12 CI passed. The source
change also passed the actual isolated MediaDive replay and full-row comparison;
the later fixture repairs were independently reviewed. Detailed terminal gates:
PR acceptance comment.
The topic branch is deleted locally and remotely, with its detached worktree
and original failed/successful review evidence retained. This closes the bounded
implementation/review finding, not umbrella #650/#286 production acceptance.
The final combined correction and fresh all-source/candidate-merge reviews remain
separate. The existing merged archive and user statistics are unchanged.