MicrobeDecoder curation gap: 5,224 distinct labels unmapped #650

Activity

1. realmarcin

   added a commit that references this issue on Aug 3, 2026

   feat(microbedecoder): emit unmapped_labels.tsv curation queue (#652)

   ...

   10bd210
2. realmarcin commented on Aug 3, 2026

   

   realmarcin

   on Aug 3, 2026

   CollaboratorAuthor

   More actions

   Resolved-partially by #652: MicrobeDecoder now emits a per-run data/transformed/microbedecoder/unmapped_labels.tsv sorted by occurrence descending. Live run produces 5,224 rows — the curation priority queue for this issue. Top pathway placeholders (chemoheterotrophy 10K×, fermentation 4.8K×, phototrophy 5.4K×) are the highest-leverage METPO alias additions.
3. realmarcin commented on Aug 3, 2026

   

   realmarcin

   on Aug 3, 2026

   CollaboratorAuthor

   More actions

   PR #653 wires up the full postprocess pipeline for the unmapped-labels report:

   • scripts/dump_unmapped_microbedecoder_labels.py reduces the per-run report into mappings/microbedecoder_unmapped_labels_to_curate.tsv (tracked). Split by facet with --prefix pathway|compound|trait, threshold with --min-occurrences.
   • scripts/generate_coverage_report.py -s microbedecoder gives mapping-rate stats.
   • Full runbook at docs/MICROBEDECODER_TRANSFORM_RUNBOOK.md documents the per-facet curator workflow.
   • kg-postprocess-report skill now knows about the outputs.

   Curators can start on the top 451 rows (occurrences ≥ 100) at mappings/microbedecoder_unmapped_labels_to_curate.tsv.
4. realmarcin commented on Sep 28, 2026

   

   realmarcin

   on Sep 28, 2026

   Last edited by realmarcin

   CollaboratorAuthor

   More actions

   Reviewed implementation and current scientific scope

   Merged #1217 (shared GO checkpoint fix) and #1218 (source-specific curation),
   after independent adversarial review and green Python 3.10/3.11/3.12 CI on each
   exact PR head. Final master is bc7c6e73a13e90ddadd201d938530c59288aeeea, with
   the same Git tree as the independently reviewed and tested integrated head.
   Both topic branches are deleted locally and remotely; commits and review
   evidence remain preserved. Review issues #1212–#1216 are closed.

   The integrated head passed full pytest (4,356 passed, 10 skipped) and all
   six tox environments (including the same full test count), plus lock/config
   checks. The bounded native/source replay, combined finalization/reuse diagnostic,
   and independent Git-tree review also passed. Post-merge
   master QC
   is green on all three Python versions, and
   documentation deployment
   passed on the same final commit. Production raw data, MicrobeDecoder TSVs,
   merged archive and pinned unified export retain their original checksums.
   This is not a production rebuild or release acceptance.

   The curation implementation is evidence- and source-field-scoped:

   • 17 reviewed METPO/GO process rules cover 7,152 assertions in the saved cohort, retaining citations and FAPROTAX prediction provenance.
   • 17 group/unspecified labels remain source Attributes rather than unsupported biological processes. Four missing substrate observations remain reporting attributes.
   • One complete-record-pinned tartate observation is corrected to tartrate; no global chemical alias, consolidated mapping or MIM pin is changed.
   • 22 reviewed textual phenotype rules produce 33,144 report-only observations. They do not silently create organism phenotype assertions against incompatible pinned endpoint categories.
   • The inventory now accounts for every one of 521,217 source edges, separating crosswalks, processes, materials and source attributes.

   Independent adversarial review produced #1212–#1216: GO final-byte checkpointing, blank scientific endpoints, fallback relation validation, explicit chemical-category contradictions, and native METPO process categorization. Each was reproduced, fixed, and independently re-reviewed.

   The bounded post-review rebuild produces 20,364 MicrobeDecoder nodes and 521,217 edges, byte-identical to the earlier accepted continuation candidate. The native METPO producer changes exactly 18 categories to BiologicalProcess using asserted native ancestry. All 44 protected files are unchanged; production data and merged artifacts have not been replaced.

   Remaining work (keep #650 open)

   • Definition-level review of 61 local process IDs / 41,388 assertions; label similarity alone is insufficient.
   • Review 10 queued material IDs / 225 assertions and 3 registry-local material IDs / 15 assertions. Ambiguous mixtures/media and the two record-scoped sugar observations must retain their evidence boundaries.
   • Establish source coding and observation context for numeric/signed assay results and other reported attributes. The 6,592 source-attribute IDs are not all missing chemicals or untransformed observations.
   • Resolve phenotype graph-modeling compatibility before promoting report-only grounding into graph assertions.
   • Rebuild native ontology outputs before dependent producers, enforce source freshness, then merge and perform graph-level acceptance before release.

   The historical 5,224-label total is not a verified current backlog count or acceptance target. This issue remains the durable follow-up for these scientific interpretation tasks; code merge alone is not a release claim.
5. realmarcin

   added a commit that references this issue on Sep 29, 2026

   Curate MicrobeDecoder traits, process meanings, and material contexts (…

   ...

   66f3a03
6. realmarcin commented on Sep 29, 2026

   

   realmarcin

   on Sep 29, 2026

   CollaboratorAuthor

   More actions

   The bounded curation and review work in #1220 is now merged as
   66f3a03c7ecf12d982ef14bacc1157b93e59bc55; its local and remote topic branches
   are deleted. Review issues #1219, #1221, #1222 and #1223 are closed.

   Final checks on the exact reviewed head passed: poetry run pytest and the
   pytest stage of poetry run tox each report 4,671 passed / 22 skipped;
   all other tox stages, lock check and merge-config generation check passed.
   Python 3.10/3.11/3.12 CI is green. The merge tree equals the tested tree.

   This adds the reviewed source Attribute types, all 61 remaining process-label
   dispositions, and the complete bounded material-context catalogue. The actual
   isolated replay verified intentional changes and byte-identical preservation
   of 479,829 unaffected edges; it is not a production merge acceptance.

   Keep #650 open for production rebuild/acceptance. Source-path review found
   preexisting crosswalk contradictions, tracked separately in #1224. The user
   approved evidence-preserving quarantine of authority-contradicted links only.
   Independent complete-cohort review supports an initial finite set of 10,066
   specific NCBI/GOLD claims across 5,033 records; remaining candidates are not
   automatically rejected. The separate containment implementation is underway.
   Corroborated assembly links and other source observations are preserved.

   No production graph or mapping pin was changed, and this is not a release-ready
   claim. Local evidence and final gate receipts are under
   data/curation-goal-20260928.LFUKxk/; quarantine work is under
   data/issue1224-quarantine-20260929.xjuS9H/.
7. realmarcin commented on Sep 29, 2026

   

   realmarcin

   on Sep 29, 2026

   CollaboratorAuthor

   More actions

   The approved finite crosswalk quarantine is now shipped in PR #1228, squash 6c8f360242fd483537c24af3cce2f5743e7b487f, following independent adversarial review and all-green Python 3.10/3.11/3.12 CI. The merged tree matches the tested candidate exactly; the branch was deleted locally/remotely while retaining its detached evidence worktree.

   Actual isolated producer/finalizer replay and full-row comparison establish exactly 10,066 reviewed NCBI/GOLD claims quarantined with complete original evidence retained; all 511,151 surviving edge rows and 17,855 retained node rows are unchanged. Strict inventory and modeling review passed. The source-only orphan result remains explicit: the complete configured-supplier scan leaves exactly the same 18,374 external IMG IDs as the prior review, not a new loss or a blanket clean merged-path claim.

   Local gates: 4,761 pytest passed / 56 skipped; tox, lock, generated-config checks passed; 34 additional targeted schema tests passed with the existing pinned local schema. Full evidence: data/issue1224-quarantine-20260929.xjuS9H/SHIPPED.md and linked receipts.

   This issue remains open for actual production rebuild/acceptance. The existing production archive and user statistics remain byte-identical and have not been replaced. The full 15-source rebuild and separate candidate merge are next, followed by source-to-merge preservation, CAS reconciliation, complete graph reviews and immutable quarantine evidence companion. A pre-launch helper safety review found #1232; it will be fixed and retested before execution. No release-readiness claim is made.
8. realmarcin

   closed this as completedin #1256on Sep 29, 2026
9. realmarcin

   reopened this on Sep 29, 2026
10. realmarcin commented on Sep 29, 2026

    

    realmarcin

    on Sep 29, 2026

    CollaboratorAuthor

    More actions

    Reopening this umbrella: PR #1256 landed its finite corrections, but the fresh
    all-source rebuild and candidate KG/model/path acceptance remain outstanding.
    GitHub interpreted a closing keyword inside a negated sentence in the PR body
    as an automatic closure directive. The body has been corrected to explicitly
    state that this umbrella remains OPEN. This was a tracking correction, not
    completed production acceptance.

    PR #1256 passed independent review, full local gate phases plus a verified
    format-only continuation, and exact-head CI on all three Python versions. The
    actual MediaDive comparison preserved all 225 audit rows. Follow-up #1259 repairs
    an inherited malformed KEGG annotation before the final rebuild. The current
    canonical merged archive has not been replaced or declared release-ready.
11. realmarcin commented on Sep 30, 2026

    

    realmarcin

    on Sep 30, 2026

    CollaboratorAuthor

    More actions

    The fresh all-15 transform batch at 1408e7099d039026d7611c240938d8e177753406 is complete, and its finite pre-merge source checks have passed. The separately guarded candidate-only merge is now running after successful freshness and production source-admission checks.

    Fresh-source results:

    • MediaDive: complete accepted source graph and audit handoff verified.
    • BacDive: two corrected substrate assertions and expected annotation changes verified; remaining content preserved.
    • MicrobeDecoder: all 511,151 source assertions inventoried and model-checked. All 240 reviewed material uses survive, including the two separate unresolved sugar records.
    • Quarantine: all 10,066 reviewed contradicted claims are absent; the complete 74,178 surviving crosswalk rows match the historical evidence minus those exact claims, including provenance, context and multiplicity. Corroborated assembly and BacDive controls survive.

    The source-only quarantine result (6d5a28f2cecbd3cb0049b01c29e39be4bc220d80dd1c21b1898f6ce2363c6170) and independent acceptance report (c3f045c1802711fc13a411743e9f66dd2b09f41c1ab1b113c13ca7e4e3e4fbb9) are under data/issue1224-quarantine-20260929.xjuS9H/fresh-source-acceptance-1408/. Original failed checks and their corrected, independently reviewed successors remain preserved; no source evidence was rewritten to satisfy them.

    Actual merge attempt: data/issue1224-quarantine-20260929.xjuS9H/production-rebuild/merge-attempt-9faxocfp/. This is a new candidate destination; the canonical archive and user-maintained statistics remain protected.

    Still pending: settled merge completion; complete archive framing/projection/metadata/CAS checks; native-evidence reconciliation; full KG/model/path reviews and triage; evidence companion retention. #650, #286, #1265, #1267 and #1268 remain open. No release or exact identity for unresolved materials is claimed.
12. realmarcin commented on Sep 30, 2026

    

    realmarcin

    on Sep 30, 2026

    CollaboratorAuthor

    More actions

    The fresh-production and candidate-acceptance gates listed in comments 5886356144 and 5903022047 are now complete. Closing #650 for the documented current-data interpretation, conservative representation, quarantine, and candidate acceptance scope—not as universal grounding of all historical labels or release approval.

    Production code is 1408e7099d039026d7611c240938d8e177753406. The all-15-source rebuild passed source admission and the isolated candidate merge completed: 3,354,500 nodes / 17,637,435 edges, archive SHA-256 158e7131025c7941ae98d70af5ca14d6efb69857ed019f7f61fa9b44279b2345. Its literal manifest commit remains 1408e7099-dirty, not a substituted clean-commit assertion.

    Actual acceptance:

    • All 511,151 MicrobeDecoder assertions and their multiplicities survive source-to-archive projection. Current accounting is 74,178 crosswalk, 415,184 source Attribute, 13,484 process, and 8,305 chemical/material assertions. The 7,030 inventory rows are not the old 5,224-label backlog count.
    • All 10,066 selected contradicted crosswalk claims across 5,033 records are quarantined with original evidence retained; corroborated assembly and BacDive controls survive.
    • All 240 reviewed material assertions across 235 records survive. The two generic Sugar observations remain separate record-scoped unresolved nodes. The 45 source-defined process nodes are reviewed source meanings, not asserted exact external identities. Qualified Attributes, generic spore observations, and ambiguous VP codes retain their conservative evidence boundaries.
    • The affected-source check preserves all 113,918 MediaDive assertions and 16 finalized BacDive EC-substrate assertions, with no missing, duplicate, or unexpected full assertions. Endpoint presence is verified; this is not a claim that every merged node metadata cell equals its source cell. Six current CAS endpoints and 110 assertions survive, without new chemical-identity approval.
    • Actual finite native preservation passed its preservation checks. Eleven additional broad ChemicalEntity categories received a separate source-authority review and independent reproduction; the original machine REVIEW_REQUIRED result remains unchanged.
    • Full KG/model/GO/path/strict reviews completed and were independently triaged. There are no missing merged endpoints, no self-loops in the eight tested predicates, and no subclass DFS back-edge cycles. These are scoped topology results, not universal path or utilization proofs.

    Strict KGX is still nonconformant. Its 76,611,293 callbacks include installed-profile missing-ID/type/prefix requirements; the count is not a count of distinct bad assertions. The exact MediaDive +290 callback change is explained, but residual MicrobeDecoder and GO contributor-level attribution limits remain documented. Neither the original strict status nor the native-baseline drift marker was rewritten. This closure does not waive strict/profile findings, approve every chemical identity, or authorize publication.

    The byte-exact evidence companion completed with RETENTION_VERIFIED_ONLY: 56 retained roles / 56 content-addressed objects, 60 guarded input identities. Manifest SHA-256 e13b939e9af0201f45ef552ab31fe816b494e75149e6a8ec8840a05b3f3bfc65; original attempt SHA-256 5728d058196a0b4d34fd103a848cb8c42c2ae9baf3c8e98f846894623998ecfa. Archive/raw data are hash-bound, not copied, and not every linked review dependency is embedded. Retention does not grant execution identity, scientific acceptance, or release authority.

    Project-relative evidence root: data/issue1224-quarantine-20260929.xjuS9H/fresh-source-acceptance-1408/.

    • Final synthesis: FINAL-KG-MODEL-PATH-REVIEW-02.md, SHA-256 c79fdf30e9b8a0b39c6ad8a9aa717f3597cc0b5d207177cd458ccd6fdc811d22.
    • Full MicrobeDecoder projection: category-union-projector-review/actual-candidate-9faxocfp-01/result.json, SHA-256 288e162012d19281041f814a64eff731688cf3bf0e5cc01597439323bc9c5f48.
    • Whole affected-source projection: source-projection-02/result.json, SHA-256 c5526e13ab4b72ceab168baeac3c4310d440b96585cbf596ef5f3b29ccde0ce9.
    • Exact native disposition: NATIVE-FINITE-SCIENTIFIC-DISPOSITION.md, SHA-256 c1650e7c41ee27cdc29cffd2fcdec4ffb1b23d21398c303259320f3a6e7f611f.
    • Companion: evidence-companion-01/; observed execution receipt evidence-companion-execution-01.json, SHA-256 0dc9f3b4d167902b35016a25f0ac1becb6f3681eb8463f825d68c1f69718726f.
    • Independent terminal readback: COMPANION02-TERMINAL-INDEPENDENT-READBACK.md, SHA-256 519b3e837e0b90890ce483bd0b05db8846ee7986904afdbbc97b1605fd97693a. All 60 original inputs and 56 retained objects were independently rehashed; the bounded MicrobeDecoder curation gap: 5,224 distinct labels unmapped #650 closure was reviewed and supported.

    The production changes have landed at the commit named above; local HEAD and origin/master match it. Acceptance-checker findings #1265 and #1267–#1274 were corrected, independently reviewed, exercised against the actual candidate, and independently confirmed closed in a final live GitHub check; original failures remain retained. No additional production edits were needed in final synthesis/retention.

    #286 remains open for separate broader historical chemical curation. The canonical archive and user-owned statistics remain unchanged. This candidate has not replaced the canonical graph and has not been released.
13. realmarcin

    closed this as completedon Sep 30, 2026