Skip to content

Reconcile validate-counts vs drift checker stale-counts pattern across ecosystem #12

Closed

Description

@TMHSDigital

Background

Discovered during Phase 2 Session D-0 audit of all 9 tool repos.

Two distinct patterns for handling aggregate counts (X skills, Y rules, Z tools) currently coexist in the ecosystem:

Pattern 1: Enforce in README.md via CI

  • CFX-Developer-Tools and Unity-Developer-Tools have a validate-counts job in their validate.yml.
  • It reads .cursor-plugin/plugin.json, computes live counts, and greps README.md for the literal strings "{N} skills", "{N} rules", "{N} snippets", "{N} templates".
  • CI fails if the strings do not match. README is forced to stay in sync.

Pattern 2: Flag as stale in agent files via drift checker

  • Phase 2 Session B's stale-counts check flags any \d+\s+(skills?|rules?|MCP\s+tools?|tools?|commands?|hooks?) pattern in agent files (AGENTS.md, CLAUDE.md, SKILL.md, .mdc).
  • Per Q3 resolution: warn-only, suggest removal, do not sync.
  • Smoke test surfaced 62 such warnings across 8 repos (mostly AGENTS.md / CLAUDE.md narratives like 177 skills, 71 rules).

The tension

The two patterns disagree on what aggregate counts are:

  • Pattern 1 treats them as canonical surface that must stay accurate.
  • Pattern 2 treats them as stale narrative that should be removed and replaced with a registry pointer.

Today they don't actually conflict at the file level (Pattern 1 only inspects README.md; Pattern 2 only inspects agent files). But the ecosystem has not picked a position. Tool-repo authors are getting two contradictory signals as we add the drift checker.

What needs deciding

  1. Do aggregate counts belong in agent files at all?
  2. If yes, should the drift checker stop flagging them? (Tighten stale-counts to specific contexts; add a config knob; or remove the check entirely.)
  3. If no, who is responsible for removing them? Mass-edit PR? Per-repo grooming?
  4. What about README.md? Keep the CFX/Unity validate-counts pattern, or migrate to "registry is source of truth, README links to it"?

Likely path

Probable resolution direction (to validate):

  • Agent files: aggregate narratives are stale. stale-counts stays as warn. Tool-repo authors should remove or rephrase as see registry.json for live counts.
  • README.md: keep the validate-counts pattern for now; reconsider in a future ecosystem-wide grooming session.

Acceptance criteria

  • Documented decision in standards/ (likely a new section in versioning.md or a new file like standards/aggregate-counts.md).
  • If keeping stale-counts as warn: a known-false-positive list or skip-pattern for legitimate technical counts (e.g., firewall rule counts in Home-Lab; see related issue).
  • If removing the check: bump checker minor and document rationale.

References

Activity

  1. TMHSDigital commented on Apr 24, 2026

    @TMHSDigital
    OwnerAuthor

    Cross-link: see #13 for the SKILL.md sub-question (technical-noun false positives).

  2. TMHSDigital commented on Apr 25, 2026

    @TMHSDigital
    OwnerAuthor

    Phase 2 Session D D-1 canary surfaced this tension concretely. Suppressing stale-counts at the cursor-plugin type level via standards/drift-checker.config.json to unblock the rollout (#16, v1.7.5).

    This is not a final answer to the policy question. The interim decision: agent files (AGENTS.md, CLAUDE.md) carry narrative aggregates as descriptive prose; aggregate truth-bearing checks belong in CFX/Unity-style validate-counts jobs against README.md (the user-facing aggregate).

    Open question for future work: can stale-counts be made smarter to distinguish narrative ("the skills/ directory contains 9 SKILL.md files") from truth claims ("CFX has 9 skills")? If so, re-enable it.

    Leaving this issue open. Updated by Phase 2 Session D D-1.

  3. TMHSDigital commented on Apr 25, 2026

    @TMHSDigital
    OwnerAuthor

    DTD#13 investigation completed and produced a finding relevant to this policy discussion.

    The hypothesis going in was that SKILL.md findings would mirror the AGENTS.md/CLAUDE.md narrative-aggregate case (suppressing them via type-level config like the existing cursor-plugin suppression).

    The data refuted this hypothesis. Investigation of 5 SKILL.md findings showed:

    • 4 category A (genuine count claims)
    • 1 category C (false positive from example dialogue)
    • 0 category B (narrative aggregates)

    One of the category A findings was a real off-by-one drift in steam-api-reference/SKILL.md (claimed 7 tools, table had 8). Drift-check did its job. Extending type-level suppression to SKILL.md would have masked this bug.

    Starting position for the eventual DTD#12 policy session:

    1. Do NOT extend cursor-plugin type-level stale-counts suppression to SKILL.md.
    2. The category C false positive (example dialogue) is better solved by a drift-check enhancement (filed as drift-check: stale-counts should skip example dialogue sections in SKILL.md #37) rather than blanket suppression.
    3. The longer-term direction may be validate-counts integration: auto-count table rows under (N tools) headers and compare to the claim, rather than the current generic stale-counts pattern.

    DTD#13 closure: investigation complete, follow-ups filed (Steam content-fix PR TMHSDigital/Steam-Cursor-Plugin#13 merged, drift-check enhancement #37). Closing DTD#13 separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions