analyze.py: classification thresholds unvalidated against real repos

[hailcpy]

Scoring weights (+3/+2/threshold 3) pass on synthetic fixtures because the fixtures were built to match — circular. Not yet validated that the design matches real-world "architectural".

Fix: label chunks from a few real OSS repos, compute precision/recall, tune weights against that instead of the synthetic set.

[hailcpy]

The config refactor (PR incoming) adds the mechanism this needs — score weights and the arch/borderline thresholds are now tunable via --config (weights, score_arch_threshold, score_borderline_threshold) instead of hardcoded.

The actual validation work — label chunks from real OSS repos, compute precision/recall, tune against that instead of the circular synthetic set — is still open and is what this issue should track.

[hailcpy]

Additional review detail from code audit:

Finding

Running python3 scripts/analyze.py . --json on this repo itself exposes a calibration problem: the deterministic verifier work is skipped even though it is arguably the central design decision in the project. The classifier is tuned around conventional application repos and source paths, not tool/plugin/skill repos.

Impact

The current thresholds may look good on synthetic fixtures but miss important decisions in real repositories, especially when product code lives in places like scripts/, bin/, tools/, plugin manifests, generated skill files, or infra/config-heavy projects.

Suggested solution

Make calibration repo-aware:

• detect repository kind: app, library, CLI/tooling, infra, plugin/skill, monorepo module,
• derive source roots from files and tests instead of hard-coding only src/lib/core/app/services/internal/pkg,
• create a small labeled corpus from real repos and include this repo as a self-check fixture,
• report precision/recall by repo kind, not only aggregate.

Acceptance criteria

• This repo’s verifier/citation-grounding work is classified at least borderline/architectural in a self-check fixture.
• Source roots are configurable and/or auto-detected.
• Calibration tests include at least one CLI/tooling repo shape.
• Thresholds can be tuned per repo kind without hand-editing the script.