Best-effort indexing-coverage signal: mark files/regions that were not fully indexed

[DeusData]

Best-effort indexing-coverage signal

Framing (read first): this is a best-effort attempt to mark code that was not fully indexed into the graph. A flag means "we know this file/region is incomplete — prefer grep here." The absence of a flag is NOT a guarantee of completeness — it means no gap was detected, not that none exists. We never present this as a completeness proof.

Why

Today the graph can silently drop constructs and nothing surfaces it. Concrete case: cbm_path_within_root (an #ifdef-split-brace function) was entirely absent from the graph while its callers stayed linked (#960, #961). An agent searching the graph has no way to know it might be reading an incomplete picture and should fall back to grep.

What we can detect (best-effort, in decreasing confidence)

Class	Example	Signal
Purposely ignored	gitignore / excluded dirs / .cbmignore / vendored	already in the index response excluded — surface as not_indexed_by_design (a design note, kept separate from gaps)
File-level skip	unreadable / oversized / parse-timeout / worker crash	already in skipped[] (#785)
Parse-partial	#ifdef-split brace, truncated file, unparseable syntax	NEW — ts_node_has_error(root) + ERROR/MISSING line ranges. Verified: tree-sitter reports has_error=TRUE for exactly this case.
Extractor gap	tree-sitter parsed a function_definition, extractor didn't emit a node	NEW — per-file reconciliation of tree-sitter recognized-construct count vs. emitted node count (analog of the existing dump_verify). Per-language, only where calibrated to zero false positives.
Clean-but-wrong misparse	valid tree, mis-categorized def, count matches	Undetectable — the honest residual; part of why this is best-effort.

Deliverables

• Signal A (first increment): capture has_error + ERROR line-ranges per file → mark the File node (a present-when-incomplete marker, so absence never implies "complete") → surface in the index_repository response as parse_partial[]. Reproduce-first test with the #ifdef fixture (RED before, GREEN after). This alone would have caught cbm_path_within_root.
• Signal B (follow-up): parse-vs-extract count reconciliation, per calibrated language.
• Agent surfacing: the File-node marker so search/trace/snippet results carry "this file is partial" inline; and an index_coverage view (extend index_status) with grep-fallback guidance.
• Docs: state the best-effort guarantee explicitly wherever the signal is exposed.

Non-goals / honesty guards

• No claim of exhaustive completeness. Absence of a flag ≠ complete.
• Signal B must not false-positive — uncalibrated languages get Signal A only.
• not_indexed_by_design (ignored files) stays a separate, clearly-labelled list — never mixed into "gaps."

Relates to #961 (the extractor limitation this makes visible).

[DeusData]

Shipped via PR #968 (merge commit 97c6750, 8 commits a4c17bd…3eb294f, milestone 0.9.1-rc):

• Capture: cbm_extract_file flags files whose parse tree contains ERROR/MISSING regions, with 1-based line ranges (top-most regions, capped at 64). Definite recovery is subtracted first — a region fully re-extracted as definitions is not a miss, so e.g. a recovered def broken(: no longer produces a false flag, while the #ifdef-split case stays flagged (its first-branch function is genuinely lost).
• Report: index_repository returns parse_partial (distinct from skipped[] — these files WERE indexed); the per-run logfile lists both classes.
• Persist: separate index_coverage table written in the same persist step as all other post-dump SQL writes; full runs replace, incremental runs merge (fixed file → flag clears, deleted file → pruned). Coverage never mixes into the graph tables.
• Query: index_status carries the coverage report (no dedicated tool); query_graph(graph="missed") queries the miss graph — the file structure of only the not-fully-indexed files, materialized under a shadow project.
• Agent surfaces: the PreToolUse augmenter (matcher now Grep|Glob|Read) injects a coverage note when a flagged file is read; get_code_snippet responses from flagged files carry a correlated coverage_note. Both non-blocking, both anchored to a concrete file — no uncorrelated noise.
• UI: the missed graph renders as a white skeleton satellite beside the code galaxy (click to focus, click empty space to return, toggle in the sidebar); selecting it opens a report-an-edge-case callout with a prefilled upstream issue link and a copy-paste agent prompt.

Everything is deliberately best-effort: a flag means constructs in the listed ranges MAY be missing (prefer grep there); absence of a flag is NOT a completeness guarantee.

Validation on this repo's own index: 44 files flagged; the two large regions each contained exactly one genuinely lost function (cbm_path_within_root, handle_process_kill — both the in-function #ifdef _WIN32 split tracked in #961).

Remaining scope, keeping this issue open: the deliberately-ignored files listing ("purposely not indexed" — gitignore/excludes/mode-skips as a by-design class, exhaustive and deterministic), planned as a follow-up PR on the same milestone.