Skip to content

fix(watch): require deletion evidence before evicting nodes for a missing source#1795

Closed
CJNA wants to merge 1 commit into
Graphify-Labs:v8from
CJNA:fix/eviction-requires-deletion-evidence
Closed

fix(watch): require deletion evidence before evicting nodes for a missing source#1795
CJNA wants to merge 1 commit into
Graphify-Labs:v8from
CJNA:fix/eviction-requires-deletion-evidence

Conversation

@CJNA

@CJNA CJNA commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Problem

_reconcile_existing_graph treats "source identity not in the collected corpus" as "source deleted" and evicts its nodes, edges, and hyperedges. But corpus absence is an ambiguous signal — it is also what you observe when the file still exists and merely stopped being collected because ignore rules or filters changed.

Real-world incident (27k-node graph): upgrading 0.8.x → 0.9.x brought the merged-.gitignore scan semantics (#1363). The repo's .gitignore excludes a local-only docs directory that had been deliberately built into the graph. On the first incremental rebuild after the upgrade, all 655 nodes from that directory — files present on disk the whole time — were silently evicted as "stale sources". The rebuild log line reported success; recovery was only possible because a manual pre-upgrade graph.json backup existed.

This is a vacuous-pass failure: the eviction logic reads absence of the file from a list as positive evidence of deletion.

Fix

Fail-closed eviction: before treating a corpus-absent identity as deleted, require the actual deletion evidence — Path(identity).exists() returning False. If the file is alive, its records are preserved and one loud log line reports how many nodes from how many files were kept and why:

[graphify watch] fail-closed: kept 655 node(s) from 37 file(s) that left the scan corpus but still exist on disk (ignore rules or filters changed?). Run a full re-extraction to purge them if the exclusion is intentional.

Behavior deliberately unchanged everywhere else:

  • True deletions evict exactly as before (file gone from disk).
  • Renames (incl. hook lists carrying only the destination) evict as before — the old path is gone from disk.
  • Deliberate exclusions are still purgeable: a full re-extraction drops AST-origin nodes under the watch root via the existing ownership rule, so graphify extract --force remains the intentional cleanup path. (Extending deletion-evidence semantics to full rebuilds is a larger design question, left out of scope.)

Existence checks are memoized per identity, so the cost is one stat per file that left the corpus — zero for the common case.

Tests

  • test_rebuild_code_preserves_nodes_from_excluded_but_alive_file — end-to-end incident reproduction: build a corpus with a markdown doc, then add an ignore rule covering it (file stays on disk), incremental rebuild → nodes preserved + "fail-closed: kept" message asserted.
  • test_rebuild_code_still_evicts_when_excluded_file_is_also_deleted — the guard must not weaken true-deletion eviction.
  • Full tests/test_watch.py + tests/test_build.py + tests/test_hooks.py: 151 passed, 2 skipped (all pre-existing eviction/rename/prune tests green). ruff check clean.

Sibling of #1785 (same graph, same debugging session).

🤖 Generated with Claude Code

…sing source

The rebuild reconciliation treated 'source identity not in the collected
corpus' as 'source deleted' and evicted its nodes. But absence from the
corpus is ambiguous: it also happens when the file still exists and
merely stopped being collected — ignore rules or filters changed (e.g.
the scanner newly honoring a .gitignore whose patterns cover wanted
docs). On a real 27k-node graph, upgrading into gitignore-honoring scan
semantics silently evicted all 655 nodes of a gitignored-but-present
docs directory on the first incremental rebuild; recovery was only
possible because a manual pre-upgrade backup existed.

Make eviction fail-closed: before treating a corpus-absent identity as
deleted, require the positive evidence — the file actually gone from
disk. Excluded-but-alive files keep their nodes and a loud log line
says how many were kept and why. True deletions and renames evict
exactly as before (the old path is gone from disk in both cases), and a
full re-extraction still purges deliberately excluded sources via the
existing AST ownership rule.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@CJNA

CJNA commented Jul 11, 2026

Copy link
Copy Markdown
Contributor Author

Real-world verification on the graph from the original incident (26k nodes): re-disabled the .graphifyignore re-includes so the gitignored docs directory left the corpus again, then ran an incremental rebuild with this branch installed.

[graphify watch] fail-closed: kept 655 node(s) from 35 file(s) that left the scan corpus but still exist on disk (ignore rules or filters changed?). Run a full re-extraction to purge them if the exclusion is intentional.

All 655 nodes that were silently evicted under 0.9.13 stock survived, graph byte-level intact after restoring the ignore rules. Same rebuild with the file genuinely deleted still evicts (covered by the second test).

safishamsi pushed a commit that referenced this pull request Jul 11, 2026
#1795)

_reconcile_existing_graph treated "source identity absent from the collected
corpus" as deletion and evicted its nodes/edges/hyperedges. But corpus absence
is ambiguous: it's also what you see when a file still exists and merely stopped
being collected (ignore rules or filters changed). Upgrading into the merged-
.gitignore scan semantics (#1363) mass-evicted 655 nodes from a deliberately-
built, .gitignore'd docs dir whose files were present the whole time — reported
as a successful rebuild.

Fail-closed: before evicting a corpus-absent identity, require Path(identity)
.exists() is False (identity is an absolute path). Alive-but-excluded sources
are preserved (nodes, edges, hyperedges) and a loud line reports how many were
kept and why. True deletions and renames still evict (old path gone from disk);
a full extract --force still purges deliberate exclusions via the AST ownership
rule. Existence is memoized (one stat per file that left the corpus).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@safishamsi

Copy link
Copy Markdown
Collaborator

Merged on v8, authored to you. Verified end-to-end: an excluded-but-alive file (added to .graphifyignore, kept on disk) survives an incremental rebuild with the fail-closed: kept N node(s) line, and a genuinely deleted file is still evicted. Confirmed identity is always an absolute path (absolute_identity in both branches of _StoredSourcePaths.identity), so Path(identity).exists() is reliable, and the guard continues before populating any of the node/edge/hyperedge eviction sets so all three are preserved together. 106 watch/build tests + full suite (3161) green. Sibling of #1785 — same fail-safe instinct, nicely scoped. Thanks.

@safishamsi safishamsi closed this Jul 11, 2026
safishamsi added a commit that referenced this pull request Jul 12, 2026
Opening 0.9.13 accidentally renamed the 0.9.12 header instead of inserting a new
section above it, merging already-released 0.9.12 fixes under 0.9.13. Split them
back: 0.9.13 now lists only #1756/#1768/#1781/#1784/#1785/#1795/#1796/#1797/#1799;
0.9.12 (2026-07-10) holds the fixes shipped in that release. Date 0.9.13.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants