Stack: transplant PR#70 value onto PR#78 (durable-ingestion) at 6.7.3 - #79
Open
Diego Colombo (colombod) wants to merge 6 commits into
Open
Stack: transplant PR#70 value onto PR#78 (durable-ingestion) at 6.7.3#79Diego Colombo (colombod) wants to merge 6 commits into
Diego Colombo (colombod) wants to merge 6 commits into
Conversation
Carry the last-committed cross-handler cursor durably through commit() and restore it before a rebuilt worker processes its first batch, so a respawn resumes the counters instead of reminting node ids. Retry replays a batch from the pre-attempt cursor snapshot (idempotent MERGE, no duplicate Iteration). Add a run-scoped tiebreaker so Iteration/ContentBlock/state ids are unique per orchestrator run and survive a worker rebuild. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Strip a stale IncompleteSession marker on every start/fork transition. The marker is only correct at session:end on a bare session whose start/fork was lost; its co-occurrence with a start/fork is always the out-of-order case (a forked sub-session's session:end drained before its session:start/fork). Add the one-off backfill scripts that relabel historical nodes already mislabeled before this fix, plus legacy pooled-iteration tagging. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
…lift Add a compiled SCHEMA_VERSION baseline and a graph-vs-compiled drift signal, plus the blob-carrier allowlist tripwire (an unregistered ci-blob:// carrier fails closed and dead-letters rather than minting a reclaim-invisible ref). Lift working_dir onto the Session node with a DB-level coalesce(n.working_dir, row.working_dir): populate-if-missing, an already-set value is never clobbered by a concurrent or replica writer. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
… /status re-seat Add the MaintenanceCoordinator and the allow-list gate middleware (structured 503 + Retry-After for non-allow-listed paths while a repair runs; /status, /version, /admin/maintenance never gated), the /admin/maintenance execution channel, and blob-reclaim (dry-run default, required max_delete cap, graph-wide carrier scan). On un-migrated data at boot, auto-run the repair ONLY when the writer lease is armed (cross-replica safe; a single-flight mutex prevents overlap with a concurrent /admin/maintenance), re-arm the global schema gate, and keep scheduling the retry sweep. Surface degraded_reason and schema-version drift on /status. PR#78 primitives preserved: fail_on_data_conflict=True and the schema_ready gate are untouched; deploy-safe-boot was not brought. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Add a standalone migrations CLI that wraps the existing neo4j_store repair and diagnose functions (dedup, :Node backfill, constraint re-assert) for operators rectifying an already-degraded graph out-of-band -- never at server startup. No new algorithm; SCHEMA_VERSION stays 1 (a server-version / structural rectification step, not a schema bump). 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Set server version to 6.7.3 (stacked on PR#78's 6.7.x line). SCHEMA_VERSION stays 1. Record the maintenance-mode entry in the migration manifest. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on top of #78 (
harden-durable-ingestion). Re-homes PR#70's improvements onto PR#78's storage-correct base (issue #473: cloud-FS non-atomic append). Merge #78 first, then this.PR#78 primitives preserved throughout:
ensure_neo4j_schema(fail_on_data_conflict=True)and the globalschema_readygate are untouched; PR#70's deploy-safe-boot behavior was deliberately NOT brought.Six logical commits:
feat(ingest)— durable cursor, retry-dedup, run-id tiebreaker (restore off the idle path, before the first real batch; survives worker rebuild)feat(session)— IncompleteSession heal-forward on start/fork + legacy relabel scriptsfeat(schema)— schema-version subsystem + working_dir DB-levelcoalescenon-overwrite lift + blob-carrier allowlist tripwirefeat(maintenance)— maintenance mode + gate middleware +/admin/maintenance+ lease-armed auto-repair on boot (fires only when the writer lease is armed; single-flight mutex; re-arms the global gate; a schema failure no longer skips the retry sweep) + degraded_reason +/statusdrift re-seatfeat(migrations)— out-of-band graph rectification CLI (wraps existing run_repair/diagnose; SCHEMA_VERSION stays 1)chore(release)— server version 6.7.3Verification: full non-neo4j suite 2152 passed / 7 skipped; neo4j suite green including the two finalization-OOM tests that #78's latest commit (
keep finalize-orphan worker registered) fixes — this branch is rebased onto that commit so it inherits the fix. Every transplant task was gated through an adversarial claim-verification pass before landing.🤖 Generated with Amplifier