Skip to content

feat(canonical-migration): --remap-existing to re-evaluate already-migrated edges (#45) - #208

Merged
jphein merged 1 commit into
mainfrom
feat/45-canonical-remap-existing
May 31, 2026
Merged

feat(canonical-migration): --remap-existing to re-evaluate already-migrated edges (#45)#208
jphein merged 1 commit into
mainfrom
feat/45-canonical-remap-existing

Conversation

@jphein

@jphein jphein commented May 31, 2026

Copy link
Copy Markdown
Collaborator

Why

The first-migration UPDATE in scripts/canonical_migration.py is one-shot by design: its AND NOT (properties ? 'raw_relation_type') guard makes a second run a no-op (every migrated edge already carries raw_relation_type). That's correct for the original rollout but blocks the #45 use case — after the predicate normalizer (mempalace.kg_predicate_norm, improved in mempalace#336) is upgraded, the bulk of the other bucket should be re-evaluated, and those edges are exactly the already-migrated ones a naive re-run skips.

On the production graph, 55% of 1.92M RELATION edges sit in other, each retaining its original predicate in raw_relation_type. The improved normalizer reclaims ~half of them — but only if the migration re-reads the original predicate and is allowed to overwrite an existing canonical.

What

--remap-existing (default OFF) switches both halves of the migration:

  • Frequency read uses coalesce(r.raw_relation_type, r.relation_type) so the mapper re-evaluates the original predicate, not the prior (possibly other) canonical.
  • Apply UPDATE keys the join on the same coalesced original, drops the first-migration guard, preserves the original (coalesce keeps an existing raw_relation_type, only backfilling it on never-migrated edges), and adds relation_type IS DISTINCT FROM m.canonical so only edges whose canonical actually changes are touched (no MVCC churn on no-ops).

Default behavior is byte-for-byte unchanged without the flag. Same backup + paused-worker gating applies (the --apply + --i-have-a-backup guards are untouched).

# dry-run the re-map (read-only via /cypher)
python scripts/canonical_migration.py --remap-existing
# apply (host-side, backup in hand, worker paused)
python scripts/canonical_migration.py --remap-existing --apply --dsn "$MEMPALACE_POSTGRES_DSN" --i-have-a-backup

Prerequisite chain

This is the mechanism for the #45 prod re-map. mempalace#336 (merged) improved the normalizer; this PR lets the existing graph reflect it without an LLM re-call (pure deterministic relabel of raw_relation_type). The actual prod run remains gated on a backup + explicit approval.

Tests

6 new tests in TestRemapExisting: remap cypher reads the original, apply drops the guard + keys on the coalesced original + uses IS DISTINCT FROM, default keeps the guard, plus a back-compat getattr(..., False) case for callers built before #45. 18/18 pass; ruff clean; module docstring updated with the new section + exact SQL.

🤖 Generated with Claude Code

…grated edges (#45)

The first-migration UPDATE is one-shot by design: its
`AND NOT (properties ? 'raw_relation_type')` guard makes a second run a no-op,
because every migrated edge now carries `raw_relation_type`. That's correct for
the original rollout but blocks the #45 use case — after the predicate
normalizer (mempalace.kg_predicate_norm) is improved, the bulk of the `other`
bucket should be re-evaluated, and those edges are exactly the already-migrated
ones a naive re-run skips.

`--remap-existing` (default OFF) switches both halves:
- Frequency read uses `coalesce(r.raw_relation_type, r.relation_type)` so the
  mapper re-evaluates the ORIGINAL predicate, not the prior (possibly `other`)
  canonical.
- The apply UPDATE keys the join on the same coalesced original, DROPS the
  first-migration guard, PRESERVES the original (coalesce keeps an existing
  raw_relation_type, only backfilling it on never-migrated edges), and adds
  `relation_type IS DISTINCT FROM m.canonical` so only edges whose canonical
  actually changes are touched (no MVCC churn on no-ops).

Default behavior is byte-for-byte unchanged. Same backup + paused-worker gating.
Prerequisite for the #45 prod re-map (mempalace#336 improved the normalizer;
this lets the existing 1.92M-edge graph reflect it without an LLM re-call).

6 new tests: remap cypher reads the original, apply drops the guard + keys on
coalesced original + IS DISTINCT FROM, default keeps the guard, and a
back-compat getattr-defaults-false case. 18/18 pass; ruff clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings May 31, 2026 01:08
@gemini-code-assist

Copy link
Copy Markdown

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@jphein
jphein merged commit 89a7d5a into main May 31, 2026
1 check failed
@jphein
jphein deleted the feat/45-canonical-remap-existing branch May 31, 2026 01:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants