From 225cdc62f49965fba0352d9782c89dc327bc997d Mon Sep 17 00:00:00 2001 From: Gerard Rovira Date: Fri, 30 Oct 2026 16:06:28 -0700 Subject: [PATCH 1/4] [lexical-website][lexical-y] Bug Fix: Keep every unchanged word of a line whose text nodes an update makes again with several words changed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Description When an update removes a line's text nodes and makes new ones with other text (an agent rewriting a block from its JSON), the binding keeps the start and end the new text shares with the old (#9598) and writes everything between the first and the last changed word again as new text. With two or more changed words, a word a collaborator deleted between them at the same time came back on every client, and when the collaborator then undid its delete the word showed twice (`t1z t5zn9z t3z t4z t5z t6z n8z`). A letter a collaborator typed into such a word landed outside it, and a format it gave the word was lost. Between the common start and end, the text of the nodes made now keeps every run of unchanged whole words, with the white space around it that both texts share, found by a word-by-word shortest edit script (Myers) of the replaced middle. Only the changed words are written, so a collaborator's concurrent delete there stays deleted (and is restored once by its undo), typed letters stay in their word and formats stay on it. Words are those of the existing whole-word rule (D30): runs of non-space characters, each character of a script written without spaces, an inline node's object character, a line feed. White space alone is never kept between two changes, as at the ends. The script is bounded to 128 word edits per line, past which only the common start and end are kept, so the cost stays linear in the line's length (scaling test included). DECISIONS.md D29 now states the rule, and y.md's "Edits made by code" paragraph and its stale "Text nodes made again with other text" Limits entry describe what is kept and what is still written again (changed words, a word whose joined punctuation changed, a word joined to a changed word through an inline node, lines with more than 128 changed words). A stale format whose range reaches past a link the rewrite removes restates the link on kept text (#9597). With one changed word main already shows it on the kept end; keeping the unchanged words between two changes shows it there too. `RemadeTextStaleLink.test.ts` pins those 20 cells as expected failures by an exact predicate; #9601 (the #9597 fix) makes them pass (checked on this branch merged with #9601: 36/36), so the markers flip when it lands. Closes #9621 ## Test plan New tests: `regressions/RemadeTextStretches.test.ts` (the #9621 cases in both id orders, concurrent and unread, with and without the undo; other word shapes: a script without spaces, emoji, runs with different white space; a property over random lines of unique words in 1 to 4 text nodes, some bold or italic, made again with 2 to 4 word edits while a collaborator deletes, types into, deletes and undoes, types and undoes, or bolds an unchanged word, 120 seeds per cell; a long-line scaling test), 336 new cells in `matrix/C9RemadeText.test.ts` (every inline shape of the catalog, a word changed in the first and the last text node, the collaborator types into, deletes from, or deletes from and undoes `t2z` or `t3z` between them, concurrent and unread, both id orders), and `regressions/RemadeTextStaleLink.test.ts`. ### Before On main (fcea4ffe): ``` $ vitest --run --project unit packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStaleLink.test.ts packages/lexical-y/src/__tests__/unit/matrix/C9RemadeText.test.ts ❯ unit packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStaleLink.test.ts (36 tests | 20 failed) ❯ unit packages/lexical-y/src/__tests__/unit/matrix/C9RemadeText.test.ts (564 tests | 336 failed) ❯ unit packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts (34 tests | 30 failed) Tests 386 failed | 248 passed (634) AssertionError: expected [ 't1z n9z t3z t4z t5z t6z n8z', …(2) ] to deeply equal [ 't1z n9z t3z t4z t6z n8z', …(2) ] AssertionError: expected [ 't1z t5zn9z t3z t4z t5z t6z n8z', …(2) ] to deeply equal [ 't1z n9z t3z t4z t5z t6z n8z', …(2) ] "seed 1 ids 2,1 [\"t9\",\"38z t177z t158\",\"z t1\",\"99z\"] -> \"t215z, t938z t177z t158z\"; delete \"t177z\" at 1+2: \"t215z, t938z t177z t158z\": \"t7z\" missing, \"t177z\" came back" ``` The property fails 307 of its sessions on main (delete 103, type 96, delete and undo 81, bold 27). The 20 `RemadeTextStaleLink` failures on main are its expected-failure cells passing, because main writes the middle anew. ### After ``` $ vitest --run --project unit packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStaleLink.test.ts packages/lexical-y/src/__tests__/unit/matrix/C9RemadeText.test.ts ✓ unit packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStaleLink.test.ts (36 tests) ✓ unit packages/lexical-y/src/__tests__/unit/matrix/C9RemadeText.test.ts (564 tests) ✓ unit packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts (34 tests) Tests 614 passed | 20 expected fail (634) $ pnpm run ci-check EXIT 0 $ vitest --run --project unit --project scripts-unit Test Files 637 passed | 4 skipped (641) Tests 26003 passed | 210 expected fail | 135 skipped (26348) ``` (main: 25617 passed | 190 expected fail | 135 skipped; the difference is the new tests.) ``` $ FUZZ_SEED_START=1 FUZZ_SEEDS=3000 vitest --run --project unit packages/lexical-y/src/__tests__/unit/Fuzz.test.ts Tests 1 failed | 123 passed (124) AssertionError: expected [ 'seed 2277: dup' ] to deeply equal [] $ FUZZ_SEED_START=50000 FUZZ_SEEDS=2000 vitest --run --project unit packages/lexical-y/src/__tests__/unit/Fuzz.test.ts Tests 7 failed | 77 passed (84) (seeds 50180, 50231, 50372, 51439, 51507, 51972, 51976) ``` The same seeds fail, with the same signatures, on main. --- .../lexical-y-remade-text-unchanged-words.md | 7 + .../lexical-website/docs/collaboration/y.md | 48 +- packages/lexical-y/DECISIONS.md | 4 +- .../unit/matrix/C9RemadeText.test.ts | 108 ++++ .../regressions/RemadeTextStaleLink.test.ts | 159 ++++++ .../regressions/RemadeTextStretches.test.ts | 531 ++++++++++++++++++ packages/lexical-y/src/internal/Inline.ts | 198 +++++++ packages/lexical-y/src/internal/TextEdits.ts | 25 +- 8 files changed, 1047 insertions(+), 33 deletions(-) create mode 100644 .changeset/lexical-y-remade-text-unchanged-words.md create mode 100644 packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStaleLink.test.ts create mode 100644 packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts diff --git a/.changeset/lexical-y-remade-text-unchanged-words.md b/.changeset/lexical-y-remade-text-unchanged-words.md new file mode 100644 index 00000000000..c09d2b7984f --- /dev/null +++ b/.changeset/lexical-y-remade-text-unchanged-words.md @@ -0,0 +1,7 @@ +--- +'@lexical/y': patch +--- + +[fix] Keep every unchanged word of a line whose text nodes an update makes again with several words changed, so a collaborator's concurrent delete between two changed words stays deleted + +Verify: an update that removes a line's text nodes and makes new ones with two or more words changed (an agent rewriting a block from its JSON) writes only the changed words. A word a collaborator deletes between two of them at the same time stays deleted, and shows once when the collaborator undoes the delete; a letter a collaborator types into an unchanged word between them stays in that word. diff --git a/packages/lexical-website/docs/collaboration/y.md b/packages/lexical-website/docs/collaboration/y.md index f47d0159968..efd71478179 100644 --- a/packages/lexical-website/docs/collaboration/y.md +++ b/packages/lexical-website/docs/collaboration/y.md @@ -800,10 +800,13 @@ person's: then edits them. Where the line's text changed, the text of nodes made in place of removed ones keeps the start and end it shares with the line's text as it was, by character, whatever nodes held it (as y-prosemirror - keeps a text run's): a block's text nodes made again from its JSON with - one word changed write that word alone, so a letter a collaborator types - into another word of the line at the same time stays in that word, and a - character a collaborator deletes there stays deleted. A replacement + keeps a text run's), and between two changes every run of unchanged + words, with the white space around it that the two texts share: a block's + text nodes made again from its JSON with some words changed write those + words alone, so a letter a collaborator types into another word of the + line at the same time stays in that word, and a character a collaborator + deletes there stays deleted (and is not shown twice when the collaborator + undoes the delete). A replacement replaces whole words, as below, and so does an insertion between two letters or digits of a word or a deletion inside one, so that an edit computed before it that replaces the old word takes nothing out of the new @@ -813,8 +816,10 @@ person's: between the changed words and the kept text is written with them, unless it was a node of its own (a tab, a line break, a node of spaces). A word joined to a changed word through an inline node, with no space between, is - written again with it; text between two separate changes is written - again, and an inline node made there is new where it is put. Prefer + written again with it, and an inline node made between two changes is new + where it is put. Where a line has more than 128 changed words, only its + common start and end are kept, so the comparison stays linear in the + line's length. Prefer these to `setTextContent`, which sets a node's text whole: such a change (and a change the browser makes to the DOM text, as an IME composition does) keeps the start and end its text shares with the node's old text, @@ -992,19 +997,24 @@ not merge as a user might expect: collaborator types in those items at the same time is lost. - **Text nodes made again with other text.** Where an update replaces a line's text nodes with new ones and the line's text changed (an AI rewrite - that makes a block's text nodes again with one word changed, or removes a - block's children and appends one new text node), a replaced node whose - text is not all there unchanged is written again whole: its characters are - deleted, and the new text is inserted. What a collaborator types into that - text at the same time, which the update's author has not seen, stays where - it was, so it lands outside the new text, before it (a letter typed at the - end of a word ends up in front of the rewritten text, out of its word), - and the collaborator's formatting of that text at the same time is not - kept. The concurrent typed text is kept, and every client shows the same - result. Edit the changed - words in place instead (`TextNode#spliceText`, the selection's - `insertText`; see [Edits made by code](#edits-made-by-code-agents-tools)): - only the changed characters are written. + that makes a block's text nodes again from their JSON with some words + changed, or removes a block's children and appends one new text node), the + unchanged text keeps its characters (see + [Edits made by code](#edits-made-by-code-agents-tools)), and each changed + word is written again whole: its characters are deleted, and the new word + is inserted. A word counts as changed when any character of it changed, + punctuation joined to it included (`t5z,` made again as `t5z`), and so + does a word joined to a changed word through an inline node. What a + collaborator types into a changed word at the same time, which the + update's author has not seen, stays where it was, so it lands next to the + new word, not in it (a letter typed at the end of the word ends up in + front of the new word), and the collaborator's formatting of the changed + word at the same time is not kept. In a line with more than 128 changed + words, everything between the first and the last change is written again. + The concurrent typed text is kept, and every client shows the same + result. Edit the changed words in place instead + (`TextNode#spliceText`, the selection's `insertText`): only the changed + characters are written. - **Delete wins.** Text typed into a container that a collaborator deletes at the same time is deleted with it, and so is a block moved into it. - **Last writer wins** for the fields of one link (URL and target change diff --git a/packages/lexical-y/DECISIONS.md b/packages/lexical-y/DECISIONS.md index 5393d6811fd..6dbfa56d825 100644 --- a/packages/lexical-y/DECISIONS.md +++ b/packages/lexical-y/DECISIONS.md @@ -60,7 +60,7 @@ issues and docs that carry it. | D26 | **Undo favours people taking turns over concurrent races.** Where a fix makes undo correct for sequential edits at the cost of rare races between an undo and a simultaneous edit of the same text, the sequential case wins. | Taking turns is the common case; a race with a simultaneous edit of the same text is rare. Picked on 10-28, confirmed on 10-30. | `DECIDED` | 10-30 | [#9400] | | D27 | **Moves are a delete and a re-insert** at launch; the binding rewrites the cheaper side. Real moves within a container ([#9449], records after Yjs's unmerged `ContentMove` design) are paused and land last, backward compatible: a new layout version that older clients open read only. Move records leave tombstones, as every Yjs binding does. Moves across containers stay copies. | Every Yjs binding turns moves into delete plus insert; Yjs dropped its own move for complexity. Reliability of the rest came first. | `DECIDED` | 10-25 | [#9449], [#9455], [y.md: Limits][limits] | | D28 | **Text identity is fixed at the source.** While a binding is attached, core records text edits (`spliceText`, insert and remove, split and merge), and text entity plugins record a grow as a move and a remake as a retype. Text only `setTextContent` or the DOM changed is diffed. The binding adds no per-case heuristics. | No binding can recover intent an API dropped; special-case rules were fragile and regressed neighbouring cases. | `DECIDED` | 10-27 | [#9483], [y.md: Edits made by code][y-code] | -| D29 | **Remade text keeps its common start and end by character**, whatever the node boundaries (y-prosemirror's `updateYText` rule), and never brings back a collaborator's delete. It replaces the word matching of [#9565]. | It is the industry-standard behaviour, and word matching glued words together and merged numbers oddly. | `DECIDED` | 10-30 | [#9565] | +| D29 | **Remade text keeps its common start and end by character**, whatever the node boundaries (y-prosemirror's `updateYText` rule), **and between two changes every run of unchanged whole words** (a word-by-word shortest edit script, bounded to 128 word edits per line so it stays linear; past it, only the start and end), and never brings back a collaborator's delete. It replaces the word matching of [#9565]. | It is the industry-standard behaviour, and word matching glued words together and merged numbers oddly. Keeping only the start and end wrote everything between two changed words again, so a collaborator's delete there came back, and showed twice after the collaborator's undo ([#9621]). | `DECIDED` | 10-30 | [#9565], [#9598], [#9621] | | D30 | **Diffed rewrites snap to whole words**; keystrokes and IME composition keep the exact diff. Scripts without spaces treat each character as a boundary. | Concurrent rewrites otherwise split words. Real word segmentation is a follow-up ([#9467]). | `DECIDED` | 10-26 | [#9400], [#9467] | | D31 | **A list item's nested list gets an item of its own.** `NodeStructureExtension`'s list rule prevents an item holding a nested list next to other content when it is written; the binding does not repair it afterwards. | That shape lost text in outdents, Backspaces and offline merges. Preventing it at write time follows D20. | `DECIDED` | 10-26 | [#9452], [y.md: Limits][limits] | | D32 | **List merging ignores `start`.** | Existing Lexical behaviour, intended. | `DECIDED` | 10-23 | [#9452] | @@ -191,6 +191,7 @@ issues and docs that carry it. [#9561]: https://github.com/meta/lexical/pull/9561 [#9582]: https://github.com/meta/lexical/issues/9582 [#9593]: https://github.com/meta/lexical/issues/9593 +[#9598]: https://github.com/meta/lexical/pull/9598 [#9599]: https://github.com/meta/lexical/issues/9599 [#9603]: https://github.com/meta/lexical/issues/9603 [#9605]: https://github.com/meta/lexical/pull/9605 @@ -198,3 +199,4 @@ issues and docs that carry it. [#9608]: https://github.com/meta/lexical/issues/9608 [#9609]: https://github.com/meta/lexical/issues/9609 [#9610]: https://github.com/meta/lexical/issues/9610 +[#9621]: https://github.com/meta/lexical/issues/9621 diff --git a/packages/lexical-y/src/__tests__/unit/matrix/C9RemadeText.test.ts b/packages/lexical-y/src/__tests__/unit/matrix/C9RemadeText.test.ts index 13c96e8b41f..d8ef2d244ae 100644 --- a/packages/lexical-y/src/__tests__/unit/matrix/C9RemadeText.test.ts +++ b/packages/lexical-y/src/__tests__/unit/matrix/C9RemadeText.test.ts @@ -332,3 +332,111 @@ describe('C9: a highlighted code line made again with one token changed', () => } } }); + +/** Every text in a JSON tree, with `changes` made in it. */ +function changed( + json: SerializedLexicalNode, + changes: readonly [string, string][], +): SerializedLexicalNode { + const node = {...json} as SerializedLexicalNode & { + children?: SerializedLexicalNode[]; + text?: string; + }; + if (typeof node.text === 'string') { + for (const [from, to] of changes) { + node.text = node.text.replace(from, to); + } + } + if (node.children) { + node.children = node.children.map(c => changed(c, changes)); + } + return node; +} + +/** Makes the line's nodes again from their JSON, with `changes`. */ +function $rewriteAll(changes: readonly [string, string][]) { + const line = $line(); + const json = line.getChildren().map(c => changed($deepJSON(c), changes)); + for (const child of line.getChildren()) { + child.remove(); + } + line.append(...json.map(j => $parseSerializedNode(j))); + $setSelection(null); +} + +// #9621: with a word changed on each side (in the first and the last text +// node), the words between the changes keep their characters too: the +// peer's edit of `t2z` or `t3z` stays where it was made, and a delete it +// undoes leaves the word once. +describe('C9: a line made again from its JSON with a word changed on each side keeps a collaborator edit between them', () => { + const CHANGES: [string, string][] = [ + ['t1z', 't8z'], + ['t4z', 't9z'], + ]; + for (const [name, $shape] of SHAPES) { + for (const [editName, edit, kind] of [ + ...EDITS, + [ + 'deletes and undoes the digit of', + (leaf: TextNode, at: number) => + void leaf.spliceText(at + 1, 1, '', false), + 'undo', + ] as const, + ]) { + for (const word of ['t2z', 't3z']) { + for (const timing of ['concurrent', 'unread'] as const) { + for (const ids of [ + [1, 2], + [2, 1], + ]) { + test(`${name}: the peer ${editName} "${word}", ${timing} (ids ${ids})`, async () => { + const t = await setup( + () => { + $getRoot().append( + $createParagraphNode().append($t('Title')), + $createParagraphNode().append( + $t('t1z t2z '), + ...$shape(), + $t(' t3z t4z'), + ), + ); + }, + {ids, nodes: [...NODES, HashtagNode]}, + ); + const before = lineText(t.a); + t.offline(); + if (timing === 'unread') { + t.b.update($edit(word, edit)); + t.a.editor.update(() => $rewriteAll(CHANGES)); + await t.sync(); + await flush(); + await t.sync(); + } else { + t.a.update(() => $rewriteAll(CHANGES)); + t.b.update($edit(word, edit)); + await t.sync(); + } + if (kind === 'undo') { + t.b.undo(); + await t.sync(); + } + let expected = + kind === 'undo' + ? before + : before.replace(word, edited(word, kind)); + for (const [from, to] of CHANGES) { + expected = expected.replace(from, to); + } + const fresh = await freshReader(t.a, [...NODES, HashtagNode]); + expect([...t.peers, fresh].map(lineText)).toEqual( + [...t.peers, fresh].map(() => expected), + ); + await expectConverged(t.peers); + expect(t.peers.flatMap(p => p.warnings)).toEqual([]); + }); + } + } + } + } + } +}); diff --git a/packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStaleLink.test.ts b/packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStaleLink.test.ts new file mode 100644 index 00000000000..55fc4b01f6b --- /dev/null +++ b/packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStaleLink.test.ts @@ -0,0 +1,159 @@ +/** + * Copyright (c) Meta Platforms, Inc. and affiliates. + * + * This source code is licensed under the MIT license found in the + * LICENSE file in the root directory of this source tree. + * + */ + +// @vitest-environment node + +// A line holding a link is made again as one plain text node with a word +// changed at each end (an agent rewriting a block: the link goes away), +// while a stale edit computed before the rewrite formats a range of it. The +// rewrite keeps the unchanged words between the changes as the same +// characters (#9621), as it keeps a line's unchanged start or end with one +// changed word (#9598), so a stale format whose range reaches past the link +// restates the link on them: #9597, the mechanism by which a link a +// collaborator removes comes back when another formats a range covering it +// at the same time. Those cells are expected failures until the fix for +// #9597 (#9601) lands. + +import {$createLinkNode, $isLinkNode, LinkExtension} from '@lexical/link'; +import {RichTextExtension} from '@lexical/rich-text'; +import * as Y from '@y/y'; +import { + $createParagraphNode, + $createRangeSelection, + $createTextNode, + $getRoot, + $setSelection, + type ElementNode, + type LexicalNode, + type TextFormatType, +} from 'lexical'; +import {expect, test} from 'vitest'; + +import { + createYDoc, + createYWriter, + editYState, + type HeadlessYConfig, + readYState, + seedYState, +} from '../../../headless'; + +const config: HeadlessYConfig = { + extensions: [RichTextExtension, LinkExtension], + onWarn: () => {}, + schemaVersion: 1, +}; + +const edit = (state: Uint8Array, who: string, fn: () => void) => + editYState(state, config, fn, {writer: createYWriter(who)}).update!; + +/** The words shown inside a link. */ +const linkedWords = (state: Uint8Array) => + readYState(state, config, () => { + const out: string[] = []; + for (const leaf of $getRoot().getAllTextNodes()) { + let node: LexicalNode | null = leaf.getParent(); + while (node !== null && !$isLinkNode(node)) { + node = node.getParent(); + } + if (node !== null) { + out.push(...leaf.getTextContent().split(/\s+/).filter(Boolean)); + } + } + return out; + }); + +/** Selects the characters of words `from` to `to` of the line. */ +function $selectWords(from: string, to: string) { + const block = $getRoot().getFirstChildOrThrow() as ElementNode; + const text = block.getTextContent(); + const start = text.indexOf(from); + const end = text.indexOf(to) + to.length; + const at = (offset: number, isStart: boolean): [string, number] => { + let acc = 0; + for (const leaf of block.getAllTextNodes()) { + const n = leaf.getTextContentSize(); + if (isStart ? offset < acc + n : offset <= acc + n) { + return [leaf.getKey(), offset - acc]; + } + acc += n; + } + throw new Error('no text node'); + }; + const selection = $createRangeSelection(); + selection.anchor.set(...at(start, true), 'text'); + selection.focus.set(...at(end, false), 'text'); + $setSelection(selection); + return selection; +} + +const WORDS = ['t23z', 't24z', 't25z', 't26z', 't27z']; + +for (const link of [['t25z', 't26z'], ['t24z', 't25z'], ['t25z']]) { + for (const format of ['bold', 'italic'] as TextFormatType[]) { + for (const range of [ + ['t24z', 't25z'], + ['t25z', 't25z'], + ['t23z', 't26z'], + ]) { + // The stale format reaches past the link (#9597). + const reachesPast = WORDS.slice( + WORDS.indexOf(range[0]), + WORDS.indexOf(range[1]) + 1, + ).some(w => !link.includes(w)); + for (const order of ['rewrite first', 'format first']) { + (reachesPast ? test.fails : test)( + `link on ${link.join(' ')}, a stale ${format} of ${range.join('..')}, ${order}: the removed link stays removed${reachesPast ? ' (#9597, fixed by #9601)' : ''}`, + () => { + const s0 = seedYState(config, () => { + const p = $createParagraphNode(); + const i = WORDS.indexOf(link[0]); + const j = WORDS.indexOf(link[link.length - 1]); + if (i > 0) { + p.append($createTextNode(WORDS.slice(0, i).join(' ') + ' ')); + } + p.append( + $createLinkNode('https://example.com').append( + $createTextNode(link.join(' ')), + ), + ); + if (j < WORDS.length - 1) { + p.append($createTextNode(' ' + WORDS.slice(j + 1).join(' '))); + } + $getRoot().append(p); + }); + const rewrite = edit(s0, 'rewrite', () => { + const block = $getRoot().getFirstChildOrThrow() as ElementNode; + const next = block + .getTextContent() + .replace('t23z', 't66z') + .replace('t27z', 't68z'); + for (const child of block.getChildren()) { + child.remove(); + } + block.append($createTextNode(next)); + }); + const stale = edit(s0, 'stale', () => { + $selectWords(range[0], range[1]).formatText(format); + }); + const doc = createYDoc(s0); + for (const u of order === 'rewrite first' + ? [rewrite, stale] + : [stale, rewrite]) { + Y.applyUpdate(doc, u); + } + const alone = createYDoc(s0); + Y.applyUpdate(alone, rewrite); + expect(linkedWords(Y.encodeStateAsUpdate(alone))).toEqual([]); + expect(linkedWords(Y.encodeStateAsUpdate(doc))).toEqual([]); + }, + ); + } + } + } +} diff --git a/packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts b/packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts new file mode 100644 index 00000000000..e1885ac50b2 --- /dev/null +++ b/packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts @@ -0,0 +1,531 @@ +/** + * Copyright (c) Meta Platforms, Inc. and affiliates. + * + * This source code is licensed under the MIT license found in the + * LICENSE file in the root directory of this source tree. + * + */ + +// @vitest-environment node + +// #9621: a line's text nodes made again with two or more words changed (an +// agent rewriting a block from its JSON) kept only the common start and end +// of the line's text (#9598) and wrote everything between the first and the +// last changed word again. A word a collaborator deleted there at the same +// time came back, and when the collaborator undid its delete the word showed +// twice. Every unchanged stretch of whole words between the changes now +// keeps its characters, so only the changed words are written. + +import { + $createParagraphNode, + $createTextNode, + $getRoot, + $isTextNode, + $parseSerializedNode, + $setSelection, + type ElementNode, + type SerializedTextNode, + type TextNode, +} from 'lexical'; +import {describe, expect, test} from 'vitest'; + +import { + expectConverged, + flush, + freshReader, + type Peer, + setup, +} from '../../utils/harness'; + +const $line = () => $getRoot().getChildAtIndex(1) as ElementNode; +const lineText = (p: Peer) => p.read(() => $line().getTextContent()); + +/** The line's text nodes made again from their JSON, with `changes`. */ +const $remake = (changes: [string, string][]) => () => { + const olds = $line().getChildren().filter($isTextNode) as TextNode[]; + const made = olds.map(o => { + const json = o.exportJSON() as SerializedTextNode; + let text = json.text; + for (const [from, to] of changes) { + text = text.replace(from, to); + } + return $parseSerializedNode({...json, text}); + }); + for (const o of olds) { + o.remove(); + } + $line().append(...made); + $setSelection(null); +}; + +/** + * `n` characters of the line from character `at` spliced with `insert`, + * whatever text nodes hold them. + */ +const $splice = + (at: number, n: number, insert = '') => + () => { + let acc = 0; + const spans: [TextNode, number, number][] = []; + for (const leaf of $line().getAllTextNodes()) { + const size = leaf.getTextContentSize(); + const s0 = Math.max(at, acc); + const s1 = Math.min(at + n, acc + size); + if (s1 > s0 || (n === 0 && at >= acc && at <= acc + size)) { + spans.push([leaf, s0 - acc, Math.max(0, s1 - s0)]); + if (n === 0) { + break; + } + } + acc += size; + } + spans.reverse().forEach(([leaf, offset, count], i) => { + leaf.spliceText( + offset, + count, + i === spans.length - 1 ? insert : '', + false, + ); + }); + $setSelection(null); + }; + +const IDS = [ + [1, 2], + [2, 1], +]; + +const LINE = 't1z t2z t3z t4z t5z t6z t7z'; + +describe('#9621: a delete between two words a remade line changes stays deleted', () => { + for (const ids of IDS) { + for (const [name, changes, expected] of [ + [ + 'two changes around it', + [ + ['t2z', 'n9z'], + ['t7z', 'n8z'], + ], + 't1z n9z t3z t4z t6z n8z', + ], + [ + 'three changes, one on each side and one next to it', + [ + ['t1z', 'n7z'], + ['t4z', 'n9z'], + ['t7z', 'n8z'], + ], + 'n7z t2z t3z n9z t6z n8z', + ], + ] as [string, [string, string][], string][]) { + for (const timing of ['concurrent', 'unread'] as const) { + test(`${name}: B deletes "t5z", ${timing} (ids ${ids})`, async () => { + const t = await setup( + () => { + $getRoot().append( + $createParagraphNode().append($createTextNode('Title')), + $createParagraphNode().append($createTextNode(LINE)), + ); + }, + {ids}, + ); + t.offline(); + if (timing === 'unread') { + t.b.update($splice(16, 3)); + t.a.editor.update($remake(changes)); + await t.sync(); + await flush(); + await t.sync(); + } else { + t.b.update($splice(16, 3)); + t.a.update($remake(changes)); + await t.sync(); + } + const fresh = await freshReader(t.a); + expect([...t.peers, fresh].map(lineText)).toEqual( + [...t.peers, fresh].map(() => expected), + ); + await expectConverged(t.peers); + expect(t.peers.flatMap(p => p.warnings)).toEqual([]); + }); + } + + for (const live of [false, true]) { + test(`${name}: B deletes "t5z" and undoes ${live ? 'after reading the remake' : 'concurrently'} (ids ${ids})`, async () => { + const t = await setup( + () => { + $getRoot().append( + $createParagraphNode().append($createTextNode('Title')), + $createParagraphNode().append($createTextNode(LINE)), + ); + }, + {ids}, + ); + t.offline(); + t.b.update($splice(16, 3)); + t.a.update($remake(changes)); + if (live) { + await t.sync(); + } + t.b.undo(); + await t.sync(); + const want = expected.replace(' ', ' t5z '); + const fresh = await freshReader(t.a); + expect([...t.peers, fresh].map(lineText)).toEqual( + [...t.peers, fresh].map(() => want), + ); + await expectConverged(t.peers); + expect(t.peers.flatMap(p => p.warnings)).toEqual([]); + }); + } + } + } +}); + +// Words of other shapes between the two changes: each character of a script +// written without spaces is a word of its own, an emoji is part of a word, +// and two runs of unchanged words with other white space between them stay +// two runs. +describe('#9621: unchanged words of other shapes between two changes keep a delete', () => { + for (const ids of IDS) { + for (const [name, line, changes, del, expected] of [ + [ + 'a script written without spaces', + '今日は晴れです明日は雨です', + [ + ['今', '昨'], + ['雨です', '雪でした'], + ], + '晴れ', + '昨日はです明日は雪でした', + ], + [ + 'emoji', + '😀a t1z 😁 t2z 😂b', + [ + ['😀a', '🙂a'], + ['😂b', '🙃b'], + ], + 't1z', + '🙂a 😁 t2z 🙃b', + ], + [ + 'runs with other white space between them', + 't1z t2z t3z\tt4z t5z', + [ + ['t1z', 'n1z'], + ['t5z', 'n5z'], + [' ', ' '], + ], + 't3z', + 'n1z t2z \tt4z n5z', + ], + ] as [string, string, [string, string][], string, string][]) { + test(`${name}: B deletes "${del}" (ids ${ids})`, async () => { + const t = await setup( + () => { + $getRoot().append( + $createParagraphNode().append($createTextNode('Title')), + $createParagraphNode().append($createTextNode(line)), + ); + }, + {ids}, + ); + const before = lineText(t.a); + t.offline(); + t.b.update($splice(before.indexOf(del), del.length)); + t.a.update($remake(changes)); + await t.sync(); + const fresh = await freshReader(t.a); + expect([...t.peers, fresh].map(lineText)).toEqual( + [...t.peers, fresh].map(() => expected), + ); + await expectConverged(t.peers); + expect(t.peers.flatMap(p => p.warnings)).toEqual([]); + }); + } + } +}); + +/** A seeded random integer below `n`. */ +function random(seed: number): (n: number) => number { + let s = seed >>> 0; + return n => { + s = (s + 0x6d2b79f5) >>> 0; + let x = Math.imul(s ^ (s >>> 15), s | 1); + x ^= x + Math.imul(x ^ (x >>> 7), x | 61); + return Math.floor((((x ^ (x >>> 14)) >>> 0) / 4294967296) * n); + }; +} + +/** `text` cut into 1 to `most` pieces at random places. */ +function cuts(rnd: (n: number) => number, text: string, most: number) { + const at = new Set(); + const count = rnd(most); + for (let i = 0; i < count && text.length > 1; i++) { + at.add(1 + rnd(text.length - 1)); + } + const out: string[] = []; + let prev = 0; + for (const c of [...[...at].sort((x, y) => x - y), text.length]) { + out.push(text.slice(prev, c)); + prev = c; + } + return out; +} + +const SEEDS = Number(process.env.LEXICAL_Y_SEEDS || 120); + +type Edit = 'delete' | 'type' | 'delete, undo' | 'type, undo' | 'bold'; +const EDITS: readonly Edit[] = [ + 'delete', + 'type', + 'delete, undo', + 'type, undo', + 'bold', +]; + +// The property: a line of unique words (one or two spaces or ", " between +// them) in 1 to 4 text nodes, some bold or italic, is made again as 1 to 3 +// plain nodes with 2 to 4 word edits (words replaced, inserted or deleted +// anywhere), while a collaborator edits a word no edit changes: deletes some +// of its characters, types a letter into it, either of them and then undoes +// it, or bolds it; as a concurrent edit or one the writer has not read yet, +// in both client id orders. Every client and a fresh reader agree, every +// word shows at most once, every word the rewrite made is whole, the +// collaborator's edit is where it was made (a delete stays deleted, an undone +// edit leaves the word as it was), and nothing warns. +describe('#9621: nothing a collaborator deleted comes back, nothing shows twice', () => { + for (const timing of ['unread', 'concurrent'] as const) { + for (const edit of EDITS) { + test(`${timing}: random lines made again with 2 to 4 word edits ‖ the peer ${edit}s an unchanged word`, async () => { + const failures: string[] = []; + for (let seed = 1; seed <= SEEDS; seed++) { + const rnd = random( + seed * 7919 + + EDITS.indexOf(edit) * 131 + + (timing === 'unread' ? 17 : 0), + ); + const count = 4 + rnd(8); + const numbers = new Set(); + while (numbers.size < count + 4) { + numbers.add(100 + rnd(900)); + } + // Three digits each, so a delete of digits never makes another + // word; some words end in punctuation, each keeps the separator + // before it. + const pool = [...numbers].map( + n => `t${n}z${['', '', '', ',', '.'][rnd(5)]}`, + ); + const words = pool.slice(0, count); + const fresh = pool.slice(count); + const sep = new Map(pool.map(w => [w, [' ', ' ', ' '][rnd(3)]])); + const join = (ws: string[]) => + ws.map((w, i) => (i === 0 ? w : sep.get(w) + w)).join(''); + const old = join(words); + let next = [...words]; + const edits = 2 + rnd(3); + for (let e = 0; e < edits; e++) { + const kind = rnd(3); + const at = rnd(next.length); + if (kind === 0 || next.length < 3) { + next[at] = fresh[e]; + } else if (kind === 1) { + next.splice(at, 0, fresh[e]); + } else { + next.splice(at, 1); + } + } + next = next.filter(Boolean); + const unchanged = words.filter(w => next.includes(w)); + if (unchanged.length === 0) { + continue; + } + const made = join(next); + const victim = unchanged[rnd(unchanged.length)]; + const vAt = old.indexOf(victim); + const from = 1 + rnd(3); + const len = 1 + rnd(4 - from); + const oldPieces = cuts(rnd, old, 4); + const formats = oldPieces.map(() => rnd(3)); + const ids = rnd(2) === 0 ? [1, 2] : [2, 1]; + const t = await setup( + () => { + $getRoot().append( + $createParagraphNode().append($createTextNode('Title')), + $createParagraphNode().append( + ...oldPieces.map((text, i) => { + const n = $createTextNode(text); + return formats[i] === 1 + ? n.toggleFormat('bold') + : formats[i] === 2 + ? n.toggleFormat('italic') + : n; + }), + ), + ); + }, + {ids}, + ); + const $remakeAll = () => { + const olds = $line().getChildren(); + for (const o of olds) { + o.remove(); + } + $line().append( + ...cuts(rnd, made, 3).map(text => $createTextNode(text)), + ); + $setSelection(null); + }; + const $peer = edit.startsWith('delete') + ? $splice(vAt + from, len) + : $splice(vAt + from, 0, 'Q'); + const $bold = () => { + // Bold the victim: split its text out and toggle the format. + let acc = 0; + for (const leaf of $line().getAllTextNodes()) { + const size = leaf.getTextContentSize(); + const s0 = Math.max(vAt, acc); + const s1 = Math.min(vAt + victim.length, acc + size); + if (s1 > s0) { + const parts = leaf.splitText(s0 - acc, s1 - acc); + const mid = s0 - acc === 0 ? parts[0] : parts[1]; + if (!mid.hasFormat('bold')) { + mid.toggleFormat('bold'); + } + } + acc += size; + } + $setSelection(null); + }; + const $edit = edit === 'bold' ? $bold : $peer; + t.offline(); + if (timing === 'unread') { + t.b.update($edit); + t.a.editor.update($remakeAll); + await t.sync(); + await flush(); + await t.sync(); + } else { + t.a.update($remakeAll); + t.b.update($edit); + await t.sync(); + } + if (edit.endsWith('undo')) { + t.b.undo(); + await t.sync(); + } + const reader = await freshReader(t.a); + const texts = [...t.peers, reader].map(lineText); + const got = texts[0].split(/ +/).filter(Boolean); + const problems: string[] = []; + if (new Set(texts).size > 1) { + problems.push(`diverged ${JSON.stringify(texts)}`); + } + const dups = got.filter((w, i) => got.indexOf(w) !== i); + if (dups.length > 0) { + problems.push(`twice: ${dups.join(' ')}`); + } + const edited = + edit === 'delete' + ? victim.slice(0, from) + victim.slice(from + len) + : edit === 'type' + ? victim.slice(0, from) + 'Q' + victim.slice(from) + : victim; + if (!got.includes(edited)) { + problems.push(`"${edited}" missing`); + } + if (edited !== victim && got.includes(victim)) { + problems.push(`"${victim}" came back`); + } + for (const w of next) { + if (w !== victim && !got.includes(w)) { + problems.push(`"${w}" not whole`); + } + } + // The rewrite makes plain text: a bold it keeps is the peer's, + // on a word that was plain (the rewrite did not change its + // format). + let acc = 0; + const wasPlain = oldPieces.every((piece, i) => { + const s0 = acc; + acc += piece.length; + return formats[i] === 0 || acc <= vAt || s0 >= vAt + victim.length; + }); + if (edit === 'bold' && wasPlain) { + const bold = t.a.read(() => + $line() + .getAllTextNodes() + .filter(n => n.hasFormat('bold')) + .map(n => n.getTextContent()) + .join(''), + ); + if (!bold.includes(victim)) { + problems.push( + `"${victim}" not bold (bold: ${JSON.stringify(bold)})`, + ); + } + } + const warnings = [...t.peers, reader].flatMap(p => p.warnings); + if (warnings.length > 0) { + problems.push(`warnings ${warnings.join('; ')}`); + } + if (problems.length > 0) { + failures.push( + `seed ${seed} ids ${ids} ${JSON.stringify(oldPieces)} -> ${JSON.stringify(made)}; ${edit} "${victim}" at ${from}+${len}: ${JSON.stringify(texts[0])}: ${problems.join(', ')}`, + ); + } + } + expect(failures).toEqual([]); + }, 240_000); + } + } +}); + +// The word-by-word diff of a remade line's changed middle stays linear in +// the line's length: two changes at its ends (the whole line between them +// diffed), and a change every fourth word (past the diff's bound of edits, +// where only the common start and end are kept). +describe('#9621: a long remade line stays linear', () => { + const timed = async (n: number, every: number) => { + const words = Array.from({length: n}, (_, i) => `t${i}z`); + const t = await setup(() => { + $getRoot().append( + $createParagraphNode().append($createTextNode('Title')), + $createParagraphNode().append($createTextNode(words.join(' '))), + ); + }); + const next = words.map((w, i) => + i % every === 0 || i === n - 1 ? `n${i}z` : w, + ); + const start = performance.now(); + t.a.update(() => { + for (const child of $line().getChildren()) { + child.remove(); + } + $line().append($createTextNode(next.join(' '))); + $setSelection(null); + }); + const ms = performance.now() - start; + expect(lineText(t.a)).toBe(next.join(' ')); + await t.sync(); + expect(lineText(t.b)).toBe(next.join(' ')); + return ms; + }; + for (const [name, every] of [ + ['changes at both ends', Infinity], + ['a change every fourth word', 4], + ] as const) { + test( + name, + async () => { + await timed(500, every); + const small = await timed(2_000, every); + const big = await timed(8_000, every); + // Linear is 4x, quadratic 16x. + expect(big / Math.max(small, 20)).toBeLessThan(10); + }, + 120_000, + ); + } +}); diff --git a/packages/lexical-y/src/internal/Inline.ts b/packages/lexical-y/src/internal/Inline.ts index 55a7053501b..55e86f9ecac 100644 --- a/packages/lexical-y/src/internal/Inline.ts +++ b/packages/lexical-y/src/internal/Inline.ts @@ -790,5 +790,203 @@ export function keptEnds( }; } +/** + * The most edits the diff of a remade line's changed middle looks for (see + * keptStretches): past it, only the common start and end are kept, so the + * diff stays linear in the line's length. + */ +const MAX_EDITS = 128; + +/** White space other than a line feed (a line break is a node of its own). */ +const isSeparator = (s: string, i: number): boolean => + s[i] !== '\n' && SPACE.test(s[i]); + +/** + * The words of `s` from `from` to `to`, as `[start, end)` pairs: runs of + * word characters (see isWordChar), and single other characters that are not + * white space (an object character, a line feed, a character of a script + * written without spaces), each a whole character. White space between them + * is no word: it is compared only between two kept words. + */ +function words(s: string, from: number, to: number): [number, number][] { + const out: [number, number][] = []; + let i = from; + while (i < to) { + let j = i + 1; + if (isWordChar(s, i)) { + while (j < to && isWordChar(s, j)) { + j++; + } + } else if (j < to && isLowSurrogate(s.charCodeAt(j))) { + j++; + } + if (!isSeparator(s, i)) { + out.push([i, j]); + } + i = j; + } + return out; +} + +/** + * The tokens equal in `a` and `b` that a shortest edit script keeps, as + * pairs of indexes, in order: Myers' diff, in O((n + m) * max) time, or + * null past `max` edits. + */ +function commonTokens( + a: readonly number[], + b: readonly number[], + max: number, +): [number, number][] | null { + const n = a.length; + const m = b.length; + const off = max + 1; + const v = new Int32Array(2 * off + 1); + const trace: Int32Array[] = []; + for (let d = 0; d <= max; d++) { + trace.push(v.slice()); + for (let k = -d; k <= d; k += 2) { + let x = + k === -d || (k !== d && v[off + k - 1] < v[off + k + 1]) + ? v[off + k + 1] + : v[off + k - 1] + 1; + let y = x - k; + while (x < n && y < m && a[x] === b[y]) { + x++; + y++; + } + v[off + k] = x; + if (x >= n && y >= m) { + // Walk back through the trace, collecting the diagonals. + const pairs: [number, number][] = []; + let cx = n; + let cy = m; + for (let e = d; e > 0; e--) { + const pv = trace[e]; + const ck = cx - cy; + const pk = + ck === -e || (ck !== e && pv[off + ck - 1] < pv[off + ck + 1]) + ? ck + 1 + : ck - 1; + const px = pv[off + pk]; + const py = px - pk; + const sx = pk === ck + 1 ? px : px + 1; + const sy = sx - ck; + while (cx > sx && cy > sy) { + pairs.push([--cx, --cy]); + } + cx = px; + cy = py; + } + while (cx > 0 && cy > 0) { + pairs.push([--cx, --cy]); + } + return pairs.reverse(); + } + } + } + return null; +} + +/** + * What text made in place of removed text (`b`, see TextReplay#stretch) + * keeps of that text (`a`), as ranges `[x0, x1, dx]` of `b` whose character + * `x` was character `x + dx` of `a`, in order: the common start and end + * (keptEnds), and between them, where `b` replaced text, every run of whole + * words that a shortest word-by-word edit script of the rest keeps, with the + * white space around it that both texts share (`t1z t2z t3z t4z` made again + * as `t1z n9z t3z n8z` keeps ` t3z `), so that only the changed words are + * written: a character a collaborator deletes in an unchanged word between + * two changed ones stays deleted, and a letter it types there stays in that + * word. White space alone is never kept between two changes, as at the ends. + * Past MAX_EDITS word edits, only the common start and end. + */ +export function keptStretches( + a: string, + b: string, + cursor: number, + composing: boolean, +): [number, number, number][] { + const {prefix, suffix} = keptEnds(a, b, cursor, composing); + const shift = a.length - b.length; + const ends: [number, number, number][] = [ + [0, prefix, 0], + [b.length - suffix, b.length, shift], + ]; + const a1 = a.length - suffix; + const b1 = b.length - suffix; + if (composing || a1 <= prefix || b1 <= prefix) { + return ends; + } + const wa = words(a, prefix, a1); + const wb = words(b, prefix, b1); + const ids = new Map(); + const id = (text: string, [x0, x1]: [number, number]): number => { + const t = text.slice(x0, x1); + let x = ids.get(t); + if (x === undefined) { + x = ids.size; + ids.set(t, x); + } + return x; + }; + const pairs = commonTokens( + wa.map(w => id(a, w)), + wb.map(w => id(b, w)), + MAX_EDITS, + ); + if (pairs === null) { + return ends; + } + const out: [number, number, number][] = [ends[0]]; + let lastA = prefix; + let lastB = prefix; + for (let i = 0; i < pairs.length; ) { + // Words kept one after the other with the same white space between + // them in both texts are one stretch. + let j = i + 1; + while ( + j < pairs.length && + pairs[j][0] === pairs[j - 1][0] + 1 && + pairs[j][1] === pairs[j - 1][1] + 1 && + a.slice(wa[pairs[j - 1][0]][1], wa[pairs[j][0]][0]) === + b.slice(wb[pairs[j - 1][1]][1], wb[pairs[j][1]][0]) + ) { + j++; + } + let ax = wa[pairs[i][0]][0]; + let bx = wb[pairs[i][1]][0]; + let bEnd = wb[pairs[j - 1][1]][1]; + let aEnd = ax + bEnd - bx; + // With the white space around it that is the same in both texts (up to + // the stretch before it), as the start and end keep theirs: only the + // changed words are written. + while ( + ax > lastA && + bx > lastB && + a.charCodeAt(ax - 1) === b.charCodeAt(bx - 1) && + isSeparator(a, ax - 1) + ) { + ax--; + bx--; + } + while ( + aEnd < a1 && + bEnd < b1 && + a.charCodeAt(aEnd) === b.charCodeAt(bEnd) && + isSeparator(a, aEnd) + ) { + aEnd++; + bEnd++; + } + out.push([bx, bEnd, ax - bx]); + lastA = aEnd; + lastB = bEnd; + i = j; + } + out.push(ends[1]); + return out; +} + const isHighSurrogate = (c: number): boolean => c >= 0xd800 && c <= 0xdbff; const isLowSurrogate = (c: number): boolean => c >= 0xdc00 && c <= 0xdfff; diff --git a/packages/lexical-y/src/internal/TextEdits.ts b/packages/lexical-y/src/internal/TextEdits.ts index 313d8a68329..1bcdb315881 100644 --- a/packages/lexical-y/src/internal/TextEdits.ts +++ b/packages/lexical-y/src/internal/TextEdits.ts @@ -19,7 +19,7 @@ import { type NodeMap, } from 'lexical'; -import {diffStrings, keptEnds, snapToWords} from './Inline'; +import {diffStrings, keptStretches, snapToWords} from './Inline'; import {$isWrapper} from './Schema'; const __DEV__ = process.env.NODE_ENV !== 'production'; @@ -1771,27 +1771,26 @@ export class TextReplay { continue; } const w = d.slice(o0, o1); - let prefix: number; - let suffix: number; + // What it keeps, as ranges of `text` and how far on in `w` each is. + const tl = text.length; + const shift = w.length - tl; + let keep: [number, number, number][]; if (made) { - ({prefix, suffix} = keptEnds(w, text, caret, this.composing)); + keep = keptStretches(w, text, caret, this.composing); } else { const diff = diffStrings(w, text, caret); - ({prefix, suffix} = snapToWords( + const {prefix, suffix} = snapToWords( w, text, diff.prefix, diff.suffix, this.composing, - )); + ); + keep = [ + [0, prefix, 0], + [tl - suffix, tl, shift], + ]; } - // What it keeps, as ranges of `text` and how far on in `w` each is. - const tl = text.length; - const shift = w.length - tl; - let keep: [number, number, number][] = [ - [0, prefix, 0], - [tl - suffix, tl, shift], - ]; if (sure.length > 0) { // Only what the diff that takes the common end first keeps too (see // `sure`): in `w`, and where text that left was deleted whole in From 598c3c19ed58b5e831bb2eab9768961c0ec1edc0 Mon Sep 17 00:00:00 2001 From: Gerard Rovira Date: Fri, 30 Oct 2026 16:31:45 -0700 Subject: [PATCH 2/4] [lexical-website][lexical-y] Bug Fix: Keep unchanged words past the edit-script bound, keep remade lines linear in their kept runs, and read an old line's inline nodes in the old state MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Description Follow-up to the first commit, from review: - Past 128 word edits, the diff of a remade line kept only its common start and end, so a collaborator's concurrent edit of an unchanged word came back, showed twice after an undo, landed at the start of the line or lost its format, as before this pull request. Now the words found once in both texts anchor the rest (the longest run of them in the same order, patience diff), and each stretch between two of them gets its own bounded edit script. Linear in the line's length up to a log factor. - A rewrite that changes only the white space between the words of a line of one alternately formatted text node per word made a kept run per word and a node per run, and two loops scanned every kept run per node (O(n^2), 5 s at 4,000 words). The loop over kept runs keeps a cursor, and runs are found by binary search. - Remaking a line holding an inline node whose class reads its latest version in `isInline()` (the playground's `EquationNode`) threw "Lexical node does not exist in active editor state", on main too: the old line's layout asked the removed node in the new state. It is now read in the previous state. ## Changeset `.changeset/lexical-y-remade-line-inline-node.md` for the throw; the cap and the cost are part of the first changeset's fix. ## Test plan ### Before On the first commit (225cdc62): ``` $ vitest --run --project unit packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts -t 'one edit script covers|linear' × 65 words replaced ‖ the peer deletes an unchanged word (ids 1,2) ... (all 8 cells for 65 and for 200 words) × the white space between every two words, one node per word Tests 17 failed | 10 passed | 32 skipped (59) ``` On main, the catalog inline shapes: ``` $ vitest --run --project unit packages/lexical-y/src/__tests__/unit/matrix/C9RemadeText.test.ts -t 'keeps the shape' × equation/inline Error: Lexical node does not exist in active editor state (key 17, type equation). ❯ EquationNode.isInline packages/lexical-playground/src/nodes/EquationNode.tsx:142:17 ❯ kindOf packages/lexical-y/src/internal/TextEdits.ts:203:42 ❯ TextReplay.oldLayout packages/lexical-y/src/internal/TextEdits.ts:1200:5 Tests 1 failed | 22 passed | 564 skipped (587) ``` ### After ``` $ vitest --run --project unit packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStaleLink.test.ts packages/lexical-y/src/__tests__/unit/matrix/C9RemadeText.test.ts ✓ unit packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStaleLink.test.ts (36 tests) ✓ unit packages/lexical-y/src/__tests__/unit/matrix/C9RemadeText.test.ts (587 tests) ✓ unit packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts (59 tests) Tests 662 passed | 20 expected fail (682) $ pnpm run ci-check EXIT 0 $ vitest --run --project unit --project scripts-unit Test Files 637 passed | 4 skipped (641) Tests 26051 passed | 210 expected fail | 135 skipped (26396) $ FUZZ_SEED_START=1 FUZZ_SEEDS=3000 vitest --run --project unit packages/lexical-y/src/__tests__/unit/Fuzz.test.ts Tests 1 failed | 123 passed (124) AssertionError: expected [ 'seed 2277: dup' ] to deeply equal [] $ FUZZ_SEED_START=50000 FUZZ_SEEDS=2000 vitest --run --project unit packages/lexical-y/src/__tests__/unit/Fuzz.test.ts Tests 7 failed | 77 passed (84) (seeds 50180, 50231, 50372, 51439, 51507, 51972, 51976, as on main) ``` Whitespace-only remake of one alternately formatted node per word, 500 / 1,000 / 2,000 / 4,000 words: 34 / 62 / 107 / 216 ms (the first commit: 114 / 353 / 1,258 / 5,162 ms; main: 70 / 70 / 108 / 210 ms). --- .../lexical-y-remade-line-inline-node.md | 7 + .../lexical-website/docs/collaboration/y.md | 10 +- packages/lexical-y/DECISIONS.md | 2 +- .../unit/matrix/C9RemadeText.test.ts | 40 +++++ .../regressions/RemadeTextStretches.test.ts | 145 +++++++++++++++--- packages/lexical-y/src/internal/Inline.ts | 113 ++++++++++++-- packages/lexical-y/src/internal/TextEdits.ts | 52 +++++-- packages/lexical-y/src/internal/Write.ts | 2 +- 8 files changed, 328 insertions(+), 43 deletions(-) create mode 100644 .changeset/lexical-y-remade-line-inline-node.md diff --git a/.changeset/lexical-y-remade-line-inline-node.md b/.changeset/lexical-y-remade-line-inline-node.md new file mode 100644 index 00000000000..b4ed4d65f22 --- /dev/null +++ b/.changeset/lexical-y-remade-line-inline-node.md @@ -0,0 +1,7 @@ +--- +'@lexical/y': patch +--- + +[fix] Stop throwing when an update makes again the nodes of a line holding an inline node whose `isInline()` reads the node's latest version + +Verify: removing a line's children and appending new ones made from their JSON, in a bound editor, where the line held an inline decorator or element whose class computes `isInline()` from `getLatest()` (as the playground's `EquationNode` does), writes the new line instead of throwing that the node does not exist in the active editor state. diff --git a/packages/lexical-website/docs/collaboration/y.md b/packages/lexical-website/docs/collaboration/y.md index efd71478179..85ba673e5d2 100644 --- a/packages/lexical-website/docs/collaboration/y.md +++ b/packages/lexical-website/docs/collaboration/y.md @@ -817,9 +817,10 @@ person's: it was a node of its own (a tab, a line break, a node of spaces). A word joined to a changed word through an inline node, with no space between, is written again with it, and an inline node made between two changes is new - where it is put. Where a line has more than 128 changed words, only its - common start and end are kept, so the comparison stays linear in the - line's length. Prefer + where it is put. The comparison stays linear in the line's length: where + a line has more than 128 changed words, the unchanged words found once in + both texts anchor it, so a word that occurs more than once in the line + and sits between such a word and a change may be written again. Prefer these to `setTextContent`, which sets a node's text whole: such a change (and a change the browser makes to the DOM text, as an IME composition does) keeps the start and end its text shares with the node's old text, @@ -1010,7 +1011,8 @@ not merge as a user might expect: new word, not in it (a letter typed at the end of the word ends up in front of the new word), and the collaborator's formatting of the changed word at the same time is not kept. In a line with more than 128 changed - words, everything between the first and the last change is written again. + words, an unchanged word that occurs more than once in the line may be + written again. The concurrent typed text is kept, and every client shows the same result. Edit the changed words in place instead (`TextNode#spliceText`, the selection's `insertText`): only the changed diff --git a/packages/lexical-y/DECISIONS.md b/packages/lexical-y/DECISIONS.md index 6dbfa56d825..3163438a335 100644 --- a/packages/lexical-y/DECISIONS.md +++ b/packages/lexical-y/DECISIONS.md @@ -60,7 +60,7 @@ issues and docs that carry it. | D26 | **Undo favours people taking turns over concurrent races.** Where a fix makes undo correct for sequential edits at the cost of rare races between an undo and a simultaneous edit of the same text, the sequential case wins. | Taking turns is the common case; a race with a simultaneous edit of the same text is rare. Picked on 10-28, confirmed on 10-30. | `DECIDED` | 10-30 | [#9400] | | D27 | **Moves are a delete and a re-insert** at launch; the binding rewrites the cheaper side. Real moves within a container ([#9449], records after Yjs's unmerged `ContentMove` design) are paused and land last, backward compatible: a new layout version that older clients open read only. Move records leave tombstones, as every Yjs binding does. Moves across containers stay copies. | Every Yjs binding turns moves into delete plus insert; Yjs dropped its own move for complexity. Reliability of the rest came first. | `DECIDED` | 10-25 | [#9449], [#9455], [y.md: Limits][limits] | | D28 | **Text identity is fixed at the source.** While a binding is attached, core records text edits (`spliceText`, insert and remove, split and merge), and text entity plugins record a grow as a move and a remake as a retype. Text only `setTextContent` or the DOM changed is diffed. The binding adds no per-case heuristics. | No binding can recover intent an API dropped; special-case rules were fragile and regressed neighbouring cases. | `DECIDED` | 10-27 | [#9483], [y.md: Edits made by code][y-code] | -| D29 | **Remade text keeps its common start and end by character**, whatever the node boundaries (y-prosemirror's `updateYText` rule), **and between two changes every run of unchanged whole words** (a word-by-word shortest edit script, bounded to 128 word edits per line so it stays linear; past it, only the start and end), and never brings back a collaborator's delete. It replaces the word matching of [#9565]. | It is the industry-standard behaviour, and word matching glued words together and merged numbers oddly. Keeping only the start and end wrote everything between two changed words again, so a collaborator's delete there came back, and showed twice after the collaborator's undo ([#9621]). | `DECIDED` | 10-30 | [#9565], [#9598], [#9621] | +| D29 | **Remade text keeps its common start and end by character**, whatever the node boundaries (y-prosemirror's `updateYText` rule), **and between two changes every run of unchanged whole words** (a word-by-word shortest edit script; past 128 word edits in a line, the words found once in both texts anchor it, so it stays linear), and never brings back a collaborator's delete. It replaces the word matching of [#9565]. | It is the industry-standard behaviour, and word matching glued words together and merged numbers oddly. Keeping only the start and end wrote everything between two changed words again, so a collaborator's delete there came back, and showed twice after the collaborator's undo ([#9621]). | `DECIDED` | 10-30 | [#9565], [#9598], [#9621] | | D30 | **Diffed rewrites snap to whole words**; keystrokes and IME composition keep the exact diff. Scripts without spaces treat each character as a boundary. | Concurrent rewrites otherwise split words. Real word segmentation is a follow-up ([#9467]). | `DECIDED` | 10-26 | [#9400], [#9467] | | D31 | **A list item's nested list gets an item of its own.** `NodeStructureExtension`'s list rule prevents an item holding a nested list next to other content when it is written; the binding does not repair it afterwards. | That shape lost text in outdents, Backspaces and offline merges. Preventing it at write time follows D20. | `DECIDED` | 10-26 | [#9452], [y.md: Limits][limits] | | D32 | **List merging ignores `start`.** | Existing Lexical behaviour, intended. | `DECIDED` | 10-23 | [#9452] | diff --git a/packages/lexical-y/src/__tests__/unit/matrix/C9RemadeText.test.ts b/packages/lexical-y/src/__tests__/unit/matrix/C9RemadeText.test.ts index d8ef2d244ae..2e635bd3055 100644 --- a/packages/lexical-y/src/__tests__/unit/matrix/C9RemadeText.test.ts +++ b/packages/lexical-y/src/__tests__/unit/matrix/C9RemadeText.test.ts @@ -50,6 +50,7 @@ import { import {describe, expect, test} from 'vitest'; import {createYDoc, YCollabExtension} from '../../..'; +import {CATALOG_NODES, NODE_SHAPES} from '../../catalog/nodes'; import { $createEquation, expectConverged, @@ -440,3 +441,42 @@ describe('C9: a line made again from its JSON with a word changed on each side k } } }); + +// Every inline shape of the catalog (`catalog/nodes.ts`), its nodes made +// again from their JSON with the words around it changed: the shape comes +// back as it was, the same on every client. Reading the old line's layout +// asked a removed node whether it is inline in the new state, which a class +// that reads its latest version there (the playground's EquationNode) +// refused with a throw. +describe('C9: a line made again from its JSON around each catalog inline shape keeps the shape', () => { + const $json = (n: LexicalNode): SerializedLexicalNode => $deepJSON(n); + const CHANGES: [string, string][] = [ + ['t1z', 't8z'], + ['t4z', 't9z'], + ]; + for (const shape of NODE_SHAPES.filter(s => s.$inline !== undefined)) { + test(shape.name, async () => { + const t = await setup( + () => { + $getRoot().append( + $createParagraphNode().append($t('Title')), + $createParagraphNode().append( + $t('t1z t2z '), + ...shape.$inline!(), + $t(' t3z t4z'), + ), + ); + }, + {nodes: [...CATALOG_NODES]}, + ); + const json = (p: Peer) => p.read(() => $line().getChildren().map($json)); + const expected = json(t.a).map(j => changed(j, CHANGES)); + t.a.update(() => $rewriteAll(CHANGES)); + await t.sync(); + expect(json(t.a)).toEqual(expected); + expect(json(t.b)).toEqual(expected); + await expectConverged(t.peers); + expect(t.peers.flatMap(p => p.warnings)).toEqual([]); + }); + } +}); diff --git a/packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts b/packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts index e1885ac50b2..64e9a672899 100644 --- a/packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts +++ b/packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts @@ -482,46 +482,155 @@ describe('#9621: nothing a collaborator deleted comes back, nothing shows twice' } }); -// The word-by-word diff of a remade line's changed middle stays linear in -// the line's length: two changes at its ends (the whole line between them -// diffed), and a change every fourth word (past the diff's bound of edits, -// where only the common start and end are kept). +// Past the bound on one shortest edit script (more than 64 words replaced, +// each a delete and an insert), the words found once in both texts anchor +// the diff: a collaborator's concurrent edit of an unchanged word is kept as +// below the bound. +describe('#9621: an edit between more changed words than one edit script covers', () => { + for (const changes of [64, 65, 200]) { + for (const ids of IDS) { + for (const edit of ['delete', 'delete, undo', 'type', 'bold'] as const) { + test(`${changes} words replaced ‖ the peer ${edit}s an unchanged word (ids ${ids})`, async () => { + const words = Array.from( + {length: 2 * changes + 1}, + (_, i) => `w${i}z`, + ); + const next = words + .map((w, i) => (i % 2 === 0 && i < 2 * changes ? `r${i}z` : w)) + .join(' '); + const old = words.join(' '); + const victim = words[3]; + const at = old.indexOf(victim); + const t = await setup( + () => { + $getRoot().append( + $createParagraphNode().append($createTextNode('Title')), + $createParagraphNode().append($createTextNode(old)), + ); + }, + {ids}, + ); + t.offline(); + if (edit === 'bold') { + t.b.update(() => { + const leaf = $line().getFirstChildOrThrow() as TextNode; + const [, mid] = leaf.splitText(at, at + victim.length); + mid.toggleFormat('bold'); + $setSelection(null); + }); + } else if (edit === 'type') { + t.b.update($splice(at + 1, 0, 'Q')); + } else { + t.b.update($splice(at, victim.length)); + } + t.a.update(() => { + for (const child of $line().getChildren()) { + child.remove(); + } + $line().append($createTextNode(next)); + $setSelection(null); + }); + await t.sync(); + if (edit === 'delete, undo') { + t.b.undo(); + await t.sync(); + } + const expected = + edit === 'delete' + ? next.replace(victim, '') + : edit === 'type' + ? next.replace(victim, 'wQ3z') + : next; + const fresh = await freshReader(t.a); + expect([...t.peers, fresh].map(lineText)).toEqual( + [...t.peers, fresh].map(() => expected), + ); + if (edit === 'bold') { + expect( + t.a.read(() => + $line() + .getAllTextNodes() + .filter(n => n.hasFormat('bold')) + .map(n => n.getTextContent()), + ), + ).toEqual([victim]); + } + await expectConverged(t.peers); + expect(t.peers.flatMap(p => p.warnings)).toEqual([]); + }); + } + } + } +}); + +// The diff of a remade line stays linear in the line's length: two changes +// at its ends (the whole line between them diffed), a change every fourth +// word (past one edit script's bound, where unique words anchor it), and +// the white space between every two words changed in a line of one +// alternately formatted node per word (a kept run per word, a node per run). describe('#9621: a long remade line stays linear', () => { - const timed = async (n: number, every: number) => { + const timed = async ( + n: number, + oldNodes: (words: string[]) => TextNode[], + newNodes: (words: string[]) => TextNode[], + expected: (words: string[]) => string, + ) => { const words = Array.from({length: n}, (_, i) => `t${i}z`); const t = await setup(() => { $getRoot().append( $createParagraphNode().append($createTextNode('Title')), - $createParagraphNode().append($createTextNode(words.join(' '))), + $createParagraphNode().append(...oldNodes(words)), ); }); - const next = words.map((w, i) => - i % every === 0 || i === n - 1 ? `n${i}z` : w, - ); const start = performance.now(); t.a.update(() => { for (const child of $line().getChildren()) { child.remove(); } - $line().append($createTextNode(next.join(' '))); + $line().append(...newNodes(words)); $setSelection(null); }); const ms = performance.now() - start; - expect(lineText(t.a)).toBe(next.join(' ')); + expect(lineText(t.a)).toBe(expected(words)); await t.sync(); - expect(lineText(t.b)).toBe(next.join(' ')); + expect(lineText(t.b)).toBe(expected(words)); return ms; }; - for (const [name, every] of [ - ['changes at both ends', Infinity], - ['a change every fourth word', 4], + const changed = (every: number) => (words: string[]) => + words + .map((w, i) => (i % every === 0 || i === words.length - 1 ? `n${i}z` : w)) + .join(' '); + const perWord = (sep: string) => (words: string[]) => + words.map((w, i) => { + const node = $createTextNode(w + (i === words.length - 1 ? '' : sep)); + return i % 2 === 1 ? node.toggleFormat('bold') : node; + }); + for (const [name, oldNodes, newNodes, expected] of [ + [ + 'changes at both ends', + (words: string[]) => [$createTextNode(words.join(' '))], + (words: string[]) => [$createTextNode(changed(Infinity)(words))], + changed(Infinity), + ], + [ + 'a change every fourth word', + (words: string[]) => [$createTextNode(words.join(' '))], + (words: string[]) => [$createTextNode(changed(4)(words))], + changed(4), + ], + [ + 'the white space between every two words, one node per word', + perWord(' '), + perWord(' '), + (words: string[]) => words.join(' '), + ], ] as const) { test( name, async () => { - await timed(500, every); - const small = await timed(2_000, every); - const big = await timed(8_000, every); + await timed(500, oldNodes, newNodes, expected); + const small = await timed(1_000, oldNodes, newNodes, expected); + const big = await timed(4_000, oldNodes, newNodes, expected); // Linear is 4x, quadratic 16x. expect(big / Math.max(small, 20)).toBeLessThan(10); }, diff --git a/packages/lexical-y/src/internal/Inline.ts b/packages/lexical-y/src/internal/Inline.ts index 55e86f9ecac..d4360d3ae79 100644 --- a/packages/lexical-y/src/internal/Inline.ts +++ b/packages/lexical-y/src/internal/Inline.ts @@ -791,9 +791,9 @@ export function keptEnds( } /** - * The most edits the diff of a remade line's changed middle looks for (see - * keptStretches): past it, only the common start and end are kept, so the - * diff stays linear in the line's length. + * The most edits one shortest edit script of a remade line's changed words + * looks for (see commonWords): past it, the words unique to both texts + * anchor the rest, so the diff stays linear in the line's length. */ const MAX_EDITS = 128; @@ -888,6 +888,106 @@ function commonTokens( return null; } +/** + * The words equal in `a` and `b` that are kept, as pairs of indexes, in + * order: their common start and end, then a shortest edit script of the rest + * (commonTokens) where it takes at most MAX_EDITS edits; past that, the + * words that occur once in each text, in the longest run of them in the + * same order in both (patience diff), and the same between each two of + * those, once (between them, past MAX_EDITS edits, only their common start + * and end). O((n + m) * MAX_EDITS + n log n). + */ +function commonWords( + a: readonly number[], + b: readonly number[], + anchored = false, +): [number, number][] { + const out: [number, number][] = []; + let a0 = 0; + let b0 = 0; + let a1 = a.length; + let b1 = b.length; + while (a0 < a1 && b0 < b1 && a[a0] === b[b0]) { + out.push([a0++, b0++]); + } + const tail: [number, number][] = []; + while (a1 > a0 && b1 > b0 && a[a1 - 1] === b[b1 - 1]) { + tail.push([--a1, --b1]); + } + if (a0 < a1 && b0 < b1) { + const pairs = commonTokens(a.slice(a0, a1), b.slice(b0, b1), MAX_EDITS); + if (pairs !== null) { + for (const [x, y] of pairs) { + out.push([x + a0, y + b0]); + } + } else if (!anchored) { + // Words that occur once in each text. + const count = new Map(); + const atA = new Map(); + for (let x = a0; x < a1; x++) { + count.set(a[x], (count.get(a[x]) ?? 0) + 1); + atA.set(a[x], x); + } + const inB = new Map(); + const atB = new Map(); + for (let y = b0; y < b1; y++) { + inB.set(b[y], (inB.get(b[y]) ?? 0) + 1); + atB.set(b[y], y); + } + const unique: [number, number][] = []; + for (let y = b0; y < b1; y++) { + if (count.get(b[y]) === 1 && inB.get(b[y]) === 1) { + unique.push([atA.get(b[y])!, atB.get(b[y])!]); + } + } + // The longest run of them in the same order in both (in order of + // `b`, increasing in `a`): patience sorting. + const tops: number[] = []; + const prev = new Int32Array(unique.length).fill(-1); + for (let i = 0; i < unique.length; i++) { + let lo = 0; + let hi = tops.length; + while (lo < hi) { + const mid = (lo + hi) >> 1; + if (unique[tops[mid]][0] < unique[i][0]) { + lo = mid + 1; + } else { + hi = mid; + } + } + prev[i] = lo > 0 ? tops[lo - 1] : -1; + tops[lo] = i; + } + const anchors: [number, number][] = []; + for (let i = tops.length > 0 ? tops[tops.length - 1] : -1; i >= 0; ) { + anchors.push(unique[i]); + i = prev[i]; + } + anchors.reverse(); + let x = a0; + let y = b0; + for (const [ax, by] of [...anchors, [a1, b1] as [number, number]]) { + for (const [px, py] of commonWords( + a.slice(x, ax), + b.slice(y, by), + true, + )) { + out.push([px + x, py + y]); + } + if (ax < a1) { + out.push([ax, by]); + } + x = ax + 1; + y = by + 1; + } + } + } + for (let i = tail.length - 1; i >= 0; i--) { + out.push(tail[i]); + } + return out; +} + /** * What text made in place of removed text (`b`, see TextReplay#stretch) * keeps of that text (`a`), as ranges `[x0, x1, dx]` of `b` whose character @@ -899,7 +999,6 @@ function commonTokens( * written: a character a collaborator deletes in an unchanged word between * two changed ones stays deleted, and a letter it types there stays in that * word. White space alone is never kept between two changes, as at the ends. - * Past MAX_EDITS word edits, only the common start and end. */ export function keptStretches( a: string, @@ -930,14 +1029,10 @@ export function keptStretches( } return x; }; - const pairs = commonTokens( + const pairs = commonWords( wa.map(w => id(a, w)), wb.map(w => id(b, w)), - MAX_EDITS, ); - if (pairs === null) { - return ends; - } const out: [number, number, number][] = [ends[0]]; let lastA = prefix; let lastB = prefix; diff --git a/packages/lexical-y/src/internal/TextEdits.ts b/packages/lexical-y/src/internal/TextEdits.ts index 1bcdb315881..68132b1f399 100644 --- a/packages/lexical-y/src/internal/TextEdits.ts +++ b/packages/lexical-y/src/internal/TextEdits.ts @@ -384,13 +384,18 @@ export class TextReplay { /** The moves found unsafe in this replay (see the move entry). */ private restart: number[] | null = null; + /** The previous state's nodes, by key. */ + private readonly prevMap: NodeMap; + constructor( log: TextEditLog | null, - private readonly prevMap: NodeMap, + /** The state before this update. */ + private readonly prevState: EditorState, /** The caret of the new state, where it is in a text node. */ private readonly caret: {key: NodeKey; offset: number} | null, private readonly composing: boolean, ) { + this.prevMap = prevState._nodeMap; if (log !== null) { // Where a text set reaches text whose history a move split or joined, // or a removed node still holds such text at the end (set text aligns @@ -856,7 +861,17 @@ export class TextReplay { /** A node's layout, from the previous state, else from the new one. */ private kindNow(key: NodeKey): Kind | null { - return kindOf(this.prevMap.get(key) ?? $getNodeByKey(key)); + const old = this.prevMap.get(key); + return old !== undefined ? this.oldKind(old) : kindOf($getNodeByKey(key)); + } + + /** + * The layout of a node of the previous state, read in that state: a class + * may compute `isInline()` from its latest version, which a node this + * update removed no longer has. + */ + private oldKind(node: LexicalNode | undefined): Kind | null { + return node === undefined ? null : this.prevState.read(() => kindOf(node)); } /** An old text node's runs: its whole old text. Null if not a text node. */ @@ -1018,7 +1033,7 @@ export class TextReplay { /** The one character of a line break or inline node `key`, as it is. */ private atom(key: NodeKey, kind: Kind | null): Run { - const old = kindOf(this.prevMap.get(key)); + const old = this.oldKind(this.prevMap.get(key)); if (old !== null && old === kind) { return kind === 'break' ? {base: '\n', end: 1, kind: OLD, src: key, start: 0} @@ -1185,7 +1200,7 @@ export class TextReplay { if (node === undefined) { return; } - const kind = kindOf(node); + const kind = this.oldKind(node); out.set(c, pos); if (kind === 'text') { pos += (node as LexicalNode & {__text: string}).__text.length; @@ -1563,12 +1578,23 @@ export class TextReplay { runs: readonly [number, number, number][], ): Block[] => { const out: Block[] = []; - let r = 0; for (const b of blocks) { - while (r < runs.length && runs[r][0] + runs[r][2] <= b.o) { - r++; + // The first run that ends after the block starts. + let first = 0; + let last = runs.length; + while (first < last) { + const mid = (first + last) >> 1; + if (runs[mid][0] + runs[mid][2] <= b.o) { + first = mid + 1; + } else { + last = mid; + } } - for (let k = r; k < runs.length && runs[k][0] < b.o + b.n; k++) { + for ( + let k = first; + k < runs.length && runs[k][0] < b.o + b.n; + k++ + ) { const [o0, p0, n] = runs[k]; const lo = Math.max(b.o, o0); const hi = Math.min(b.o + b.n, o0 + n); @@ -1866,11 +1892,17 @@ export class TextReplay { } } } - for (let j = 0, t = 0; j < parts.length; j += 3) { + // `keep` is in order of where its ranges start: a cursor past the + // ranges that end before each part keeps this linear. + for (let j = 0, t = 0, kp = 0; j < parts.length; j += 3) { const p0 = parts[j]; const n = parts[j + 1] - p0; + while (kp < keep.length && keep[kp][1] <= t) { + kp++; + } if (keeps(spans[parts[j + 2]])) { - for (const [x0, x1, dx] of keep) { + for (let q = kp; q < keep.length && keep[q][0] < t + n; q++) { + const [x0, x1, dx] = keep[q]; const lo = Math.max(t, x0); const hi = Math.min(t + n, x1); if (hi > lo) { diff --git a/packages/lexical-y/src/internal/Write.ts b/packages/lexical-y/src/internal/Write.ts index 419faa0855a..656c24cd145 100644 --- a/packages/lexical-y/src/internal/Write.ts +++ b/packages/lexical-y/src/internal/Write.ts @@ -1008,7 +1008,7 @@ export class Writer { this.replay = editor._recordTextEdits ? new TextReplay( this.plan.textEdits ?? null, - this.prevState._nodeMap, + this.prevState, $isRangeSelection(selection) && selection.isCollapsed() && selection.anchor.type === 'text' From 456250b9a17ba764b1a49659e79afdfd62258c0c Mon Sep 17 00:00:00 2001 From: Gerard Rovira Date: Fri, 30 Oct 2026 17:14:54 -0700 Subject: [PATCH 3/4] [lexical-website][lexical-y] Bug Fix: Keep a repeated unchanged word of a remade line past the edit-script bound, and state when a word is written again MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Description Follow-up from review of 598c3c19. Past the 128-edit bound, the diff of a remade line anchored on words found once in both texts and, between two anchors that still needed more than 128 edits, kept only the common start and end of that stretch. A line with a word repeated between the changes and no unique word (`old echo old echo ...` made again as `new echo new echo ...`, more than 64 pairs), or such a stretch next to a unique word, wrote every `echo` again: a collaborator's concurrent delete there came back, showed twice after an undo, a letter typed into it went to the line's start, and a format on it was lost. Between two anchors past the bound, every word found in both stretches is now paired with itself by occurrence (its k-th occurrence in the old text with its k-th in the new), and the longest run of those pairs in the same order in both is kept, with a bounded edit script between them. What is kept is still a common subsequence of the two texts, found in O((n + m) * 128 + (n + m) log(n + m)). y.md and D29 state exactly which words are written again: a word the update moved past kept words, and, where the word diff needs more than 128 edits (about 64 replaced words), a repeated word whose occurrences the update added or removed between two words found once in both texts. y.md's Open issues lists #9597 (a stale format restating a removed link), which kept middle words now show as kept ends already did. ## Test plan New cells in `RemadeTextStretches.test.ts`: 64, 65 and 200 replacements for three line shapes (distinct words, a word repeated between them, the same with one unique word), with the collaborator deleting, deleting and undoing, typing into or bolding an unchanged word, both id orders; a scaling case for a repeated word between every two changed words; and an independent longest-common-subsequence check of `keptStretches` (2,000 random word sequences with repeats below the bound: kept words = LCS; 200 large ones above it: ranges are equal text in order). ### Before On 598c3c19: ``` $ vitest --run --project unit packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts × a word repeated between them, 64 replaced ‖ the peer deletes an unchanged word (ids 1,2) × a word repeated between them, 64 replaced ‖ the peer delete, undos an unchanged word (ids 1,2) × a word repeated between them, 64 replaced ‖ the peer types an unchanged word (ids 1,2) × a word repeated between them, 64 replaced ‖ the peer bolds an unchanged word (ids 1,2) ... (48 cells: both repeated-word shapes, 64, 65 and 200 replaced, every edit, both id orders) × above it: equal runs in order, and every repeated word kept where its occurrences pair up Tests 49 failed | 61 passed (110) ``` ### After ``` $ vitest --run --project unit packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts Tests 110 passed (110) $ pnpm run ci-check EXIT 0 $ vitest --run --project unit --project scripts-unit Test Files 637 passed | 4 skipped (641) Tests 26102 passed | 210 expected fail | 135 skipped (26447) $ MATRIX_FULL=1 vitest --run --project unit packages/lexical-y/src/__tests__/unit/matrix Tests 11 failed | 15974 passed | 29 expected fail | 128 skipped (16142) (the same 11 cells as main) $ FUZZ_SEED_START=1 FUZZ_SEEDS=3000 vitest --run --project unit packages/lexical-y/src/__tests__/unit/Fuzz.test.ts Tests 1 failed | 123 passed (124) (seed 2277: dup, as on main) $ FUZZ_SEED_START=50000 FUZZ_SEEDS=2000 vitest --run --project unit packages/lexical-y/src/__tests__/unit/Fuzz.test.ts Tests 7 failed | 77 passed (84) (seeds 50180, 50231, 50372, 51439, 51507, 51972, 51976, as on main) ``` Whitespace-only remake of one alternately formatted node per word, 500 / 1,000 / 2,000 / 4,000 words: 38 / 73 / 104 / 209 ms. --- .../lexical-website/docs/collaboration/y.md | 27 +- packages/lexical-y/DECISIONS.md | 2 +- .../regressions/RemadeTextStretches.test.ts | 260 +++++++++++++----- packages/lexical-y/src/internal/Inline.ts | 146 ++++++---- 4 files changed, 303 insertions(+), 132 deletions(-) diff --git a/packages/lexical-website/docs/collaboration/y.md b/packages/lexical-website/docs/collaboration/y.md index 85ba673e5d2..c8e734ce2d7 100644 --- a/packages/lexical-website/docs/collaboration/y.md +++ b/packages/lexical-website/docs/collaboration/y.md @@ -817,10 +817,16 @@ person's: it was a node of its own (a tab, a line break, a node of spaces). A word joined to a changed word through an inline node, with no space between, is written again with it, and an inline node made between two changes is new - where it is put. The comparison stays linear in the line's length: where - a line has more than 128 changed words, the unchanged words found once in - both texts anchor it, so a word that occurs more than once in the line - and sits between such a word and a change may be written again. Prefer + where it is put. The words kept are the longest run of words the two + texts share in the same order, wherever the word diff needs at most 128 + edits (about 64 replaced words, each a delete and an insert). Beyond + that, so that the comparison stays linear in the line's length, the + words found once in both texts anchor it, and between two of them every + word found in both is paired with itself by occurrence (its first with + its first, and so on). A word that is not kept is written again: one the + update moved past kept words, and beyond 128 edits, a repeated word whose + occurrences that pairing does not line up (the update added or removed + some of them). Prefer these to `setTextContent`, which sets a node's text whole: such a change (and a change the browser makes to the DOM text, as an IME composition does) keeps the start and end its text shares with the node's old text, @@ -1010,9 +1016,11 @@ not merge as a user might expect: update's author has not seen, stays where it was, so it lands next to the new word, not in it (a letter typed at the end of the word ends up in front of the new word), and the collaborator's formatting of the changed - word at the same time is not kept. In a line with more than 128 changed - words, an unchanged word that occurs more than once in the line may be - written again. + word at the same time is not kept. So is a word the update moved past + the words it kept, and, where the line's word diff needs more than 128 + edits (about 64 replaced words), a repeated word whose occurrences the + update added to or removed from between two words found once in both + texts. The concurrent typed text is kept, and every client shows the same result. Edit the changed words in place instead (`TextNode#spliceText`, the selection's `insertText`): only the changed @@ -1311,6 +1319,11 @@ reproduces it: - A comment can spread onto later words when a client types uncommented text inside it offline while a collaborator's deletes there drop the item that ended it: [#9564](https://github.com/meta/lexical/issues/9564). +- A link a collaborator removes comes back when another collaborator + formats a range covering it at the same time; an update that makes a + line's text nodes again without the link while a stale edit formats a + range reaching past the link's text shows the same: + [#9597](https://github.com/meta/lexical/issues/9597). - Turning a paragraph inside a container that is itself inside a container (both not shadow roots) into a list removes the outer container and makes no list, in every `@lexical/y` editor (it runs diff --git a/packages/lexical-y/DECISIONS.md b/packages/lexical-y/DECISIONS.md index 3163438a335..ae15397f8ad 100644 --- a/packages/lexical-y/DECISIONS.md +++ b/packages/lexical-y/DECISIONS.md @@ -60,7 +60,7 @@ issues and docs that carry it. | D26 | **Undo favours people taking turns over concurrent races.** Where a fix makes undo correct for sequential edits at the cost of rare races between an undo and a simultaneous edit of the same text, the sequential case wins. | Taking turns is the common case; a race with a simultaneous edit of the same text is rare. Picked on 10-28, confirmed on 10-30. | `DECIDED` | 10-30 | [#9400] | | D27 | **Moves are a delete and a re-insert** at launch; the binding rewrites the cheaper side. Real moves within a container ([#9449], records after Yjs's unmerged `ContentMove` design) are paused and land last, backward compatible: a new layout version that older clients open read only. Move records leave tombstones, as every Yjs binding does. Moves across containers stay copies. | Every Yjs binding turns moves into delete plus insert; Yjs dropped its own move for complexity. Reliability of the rest came first. | `DECIDED` | 10-25 | [#9449], [#9455], [y.md: Limits][limits] | | D28 | **Text identity is fixed at the source.** While a binding is attached, core records text edits (`spliceText`, insert and remove, split and merge), and text entity plugins record a grow as a move and a remake as a retype. Text only `setTextContent` or the DOM changed is diffed. The binding adds no per-case heuristics. | No binding can recover intent an API dropped; special-case rules were fragile and regressed neighbouring cases. | `DECIDED` | 10-27 | [#9483], [y.md: Edits made by code][y-code] | -| D29 | **Remade text keeps its common start and end by character**, whatever the node boundaries (y-prosemirror's `updateYText` rule), **and between two changes every run of unchanged whole words** (a word-by-word shortest edit script; past 128 word edits in a line, the words found once in both texts anchor it, so it stays linear), and never brings back a collaborator's delete. It replaces the word matching of [#9565]. | It is the industry-standard behaviour, and word matching glued words together and merged numbers oddly. Keeping only the start and end wrote everything between two changed words again, so a collaborator's delete there came back, and showed twice after the collaborator's undo ([#9621]). | `DECIDED` | 10-30 | [#9565], [#9598], [#9621] | +| D29 | **Remade text keeps its common start and end by character**, whatever the node boundaries (y-prosemirror's `updateYText` rule), **and between two changes every run of unchanged whole words** (the longest common run of words, by a shortest edit script; past 128 word edits in a line, so that it stays linear, the words found once in both texts anchor it and every word found in both between two anchors is paired with itself by occurrence; a word the update moved, or past 128 edits a repeated word whose occurrences do not pair up, is written again), and never brings back a collaborator's delete. It replaces the word matching of [#9565]. | It is the industry-standard behaviour, and word matching glued words together and merged numbers oddly. Keeping only the start and end wrote everything between two changed words again, so a collaborator's delete there came back, and showed twice after the collaborator's undo ([#9621]). | `DECIDED` | 10-30 | [#9565], [#9598], [#9621] | | D30 | **Diffed rewrites snap to whole words**; keystrokes and IME composition keep the exact diff. Scripts without spaces treat each character as a boundary. | Concurrent rewrites otherwise split words. Real word segmentation is a follow-up ([#9467]). | `DECIDED` | 10-26 | [#9400], [#9467] | | D31 | **A list item's nested list gets an item of its own.** `NodeStructureExtension`'s list rule prevents an item holding a nested list next to other content when it is written; the binding does not repair it afterwards. | That shape lost text in outdents, Backspaces and offline merges. Preventing it at write time follows D20. | `DECIDED` | 10-26 | [#9452], [y.md: Limits][limits] | | D32 | **List merging ignores `start`.** | Existing Lexical behaviour, intended. | `DECIDED` | 10-23 | [#9452] | diff --git a/packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts b/packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts index 64e9a672899..b59fe5af962 100644 --- a/packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts +++ b/packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts @@ -29,6 +29,7 @@ import { } from 'lexical'; import {describe, expect, test} from 'vitest'; +import {keptStretches} from '../../../internal/Inline'; import { expectConverged, flush, @@ -484,80 +485,126 @@ describe('#9621: nothing a collaborator deleted comes back, nothing shows twice' // Past the bound on one shortest edit script (more than 64 words replaced, // each a delete and an insert), the words found once in both texts anchor -// the diff: a collaborator's concurrent edit of an unchanged word is kept as -// below the bound. +// the diff, and between them, or where there are none, every word in both +// texts, paired by occurrence: a collaborator's concurrent edit of an +// unchanged word is kept as below the bound, also when that word repeats +// all along the line. describe('#9621: an edit between more changed words than one edit script covers', () => { - for (const changes of [64, 65, 200]) { - for (const ids of IDS) { - for (const edit of ['delete', 'delete, undo', 'type', 'bold'] as const) { - test(`${changes} words replaced ‖ the peer ${edit}s an unchanged word (ids ${ids})`, async () => { - const words = Array.from( - {length: 2 * changes + 1}, - (_, i) => `w${i}z`, - ); - const next = words - .map((w, i) => (i % 2 === 0 && i < 2 * changes ? `r${i}z` : w)) - .join(' '); - const old = words.join(' '); - const victim = words[3]; - const at = old.indexOf(victim); - const t = await setup( - () => { - $getRoot().append( - $createParagraphNode().append($createTextNode('Title')), - $createParagraphNode().append($createTextNode(old)), - ); - }, - {ids}, - ); - t.offline(); - if (edit === 'bold') { - t.b.update(() => { - const leaf = $line().getFirstChildOrThrow() as TextNode; - const [, mid] = leaf.splitText(at, at + victim.length); - mid.toggleFormat('bold'); + const LINES: [string, (n: number) => [string[], string[], number]][] = [ + [ + 'distinct words', + n => { + const words = Array.from({length: 2 * n + 1}, (_, i) => `w${i}z`); + return [ + words, + words.map((w, i) => (i % 2 === 0 && i < 2 * n ? `r${i}z` : w)), + 3, + ]; + }, + ], + [ + 'a word repeated between them', + n => [ + [ + 's1z', + ...Array.from({length: n}, () => ['o1z', 'echo']).flat(), + 'e1z', + ], + [ + 's2z', + ...Array.from({length: n}, () => ['n1z', 'echo']).flat(), + 'e2z', + ], + 4, + ], + ], + [ + 'a word repeated between them and one unique word', + n => { + const gapA = Array.from({length: n}, () => ['o1z', 'echo']).flat(); + const gapB = Array.from({length: n}, () => ['n1z', 'echo']).flat(); + return [ + ['s1z', ...gapA, 'pin', ...gapA, 'e1z'], + ['s2z', ...gapB, 'pin', ...gapB, 'e2z'], + 4, + ]; + }, + ], + ]; + for (const [shape, make] of LINES) { + for (const changes of [64, 65, 200]) { + for (const ids of IDS) { + for (const edit of [ + 'delete', + 'delete, undo', + 'type', + 'bold', + ] as const) { + test(`${shape}, ${changes} replaced ‖ the peer ${edit}s an unchanged word (ids ${ids})`, async () => { + const [oldWords, newWords, v] = make(changes); + const old = oldWords.join(' '); + const next = newWords.join(' '); + const victim = oldWords[v]; + const at = oldWords.slice(0, v).join(' ').length + 1; + const nextAt = newWords.slice(0, v).join(' ').length + 1; + const t = await setup( + () => { + $getRoot().append( + $createParagraphNode().append($createTextNode('Title')), + $createParagraphNode().append($createTextNode(old)), + ); + }, + {ids}, + ); + t.offline(); + if (edit === 'bold') { + t.b.update(() => { + const leaf = $line().getFirstChildOrThrow() as TextNode; + const [, mid] = leaf.splitText(at, at + victim.length); + mid.toggleFormat('bold'); + $setSelection(null); + }); + } else if (edit === 'type') { + t.b.update($splice(at + 1, 0, 'Q')); + } else { + t.b.update($splice(at, victim.length)); + } + t.a.update(() => { + for (const child of $line().getChildren()) { + child.remove(); + } + $line().append($createTextNode(next)); $setSelection(null); }); - } else if (edit === 'type') { - t.b.update($splice(at + 1, 0, 'Q')); - } else { - t.b.update($splice(at, victim.length)); - } - t.a.update(() => { - for (const child of $line().getChildren()) { - child.remove(); + await t.sync(); + if (edit === 'delete, undo') { + t.b.undo(); + await t.sync(); } - $line().append($createTextNode(next)); - $setSelection(null); + const expected = + edit === 'delete' + ? next.slice(0, nextAt) + next.slice(nextAt + victim.length) + : edit === 'type' + ? next.slice(0, nextAt + 1) + 'Q' + next.slice(nextAt + 1) + : next; + const fresh = await freshReader(t.a); + expect([...t.peers, fresh].map(lineText)).toEqual( + [...t.peers, fresh].map(() => expected), + ); + if (edit === 'bold') { + expect( + t.a.read(() => + $line() + .getAllTextNodes() + .filter(n => n.hasFormat('bold')) + .map(n => n.getTextContent()), + ), + ).toEqual([victim]); + } + await expectConverged(t.peers); + expect(t.peers.flatMap(p => p.warnings)).toEqual([]); }); - await t.sync(); - if (edit === 'delete, undo') { - t.b.undo(); - await t.sync(); - } - const expected = - edit === 'delete' - ? next.replace(victim, '') - : edit === 'type' - ? next.replace(victim, 'wQ3z') - : next; - const fresh = await freshReader(t.a); - expect([...t.peers, fresh].map(lineText)).toEqual( - [...t.peers, fresh].map(() => expected), - ); - if (edit === 'bold') { - expect( - t.a.read(() => - $line() - .getAllTextNodes() - .filter(n => n.hasFormat('bold')) - .map(n => n.getTextContent()), - ), - ).toEqual([victim]); - } - await expectConverged(t.peers); - expect(t.peers.flatMap(p => p.warnings)).toEqual([]); - }); + } } } } @@ -565,7 +612,8 @@ describe('#9621: an edit between more changed words than one edit script covers' // The diff of a remade line stays linear in the line's length: two changes // at its ends (the whole line between them diffed), a change every fourth -// word (past one edit script's bound, where unique words anchor it), and +// word (past one edit script's bound, where unique words anchor it), a +// word repeated between every two changed words (paired by occurrence), and // the white space between every two words changed in a line of one // alternately formatted node per word (a kept run per word, a node per run). describe('#9621: a long remade line stays linear', () => { @@ -618,6 +666,21 @@ describe('#9621: a long remade line stays linear', () => { (words: string[]) => [$createTextNode(changed(4)(words))], changed(4), ], + [ + 'a word repeated between every two changed words', + (words: string[]) => [ + $createTextNode( + words.map((_, i) => (i % 2 ? 'echo' : 'o1z')).join(' '), + ), + ], + (words: string[]) => [ + $createTextNode( + words.map((_, i) => (i % 2 ? 'echo' : 'n1z')).join(' '), + ), + ], + (words: string[]) => + words.map((_, i) => (i % 2 ? 'echo' : 'n1z')).join(' '), + ], [ 'the white space between every two words, one node per word', perWord(' '), @@ -638,3 +701,60 @@ describe('#9621: a long remade line stays linear', () => { ); } }); + +// What a remade line keeps, checked against an independent longest common +// subsequence: below the bound, as many words as the longest common run of +// words; above it, still runs of equal text in order in both texts. +describe('#9621: the kept words of a remade line', () => { + const lcs = (a: string[], b: string[]) => { + let row = Array(b.length + 1).fill(0); + for (const x of a) { + const next = [0]; + for (let j = 0; j < b.length; j++) { + next.push(x === b[j] ? row[j] + 1 : Math.max(row[j + 1], next[j])); + } + row = next; + } + return row[b.length]; + }; + /** The words kept, checking that each range is equal text, in order. */ + const kept = (a: string[], b: string[]) => { + const aa = 'old0z ' + a.join(' ') + ' old1z'; + const bb = 'new0z ' + b.join(' ') + ' new1z'; + let lastA = 0; + let lastB = 0; + let n = 0; + for (const [x0, x1, dx] of keptStretches(aa, bb, -1, false)) { + expect(x0).toBeGreaterThanOrEqual(lastB); + expect(x0 + dx).toBeGreaterThanOrEqual(lastA); + expect(bb.slice(x0, x1)).toBe(aa.slice(x0 + dx, x1 + dx)); + n += (bb.slice(x0, x1).match(/w\d+z/g) ?? []).length; + lastA = x1 + dx; + lastB = x1; + } + return n; + }; + test('below the bound: a longest common run of words', () => { + const rnd = random(9621); + for (let k = 0; k < 2000; k++) { + const words = (n: number) => + Array.from({length: 1 + rnd(n)}, () => `w${rnd(6)}z`); + const a = words(40); + const b = words(40); + expect(kept(a, b)).toBe(lcs(a, b)); + } + }); + test('above it: equal runs in order, and every repeated word kept where its occurrences pair up', () => { + const rnd = random(9622); + for (let k = 0; k < 200; k++) { + const n = 200 + rnd(400); + const a = Array.from({length: n}, () => `w${rnd(12)}z`); + const b = a.map(w => (rnd(2) === 0 ? `w${12 + rnd(1000)}z` : w)); + expect(kept(a, b)).toBeGreaterThanOrEqual(0); + } + const n = 300; + const a = Array.from({length: n}, (_, i) => (i % 2 ? 'w1z' : `w${2 + i}z`)); + const b = a.map((w, i) => (i % 2 ? w : `w${2000 + i}z`)); + expect(kept(a, b)).toBe(n / 2); + }); +}); diff --git a/packages/lexical-y/src/internal/Inline.ts b/packages/lexical-y/src/internal/Inline.ts index d4360d3ae79..7dff9bb756b 100644 --- a/packages/lexical-y/src/internal/Inline.ts +++ b/packages/lexical-y/src/internal/Inline.ts @@ -792,8 +792,8 @@ export function keptEnds( /** * The most edits one shortest edit script of a remade line's changed words - * looks for (see commonWords): past it, the words unique to both texts - * anchor the rest, so the diff stays linear in the line's length. + * looks for (see commonWords): past it, words the two texts share anchor the + * rest, so the diff stays linear in the line's length. */ const MAX_EDITS = 128; @@ -888,19 +888,97 @@ function commonTokens( return null; } +/** + * The longest run of `pairs` (in order of their second index) whose first + * indexes increase too: patience sorting, O(k log k). + */ +function increasing(pairs: readonly [number, number][]): [number, number][] { + const tops: number[] = []; + const prev = new Int32Array(pairs.length).fill(-1); + for (let i = 0; i < pairs.length; i++) { + let lo = 0; + let hi = tops.length; + while (lo < hi) { + const mid = (lo + hi) >> 1; + if (pairs[tops[mid]][0] < pairs[i][0]) { + lo = mid + 1; + } else { + hi = mid; + } + } + prev[i] = lo > 0 ? tops[lo - 1] : -1; + tops[lo] = i; + } + const out: [number, number][] = []; + for (let i = tops.length > 0 ? tops[tops.length - 1] : -1; i >= 0; ) { + out.push(pairs[i]); + i = prev[i]; + } + return out.reverse(); +} + +/** + * Pairs of the same word in `a` and `b` from `a0` to `a1` and `b0` to `b1`, + * in order of `b`: with `unique`, the words that occur once in each; else + * every word that occurs in both, its k-th occurrence in `a` with its k-th + * in `b`. + */ +function wordPairs( + a: readonly number[], + b: readonly number[], + [a0, a1, b0, b1]: [number, number, number, number], + unique: boolean, +): [number, number][] { + const inA = new Map(); + for (let x = a0; x < a1; x++) { + const at = inA.get(a[x]); + if (at === undefined) { + inA.set(a[x], [x]); + } else { + at.push(x); + } + } + const seen = new Map(); + for (let y = b0; y < b1; y++) { + seen.set(b[y], (seen.get(b[y]) ?? 0) + 1); + } + const out: [number, number][] = []; + const nth = new Map(); + for (let y = b0; y < b1; y++) { + const at = inA.get(b[y]); + if ( + at === undefined || + (unique && (at.length > 1 || seen.get(b[y])! > 1)) + ) { + continue; + } + const k = nth.get(b[y]) ?? 0; + nth.set(b[y], k + 1); + if (k < at.length) { + out.push([at[k], y]); + } + } + return out; +} + /** * The words equal in `a` and `b` that are kept, as pairs of indexes, in - * order: their common start and end, then a shortest edit script of the rest - * (commonTokens) where it takes at most MAX_EDITS edits; past that, the - * words that occur once in each text, in the longest run of them in the - * same order in both (patience diff), and the same between each two of - * those, once (between them, past MAX_EDITS edits, only their common start - * and end). O((n + m) * MAX_EDITS + n log n). + * increasing order of both: their common start and end, then a shortest edit + * script of the rest (commonTokens) where it takes at most MAX_EDITS edits. + * Past that, the words that occur once in each text anchor the rest, in the + * longest run of them in the same order in both (patience diff), and + * between two anchors, past MAX_EDITS edits again, so does every word that + * occurs in both, its k-th occurrence in one text paired with its k-th in + * the other (a run of repeated words with other words changed between them, + * `old echo old echo` made again as `new echo new echo`, keeps every + * `echo`). What is kept is a common subsequence of the two texts, so a kept + * word keeps its characters; a word is written again only where it is not + * paired. O((n + m) * MAX_EDITS + (n + m) log(n + m)). */ function commonWords( a: readonly number[], b: readonly number[], - anchored = false, + level = 0, ): [number, number][] { const out: [number, number][] = []; let a0 = 0; @@ -920,57 +998,17 @@ function commonWords( for (const [x, y] of pairs) { out.push([x + a0, y + b0]); } - } else if (!anchored) { - // Words that occur once in each text. - const count = new Map(); - const atA = new Map(); - for (let x = a0; x < a1; x++) { - count.set(a[x], (count.get(a[x]) ?? 0) + 1); - atA.set(a[x], x); - } - const inB = new Map(); - const atB = new Map(); - for (let y = b0; y < b1; y++) { - inB.set(b[y], (inB.get(b[y]) ?? 0) + 1); - atB.set(b[y], y); - } - const unique: [number, number][] = []; - for (let y = b0; y < b1; y++) { - if (count.get(b[y]) === 1 && inB.get(b[y]) === 1) { - unique.push([atA.get(b[y])!, atB.get(b[y])!]); - } - } - // The longest run of them in the same order in both (in order of - // `b`, increasing in `a`): patience sorting. - const tops: number[] = []; - const prev = new Int32Array(unique.length).fill(-1); - for (let i = 0; i < unique.length; i++) { - let lo = 0; - let hi = tops.length; - while (lo < hi) { - const mid = (lo + hi) >> 1; - if (unique[tops[mid]][0] < unique[i][0]) { - lo = mid + 1; - } else { - hi = mid; - } - } - prev[i] = lo > 0 ? tops[lo - 1] : -1; - tops[lo] = i; - } - const anchors: [number, number][] = []; - for (let i = tops.length > 0 ? tops[tops.length - 1] : -1; i >= 0; ) { - anchors.push(unique[i]); - i = prev[i]; - } - anchors.reverse(); + } else if (level < 2) { + const anchors = increasing( + wordPairs(a, b, [a0, a1, b0, b1], level === 0), + ); let x = a0; let y = b0; for (const [ax, by] of [...anchors, [a1, b1] as [number, number]]) { for (const [px, py] of commonWords( a.slice(x, ax), b.slice(y, by), - true, + level + 1, )) { out.push([px + x, py + y]); } From f64c0466f46dc78fd99b2be0f2ac1ca8de7445d5 Mon Sep 17 00:00:00 2001 From: Gerard Rovira Date: Fri, 30 Oct 2026 17:43:17 -0700 Subject: [PATCH 4/4] [lexical-website][lexical-y] Bug Fix: Find the kept words of a remade line with an exact linear-space diff, and document a word changed in part MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Description Follow-up from review of 456250b9. Past 128 edits, the diff of a remade line's words fell back to heuristics (words found once in both texts as anchors, then repeated words paired by occurrence). Each left a case where an unchanged word was not kept: pairing the k-th occurrence of a word with its k-th occurrence lost a different, surviving `aa` when the update removed the first `aa` of `(old aa bb) x 65` and a collaborator deleted that same `aa`, and dropped an unchanged `bb` there, so the collaborator's delete of it came back. The kept words are now always a longest common subsequence of the two lines' words, found by Myers' linear-space diff (middle snake) over the words both texts have (a word only one text has is never kept, and leaving it out keeps the diff short), up to 2,048 edits among those: O((n + m) * 2048) time at most, about 30 ms for a 5,000-word line, O(n + m) memory. Past that bound (a line made again with most of its words moved), only the common start and end are kept, as before this pull request. The occurrence pairing and the anchors are gone. y.md and D29 state the bound, and that a word changed only in part is written whole (#9598's rule for an insertion between two letters of a word), so a collaborator's undone delete of its unchanged part shows twice: #9635, pinned as `test.fails` with a passing control of separate words. ## Test plan In `RemadeTextStretches.test.ts`: cells for lines of repeated words with an earlier occurrence removed (64, 65, 200, every collaborator edit, both id orders); the longest-common-subsequence check now over 1,100 random lines with repeated words up to 400 words long (kept words = LCS exactly); a line made again in reverse past the bound; the #9635 cells. ### Before On 456250b9: ``` $ vitest --run --project unit packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts × repeated words, an earlier occurrence of one removed, 64 replaced ‖ the peer deletes an unchanged word (ids 1,2) ... (12 cells: 64, 65 and 200 replaced, delete and type, both id orders) × a longest common run of words, with repeated words, short and long lines Tests 13 failed | 124 passed | 2 expected fail (139) ``` ### After ``` $ vitest --run --project unit packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts Tests 137 passed | 2 expected fail (139) $ pnpm run ci-check EXIT 0 $ vitest --run --project unit --project scripts-unit Test Files 637 passed | 4 skipped (641) Tests 26129 passed | 212 expected fail | 135 skipped (26476) $ MATRIX_FULL=1 vitest --run --project unit packages/lexical-y/src/__tests__/unit/matrix Tests 11 failed | 15974 passed | 29 expected fail | 128 skipped (16142) (the same 11 cells as main) $ FUZZ_SEED_START=1 FUZZ_SEEDS=3000 vitest --run --project unit packages/lexical-y/src/__tests__/unit/Fuzz.test.ts Tests 1 failed | 123 passed (124) (seed 2277: dup, as on main) $ FUZZ_SEED_START=50000 FUZZ_SEEDS=2000 vitest --run --project unit packages/lexical-y/src/__tests__/unit/Fuzz.test.ts Tests 7 failed | 77 passed (84) (seeds 50180, 50231, 50372, 51439, 51507, 51972, 51976, as on main) ``` `keptStretches` on 5,000-word lines: every other word replaced 16 ms, repeated words with one removed 23 ms, 10 % of the words swapped 37 ms, made again in reverse (past the bound) 24 ms, 20-word vocabulary with half the words changed (past the bound) 25 ms. --- .../lexical-website/docs/collaboration/y.md | 29 +- packages/lexical-y/DECISIONS.md | 3 +- .../regressions/RemadeTextStretches.test.ts | 127 ++++++-- packages/lexical-y/src/internal/Inline.ts | 274 +++++++----------- 4 files changed, 225 insertions(+), 208 deletions(-) diff --git a/packages/lexical-website/docs/collaboration/y.md b/packages/lexical-website/docs/collaboration/y.md index c8e734ce2d7..6933a83a48a 100644 --- a/packages/lexical-website/docs/collaboration/y.md +++ b/packages/lexical-website/docs/collaboration/y.md @@ -818,15 +818,14 @@ person's: joined to a changed word through an inline node, with no space between, is written again with it, and an inline node made between two changes is new where it is put. The words kept are the longest run of words the two - texts share in the same order, wherever the word diff needs at most 128 - edits (about 64 replaced words, each a delete and an insert). Beyond - that, so that the comparison stays linear in the line's length, the - words found once in both texts anchor it, and between two of them every - word found in both is paired with itself by occurrence (its first with - its first, and so on). A word that is not kept is written again: one the - update moved past kept words, and beyond 128 edits, a repeated word whose - occurrences that pairing does not line up (the update added or removed - some of them). Prefer + texts share in the same order (where a word repeats, which of its + occurrences is kept can differ from what the update meant), wherever + that takes at most 2,048 word edits among the words both texts have, so + that the comparison stays within about 30 ms for a 5,000-word line. + Beyond that (a line made again with most of its words moved), only its + common start and end are kept. A word that is not kept is written again: + one the update moved past kept words, and beyond 2,048 edits, every word + between the first and the last change. Prefer these to `setTextContent`, which sets a node's text whole: such a change (and a change the browser makes to the DOM text, as an IME composition does) keeps the start and end its text shares with the node's old text, @@ -1017,10 +1016,14 @@ not merge as a user might expect: new word, not in it (a letter typed at the end of the word ends up in front of the new word), and the collaborator's formatting of the changed word at the same time is not kept. So is a word the update moved past - the words it kept, and, where the line's word diff needs more than 128 - edits (about 64 replaced words), a repeated word whose occurrences the - update added to or removed from between two words found once in both - texts. + the words it kept, and, where the words both texts have need more than + 2,048 edits, everything between the first and the last change. A changed + word is written again even where only part of it changed: an insertion + between two letters of a word (`bbcc` made again as `bb xxcc`) writes + `bb` again, so a collaborator who deletes the old `bb` at the same time + and undoes the delete brings it back next to the new one, and `bb` shows + twice ([#9635](https://github.com/meta/lexical/issues/9635)), as when two + clients remove the same text and both undo (see Undo below). The concurrent typed text is kept, and every client shows the same result. Edit the changed words in place instead (`TextNode#spliceText`, the selection's `insertText`): only the changed diff --git a/packages/lexical-y/DECISIONS.md b/packages/lexical-y/DECISIONS.md index ae15397f8ad..dd3de43a433 100644 --- a/packages/lexical-y/DECISIONS.md +++ b/packages/lexical-y/DECISIONS.md @@ -60,7 +60,7 @@ issues and docs that carry it. | D26 | **Undo favours people taking turns over concurrent races.** Where a fix makes undo correct for sequential edits at the cost of rare races between an undo and a simultaneous edit of the same text, the sequential case wins. | Taking turns is the common case; a race with a simultaneous edit of the same text is rare. Picked on 10-28, confirmed on 10-30. | `DECIDED` | 10-30 | [#9400] | | D27 | **Moves are a delete and a re-insert** at launch; the binding rewrites the cheaper side. Real moves within a container ([#9449], records after Yjs's unmerged `ContentMove` design) are paused and land last, backward compatible: a new layout version that older clients open read only. Move records leave tombstones, as every Yjs binding does. Moves across containers stay copies. | Every Yjs binding turns moves into delete plus insert; Yjs dropped its own move for complexity. Reliability of the rest came first. | `DECIDED` | 10-25 | [#9449], [#9455], [y.md: Limits][limits] | | D28 | **Text identity is fixed at the source.** While a binding is attached, core records text edits (`spliceText`, insert and remove, split and merge), and text entity plugins record a grow as a move and a remake as a retype. Text only `setTextContent` or the DOM changed is diffed. The binding adds no per-case heuristics. | No binding can recover intent an API dropped; special-case rules were fragile and regressed neighbouring cases. | `DECIDED` | 10-27 | [#9483], [y.md: Edits made by code][y-code] | -| D29 | **Remade text keeps its common start and end by character**, whatever the node boundaries (y-prosemirror's `updateYText` rule), **and between two changes every run of unchanged whole words** (the longest common run of words, by a shortest edit script; past 128 word edits in a line, so that it stays linear, the words found once in both texts anchor it and every word found in both between two anchors is paired with itself by occurrence; a word the update moved, or past 128 edits a repeated word whose occurrences do not pair up, is written again), and never brings back a collaborator's delete. It replaces the word matching of [#9565]. | It is the industry-standard behaviour, and word matching glued words together and merged numbers oddly. Keeping only the start and end wrote everything between two changed words again, so a collaborator's delete there came back, and showed twice after the collaborator's undo ([#9621]). | `DECIDED` | 10-30 | [#9565], [#9598], [#9621] | +| D29 | **Remade text keeps its common start and end by character**, whatever the node boundaries (y-prosemirror's `updateYText` rule), **and between two changes every run of unchanged whole words** (the longest common run of words, by a linear-space shortest edit script over the words both texts have, up to 2,048 edits, about 30 ms for a 5,000-word line; past it, only the start and end; a word the update moved is written again; a word changed only in part is written whole, so a collaborator's undone delete of its unchanged part shows twice, [#9635]), and never brings back a collaborator's delete. It replaces the word matching of [#9565]. | It is the industry-standard behaviour, and word matching glued words together and merged numbers oddly. Keeping only the start and end wrote everything between two changed words again, so a collaborator's delete there came back, and showed twice after the collaborator's undo ([#9621]). | `DECIDED` | 10-30 | [#9565], [#9598], [#9621] | | D30 | **Diffed rewrites snap to whole words**; keystrokes and IME composition keep the exact diff. Scripts without spaces treat each character as a boundary. | Concurrent rewrites otherwise split words. Real word segmentation is a follow-up ([#9467]). | `DECIDED` | 10-26 | [#9400], [#9467] | | D31 | **A list item's nested list gets an item of its own.** `NodeStructureExtension`'s list rule prevents an item holding a nested list next to other content when it is written; the binding does not repair it afterwards. | That shape lost text in outdents, Backspaces and offline merges. Preventing it at write time follows D20. | `DECIDED` | 10-26 | [#9452], [y.md: Limits][limits] | | D32 | **List merging ignores `start`.** | Existing Lexical behaviour, intended. | `DECIDED` | 10-23 | [#9452] | @@ -200,3 +200,4 @@ issues and docs that carry it. [#9609]: https://github.com/meta/lexical/issues/9609 [#9610]: https://github.com/meta/lexical/issues/9610 [#9621]: https://github.com/meta/lexical/issues/9621 +[#9635]: https://github.com/meta/lexical/issues/9635 diff --git a/packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts b/packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts index b59fe5af962..86730ce2354 100644 --- a/packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts +++ b/packages/lexical-y/src/__tests__/unit/regressions/RemadeTextStretches.test.ts @@ -248,6 +248,44 @@ describe('#9621: unchanged words of other shapes between two changes keep a dele } }); +// A word changed in part is written whole (an insertion between two of its +// letters, #9598), so a collaborator's delete of its unchanged part, undone, +// comes back next to the new word: #9635, as two clients removing the same +// text and both undoing (D50). Separate words keep theirs. +describe('#9635: an undone delete of part of a word changed in part', () => { + for (const ids of IDS) { + for (const [name, line, made, glued] of [ + ['glued word', 'aa bbcc dd', 'aa bb xxcc dd', true], + ['separate words', 'aa bb cc dd', 'aa bb xx cc dd', false], + ] as [string, string, string, boolean][]) { + (glued ? test.fails : test)( + `${name}: "${line}" made again as "${made}" ‖ the peer deletes "aa bb" and undoes (ids ${ids})${glued ? ': bb shows twice (#9635)' : ''}`, + async () => { + const t = await setup( + () => { + $getRoot().append( + $createParagraphNode().append($createTextNode('Title')), + $createParagraphNode().append($createTextNode(line)), + ); + }, + {ids}, + ); + t.offline(); + t.a.update($remake([[line.slice(3), made.slice(3)]])); + t.b.update($splice(0, 5)); + t.b.undo(); + await t.sync(); + const fresh = await freshReader(t.a); + expect([...t.peers, fresh].map(lineText)).toEqual( + [...t.peers, fresh].map(() => made), + ); + await expectConverged(t.peers); + }, + ); + } + } +}); + /** A seeded random integer below `n`. */ function random(seed: number): (n: number) => number { let s = seed >>> 0; @@ -483,14 +521,15 @@ describe('#9621: nothing a collaborator deleted comes back, nothing shows twice' } }); -// Past the bound on one shortest edit script (more than 64 words replaced, -// each a delete and an insert), the words found once in both texts anchor -// the diff, and between them, or where there are none, every word in both -// texts, paired by occurrence: a collaborator's concurrent edit of an -// unchanged word is kept as below the bound, also when that word repeats -// all along the line. -describe('#9621: an edit between more changed words than one edit script covers', () => { - const LINES: [string, (n: number) => [string[], string[], number]][] = [ +// Many changed words (64 to 200 replaced, each a delete and an insert), in +// lines of distinct words and of repeated ones, an earlier occurrence of a +// repeated word removed included: the exact diff keeps every unchanged word, +// so a collaborator's concurrent edit of one is kept. +describe('#9621: an edit between many changed words', () => { + const LINES: [ + string, + (n: number) => [string[], string[], number, number?], + ][] = [ [ 'distinct words', n => { @@ -518,6 +557,21 @@ describe('#9621: an edit between more changed words than one edit script covers' 4, ], ], + [ + 'repeated words, an earlier occurrence of one removed', + n => { + const oldWords = [ + 's1z', + ...Array.from({length: n}, () => ['o1z', 'aa', 'bb']).flat(), + 'e1z', + ]; + const newWords = oldWords.map(w => + w === 'o1z' ? 'n1z' : w === 's1z' ? 's2z' : w === 'e1z' ? 'e2z' : w, + ); + newWords.splice(2, 1); + return [oldWords, newWords, 3, 2]; + }, + ], [ 'a word repeated between them and one unique word', n => { @@ -541,12 +595,12 @@ describe('#9621: an edit between more changed words than one edit script covers' 'bold', ] as const) { test(`${shape}, ${changes} replaced ‖ the peer ${edit}s an unchanged word (ids ${ids})`, async () => { - const [oldWords, newWords, v] = make(changes); + const [oldWords, newWords, v, newV = v] = make(changes); const old = oldWords.join(' '); const next = newWords.join(' '); const victim = oldWords[v]; const at = oldWords.slice(0, v).join(' ').length + 1; - const nextAt = newWords.slice(0, v).join(' ').length + 1; + const nextAt = newWords.slice(0, newV).join(' ').length + 1; const t = await setup( () => { $getRoot().append( @@ -612,8 +666,8 @@ describe('#9621: an edit between more changed words than one edit script covers' // The diff of a remade line stays linear in the line's length: two changes // at its ends (the whole line between them diffed), a change every fourth -// word (past one edit script's bound, where unique words anchor it), a -// word repeated between every two changed words (paired by occurrence), and +// word, a word repeated between every two changed words, every word moved +// (past the exact diff's bound, where only the start and end are kept), and // the white space between every two words changed in a line of one // alternately formatted node per word (a kept run per word, a node per run). describe('#9621: a long remade line stays linear', () => { @@ -681,6 +735,12 @@ describe('#9621: a long remade line stays linear', () => { (words: string[]) => words.map((_, i) => (i % 2 ? 'echo' : 'n1z')).join(' '), ], + [ + 'every word moved (made again in reverse, past the exact bound)', + (words: string[]) => [$createTextNode(words.join(' '))], + (words: string[]) => [$createTextNode([...words].reverse().join(' '))], + (words: string[]) => [...words].reverse().join(' '), + ], [ 'the white space between every two words, one node per word', perWord(' '), @@ -703,8 +763,10 @@ describe('#9621: a long remade line stays linear', () => { }); // What a remade line keeps, checked against an independent longest common -// subsequence: below the bound, as many words as the longest common run of -// words; above it, still runs of equal text in order in both texts. +// subsequence: as many words as the longest common run of words, for lines +// of any length up to the exact diff's bound of 2,048 edits among the words +// both texts have; past it (a line made again in reverse), only its common +// start and end, still runs of equal text in order in both texts. describe('#9621: the kept words of a remade line', () => { const lcs = (a: string[], b: string[]) => { let row = Array(b.length + 1).fill(0); @@ -734,27 +796,32 @@ describe('#9621: the kept words of a remade line', () => { } return n; }; - test('below the bound: a longest common run of words', () => { + test('a longest common run of words, with repeated words, short and long lines', () => { const rnd = random(9621); - for (let k = 0; k < 2000; k++) { - const words = (n: number) => - Array.from({length: 1 + rnd(n)}, () => `w${rnd(6)}z`); - const a = words(40); - const b = words(40); + for (let k = 0; k < 1100; k++) { + const vocab = 2 + rnd(k < 1000 ? 6 : 40); + const a = Array.from( + {length: 1 + rnd(k < 1000 ? 40 : 400)}, + () => `w${rnd(vocab)}z`, + ); + const b = a + .map(w => (rnd(3) === 0 ? `w${rnd(2 * vocab)}z` : w)) + .filter(() => rnd(8) > 0); expect(kept(a, b)).toBe(lcs(a, b)); } - }); - test('above it: equal runs in order, and every repeated word kept where its occurrences pair up', () => { + }, 60_000); + test('past the exact bound: equal runs in order', () => { + const a = Array.from({length: 3000}, (_, i) => `w${i}z`); + expect(kept(a, [...a].reverse())).toBeGreaterThanOrEqual(0); const rnd = random(9622); - for (let k = 0; k < 200; k++) { - const n = 200 + rnd(400); - const a = Array.from({length: n}, () => `w${rnd(12)}z`); - const b = a.map(w => (rnd(2) === 0 ? `w${12 + rnd(1000)}z` : w)); + for (let k = 0; k < 20; k++) { + const b = [...a]; + for (let i = 0; i < 2000; i++) { + const x = rnd(b.length); + const y = rnd(b.length); + [b[x], b[y]] = [b[y], b[x]]; + } expect(kept(a, b)).toBeGreaterThanOrEqual(0); } - const n = 300; - const a = Array.from({length: n}, (_, i) => (i % 2 ? 'w1z' : `w${2 + i}z`)); - const b = a.map((w, i) => (i % 2 ? w : `w${2000 + i}z`)); - expect(kept(a, b)).toBe(n / 2); }); }); diff --git a/packages/lexical-y/src/internal/Inline.ts b/packages/lexical-y/src/internal/Inline.ts index 7dff9bb756b..36d02f16d50 100644 --- a/packages/lexical-y/src/internal/Inline.ts +++ b/packages/lexical-y/src/internal/Inline.ts @@ -790,13 +790,6 @@ export function keptEnds( }; } -/** - * The most edits one shortest edit script of a remade line's changed words - * looks for (see commonWords): past it, words the two texts share anchor the - * rest, so the diff stays linear in the line's length. - */ -const MAX_EDITS = 128; - /** White space other than a line feed (a line break is a node of its own). */ const isSeparator = (s: string, i: number): boolean => s[i] !== '\n' && SPACE.test(s[i]); @@ -829,201 +822,154 @@ function words(s: string, from: number, to: number): [number, number][] { } /** - * The tokens equal in `a` and `b` that a shortest edit script keeps, as - * pairs of indexes, in order: Myers' diff, in O((n + m) * max) time, or - * null past `max` edits. + * The most edits the exact diff of a remade line's words looks for, among + * the words both texts have (see commonWords). It finds the longest common + * run of words in O((n + m) * EXACT_EDITS) time at most (about 30 ms for a + * 5,000-word line); past it, only the common start and end are kept. + */ +const EXACT_EDITS = 2048; + +/** + * Myers' middle snake of `a[aLo, aHi)` and `b[bLo, bHi)` (both non-empty, + * different first and last words): `[x, y, u, v]`, relative to `aLo` and + * `bLo`, a run of equal words from (x, y) to (u, v) on a shortest edit + * script, or null past `max` edits. */ -function commonTokens( +function middleSnake( a: readonly number[], + aLo: number, + aHi: number, b: readonly number[], + bLo: number, + bHi: number, max: number, -): [number, number][] | null { - const n = a.length; - const m = b.length; - const off = max + 1; - const v = new Int32Array(2 * off + 1); - const trace: Int32Array[] = []; - for (let d = 0; d <= max; d++) { - trace.push(v.slice()); +): [number, number, number, number] | null { + const n = aHi - aLo; + const m = bHi - bLo; + const delta = n - m; + const odd = (delta & 1) !== 0; + const half = Math.ceil((n + m) / 2); + const off = half + 1; + const vf = new Int32Array(2 * half + 3); + const vb = new Int32Array(2 * half + 3); + for (let d = 0; d <= half; d++) { + if (2 * d - 1 > max) { + return null; + } for (let k = -d; k <= d; k += 2) { let x = - k === -d || (k !== d && v[off + k - 1] < v[off + k + 1]) - ? v[off + k + 1] - : v[off + k - 1] + 1; + k === -d || (k !== d && vf[off + k - 1] < vf[off + k + 1]) + ? vf[off + k + 1] + : vf[off + k - 1] + 1; let y = x - k; - while (x < n && y < m && a[x] === b[y]) { + const x0 = x; + const y0 = y; + while (x < n && y < m && a[aLo + x] === b[bLo + y]) { x++; y++; } - v[off + k] = x; - if (x >= n && y >= m) { - // Walk back through the trace, collecting the diagonals. - const pairs: [number, number][] = []; - let cx = n; - let cy = m; - for (let e = d; e > 0; e--) { - const pv = trace[e]; - const ck = cx - cy; - const pk = - ck === -e || (ck !== e && pv[off + ck - 1] < pv[off + ck + 1]) - ? ck + 1 - : ck - 1; - const px = pv[off + pk]; - const py = px - pk; - const sx = pk === ck + 1 ? px : px + 1; - const sy = sx - ck; - while (cx > sx && cy > sy) { - pairs.push([--cx, --cy]); - } - cx = px; - cy = py; - } - while (cx > 0 && cy > 0) { - pairs.push([--cx, --cy]); - } - return pairs.reverse(); + vf[off + k] = x; + const c = delta - k; + if (odd && c >= -(d - 1) && c <= d - 1 && x + vb[off + c] >= n) { + return [x0, y0, x, y]; } } - } - return null; -} - -/** - * The longest run of `pairs` (in order of their second index) whose first - * indexes increase too: patience sorting, O(k log k). - */ -function increasing(pairs: readonly [number, number][]): [number, number][] { - const tops: number[] = []; - const prev = new Int32Array(pairs.length).fill(-1); - for (let i = 0; i < pairs.length; i++) { - let lo = 0; - let hi = tops.length; - while (lo < hi) { - const mid = (lo + hi) >> 1; - if (pairs[tops[mid]][0] < pairs[i][0]) { - lo = mid + 1; - } else { - hi = mid; + for (let c = -d; c <= d; c += 2) { + let x = + c === -d || (c !== d && vb[off + c - 1] < vb[off + c + 1]) + ? vb[off + c + 1] + : vb[off + c - 1] + 1; + let y = x - c; + const x0 = x; + const y0 = y; + while (x < n && y < m && a[aHi - 1 - x] === b[bHi - 1 - y]) { + x++; + y++; + } + vb[off + c] = x; + const k = delta - c; + if (!odd && k >= -d && k <= d && x + vf[off + k] >= n) { + return [n - x, m - y, n - x0, m - y0]; } } - prev[i] = lo > 0 ? tops[lo - 1] : -1; - tops[lo] = i; } - const out: [number, number][] = []; - for (let i = tops.length > 0 ? tops[tops.length - 1] : -1; i >= 0; ) { - out.push(pairs[i]); - i = prev[i]; - } - return out.reverse(); + return null; } /** - * Pairs of the same word in `a` and `b` from `a0` to `a1` and `b0` to `b1`, - * in order of `b`: with `unique`, the words that occur once in each; else - * every word that occurs in both, its k-th occurrence in `a` with its k-th - * in `b`. + * The pairs of a longest common subsequence of `a[aLo, aHi)` and + * `b[bLo, bHi)`, in order, into `out` (Myers' linear space diff); false + * past `max` edits. */ -function wordPairs( +function exactWords( a: readonly number[], + aLo: number, + aHi: number, b: readonly number[], - [a0, a1, b0, b1]: [number, number, number, number], - unique: boolean, -): [number, number][] { - const inA = new Map(); - for (let x = a0; x < a1; x++) { - const at = inA.get(a[x]); - if (at === undefined) { - inA.set(a[x], [x]); - } else { - at.push(x); - } + bLo: number, + bHi: number, + max: number, + out: [number, number][], +): boolean { + while (aLo < aHi && bLo < bHi && a[aLo] === b[bLo]) { + out.push([aLo++, bLo++]); } - const seen = new Map(); - for (let y = b0; y < b1; y++) { - seen.set(b[y], (seen.get(b[y]) ?? 0) + 1); + let a1 = aHi; + let b1 = bHi; + while (a1 > aLo && b1 > bLo && a[a1 - 1] === b[b1 - 1]) { + a1--; + b1--; } - const out: [number, number][] = []; - const nth = new Map(); - for (let y = b0; y < b1; y++) { - const at = inA.get(b[y]); - if ( - at === undefined || - (unique && (at.length > 1 || seen.get(b[y])! > 1)) - ) { - continue; + if (aLo < a1 && bLo < b1) { + const snake = middleSnake(a, aLo, a1, b, bLo, b1, max); + if (snake === null) { + return false; } - const k = nth.get(b[y]) ?? 0; - nth.set(b[y], k + 1); - if (k < at.length) { - out.push([at[k], y]); + const [x, y, u, v] = snake; + exactWords(a, aLo, aLo + x, b, bLo, bLo + y, max, out); + for (let i = 0; i < u - x; i++) { + out.push([aLo + x + i, bLo + y + i]); } + exactWords(a, aLo + u, a1, b, bLo + v, b1, max, out); } - return out; + for (let i = 0; a1 + i < aHi; i++) { + out.push([a1 + i, b1 + i]); + } + return true; } /** - * The words equal in `a` and `b` that are kept, as pairs of indexes, in - * increasing order of both: their common start and end, then a shortest edit - * script of the rest (commonTokens) where it takes at most MAX_EDITS edits. - * Past that, the words that occur once in each text anchor the rest, in the - * longest run of them in the same order in both (patience diff), and - * between two anchors, past MAX_EDITS edits again, so does every word that - * occurs in both, its k-th occurrence in one text paired with its k-th in - * the other (a run of repeated words with other words changed between them, - * `old echo old echo` made again as `new echo new echo`, keeps every - * `echo`). What is kept is a common subsequence of the two texts, so a kept - * word keeps its characters; a word is written again only where it is not - * paired. O((n + m) * MAX_EDITS + (n + m) log(n + m)). + * The words equal in `a` and `b` that a remade line keeps, as pairs of + * indexes in increasing order of both: a longest common subsequence of the + * two (exactWords), found among the words both texts have (a word only one + * has is never kept, and leaving it out keeps the diff short). Past + * EXACT_EDITS edits among those, none. */ function commonWords( a: readonly number[], b: readonly number[], - level = 0, ): [number, number][] { - const out: [number, number][] = []; - let a0 = 0; - let b0 = 0; - let a1 = a.length; - let b1 = b.length; - while (a0 < a1 && b0 < b1 && a[a0] === b[b0]) { - out.push([a0++, b0++]); - } - const tail: [number, number][] = []; - while (a1 > a0 && b1 > b0 && a[a1 - 1] === b[b1 - 1]) { - tail.push([--a1, --b1]); + const inA = new Set(a); + const inB = new Set(b); + const fa: number[] = []; + const fb: number[] = []; + for (let x = 0; x < a.length; x++) { + if (inB.has(a[x])) { + fa.push(x); + } } - if (a0 < a1 && b0 < b1) { - const pairs = commonTokens(a.slice(a0, a1), b.slice(b0, b1), MAX_EDITS); - if (pairs !== null) { - for (const [x, y] of pairs) { - out.push([x + a0, y + b0]); - } - } else if (level < 2) { - const anchors = increasing( - wordPairs(a, b, [a0, a1, b0, b1], level === 0), - ); - let x = a0; - let y = b0; - for (const [ax, by] of [...anchors, [a1, b1] as [number, number]]) { - for (const [px, py] of commonWords( - a.slice(x, ax), - b.slice(y, by), - level + 1, - )) { - out.push([px + x, py + y]); - } - if (ax < a1) { - out.push([ax, by]); - } - x = ax + 1; - y = by + 1; - } + for (let y = 0; y < b.length; y++) { + if (inA.has(b[y])) { + fb.push(y); } } - for (let i = tail.length - 1; i >= 0; i--) { - out.push(tail[i]); + const sa = fa.map(x => a[x]); + const sb = fb.map(y => b[y]); + const pairs: [number, number][] = []; + if (!exactWords(sa, 0, sa.length, sb, 0, sb.length, EXACT_EDITS, pairs)) { + return []; } - return out; + return pairs.map(([x, y]) => [fa[x], fb[y]]); } /**