fix(core): stop chars ranges leaking across patterns and shared tokens

[pablofdezr]

Summary

Addresses the range-inheritance part of #169: ranged char highlights
(/foo/2, /foo/#a) leaking onto the wrong characters.

1. Keep ranges aligned with patterns

charsList always got an entry, but the parallel ranges array was pushed only
when a numeric range existed. An id-only annotation such as /foo/#a therefore
made the next pattern inherit its range.

The parser now always pushes an aligned range entry, using an empty array when
there is no numeric range.

2. Count excluded occurrences without changing the token tree

The matcher tracks occupied source offsets separately from the rendered HAST. A
range-excluded occurrence still increments the counter and prevents overlapping
patterns from claiming the same characters, but it is not split into additional
spans.

The previous rescan-based matcher could join text around visited fragments,
fabricate matches that were not present in the source, miscount later ranges,
and splice non-contiguous siblings as though they were adjacent. Matching from
stable source offsets removes that class of source corruption and makes repeated
cross-token matches exact.

Selected matches are grouped by their overlapping top-level token ranges, then
each group is applied from right to left. This avoids repeatedly rescanning the
whole line while onVisitHighlightedChars remains in the original
pattern/occurrence order.

3. Preserve transformer-owned markup

A partial match contained within a custom or transformer-generated element now
inserts the highlight at the matching text node instead of cloning semantic
ancestors. Ordinary one-text Shiki spans use a smaller linear split path: the
matched fragment reuses the original node, leftover fragments do not duplicate
its id, and array-valued properties are copied so callback mutations cannot
leak into unhighlighted fragments.

Cross-token matches may also include a transformer-owned boundary unchanged when
that boundary is selected in full. A boundary that would need to be partially
split remains intentionally unsupported.

Result

Meta / case	Problem before the final implementation	Final result
/get/1 /Length/ (2 identical lines)	master dropped line 2 Length	Length on both lines; get only 1st
/get/#a /Length/2	master swapped the ranges	get everywhere; Length only 2nd
/con/ on repeated text such as coconnection	master could fabricate a cross-gap match and corrupt the source	exact match with source preserved
/a/50000 over 50,000 as	an intermediate split-every-exclusion revision produced 50,000 fragments and 1.60 MB HTML	2 children and 50 KB HTML
cross-token split of identified spans	master could duplicate DOM IDs	each original ID remains unique
cross-token match with a whole transformer boundary	an intermediate batched revision dropped the match	match and semantic boundary are both preserved

The first row is the exact example from the issue.

Regression coverage

• Range and pattern alignment.
• Exact repeated and ordinary Shiki cross-token matching.
• Contained nested-token matching, including comments and auxiliary children.
• Transformer-owned interactive elements are not cloned.
• Whole transformer-owned first and last cross-token boundaries are preserved.
• Decorated whole-token shape and callback compatibility.
• Transformer property arrays stay isolated across split fragments.
• Split token IDs remain unique, including adjacent cross-token matches.
• Range-excluded occurrences do not materialize AST fragments.
• Existing snapshots retain the original unhighlighted Shiki spans.

The final adversarial pass checked 20,000 randomized span-tree cases covering
IDs, property isolation, overlaps, sparse/shared counters, callbacks, source
preservation, and mark ordering, with zero mismatches.

Performance

The tables below were collected on 608e568, the batching revision immediately
before the merged head (279702a). That final follow-up only extended
cross-token application to boundaries selected in full; affected cases were
rebenchmarked with no material regression. The matrix compares current master,
the original PR head (58d068d), and 608e568 across 41 direct highlighter +
HAST-serialization cases and 10 public Shiki pipeline cases. Each row reports a
warmed median.

Public Shiki pipeline matrix (10 cases)

Case	master	58d068d	608e568
no match, 100k-character source	18.770 ms	17.695 ms	18.098 ms
dense 8k matches	1,004.246 ms	56.001 ms	47.765 ms
decorated-token dense 8k	1,031.073 ms	60.110 ms	51.146 ms
sparse 50,000th occurrence	9.964 ms (0 marks)	867.914 ms	10.602 ms
500 cross-token line matches	33.366 ms	38.028 ms	31.356 ms
two patterns across 500 lines	34.083 ms	32.525 ms	32.783 ms
globally sparse final line	29.353 ms	26.379 ms	27.341 ms
overlapping aa before a, 8k chars	294.675 ms	24.236 ms	21.975 ms
2k emoji matches	86.256 ms	11.024 ms	10.497 ms
1k 16-character matches	26.255 ms	7.721 ms	8.086 ms

Representative direct highlighter matrix

Case	master	58d068d	608e568
no match, 1M-character token	0.543 ms	0.649 ms	0.537 ms
dense 20k single-token matches	7,048.077 ms	90.482 ms	61.369 ms
decorated + identified dense 4k	252.335 ms	11.910 ms	10.295 ms
sparse first of 50k	1.715 ms	405.431 ms	1.411 ms
sparse middle of 50k	0.611 ms (0 marks)	409.785 ms	1.394 ms
sparse last of 50k	0.573 ms (0 marks)	408.997 ms	1.425 ms
missing selected occurrence in 50k	0.641 ms	408.532 ms	1.376 ms
every 100th occurrence in 10k	0.110 ms (0 marks)	25.423 ms	0.665 ms
dense 5k sibling spans	692.405 ms	8.972 ms	8.405 ms
1k cross-token ab matches	54.519 ms	3.229 ms	3.474 ms
2k four-token abcd matches	411.983 ms	11.582 ms	9.680 ms
1k identified cross-token matches	59.123 ms	4.478 ms	4.422 ms
overlapping a before aa, 10k chars	1,571.396 ms	33.815 ms	21.173 ms
duplicate ranged pattern in 10k chars	0.064 ms (0 marks)	9.237 ms	0.196 ms
globally sparse final match across 5 lines	0.066 ms (0 marks)	6.282 ms	0.155 ms
callback mutating split class arrays, 5k	730.092 ms	327.579 ms	12.147 ms

The remaining direct cases covered empty/unmatched patterns, dense and sparse
sibling spans, whole-token matches, long cross-token groups, nested tokens,
direct/mixed text children, comments, emoji and combining Unicode, both overlap
orders, duplicate and 100-pattern workloads, multi-line counters, and callback
mutation. Expected counts and output sizes matched wherever the behavior is
supported.

Rows marked 0 marks are not speed wins for master: it did not produce the
requested highlight, so those timings perform less work and are not directly
comparable.

Ordinary real-world documentation corpora were effectively neutral (generally
within about ±2% benchmark noise). The large wins occur in dense or repetitive
character-annotation workloads, where avoiding repeated tree rescans and
fragmentation improves performance by roughly 3–115× in the measured cases.
Scaling remained close to linear; no quadratic explosion was observed.

The intentionally removed pathological shapes were measured too rather than
hidden: 1,000 matches inside a single 12-level wrapper or semantic link are
slower. A cross-token match that would require partially splitting a nested
transformer-owned boundary remains unsupported; a boundary selected in full is
preserved. Those combined shapes were excluded to keep the realistic Shiki path
small and linear.

Bundle size

The production-build table was measured at 608e568 against the same master:

Artifact	master	608e568	Delta
ESM output	25,184 B	26,187 B	+1,003 B
gzip	5,935 B	6,192 B	+257 B
downstream minified + gzip	4,164 B	4,281 B	+117 B

The final boundary-preservation follow-up added 56 minified bytes / 23
minified+gzip bytes in the downstream measurement. The batching revision was
also 650 raw bytes / 102 gzip bytes smaller than the previously pushed broad
recursive correctness fix (f487d89), which was removed in favor of the focused
Shiki-span path.

Validation

• pnpm test — 56/56 passing on the merged head.
• Core and transformer package typechecks.
• Biome check on all changed source/test/changeset files.
• Core production build.
• 20,000-case randomized adversarial oracle with zero mismatches.
• Public corpus validation: 6,164 blocks, 117,056 code lines, and 123 character annotations with no rendered-output regression.
• Final correctness and 51-case performance passes.
• GitHub Checks and Preview passed on the merged head.

The changeset remains a patch because this is a bug fix.

Out of scope

Highlighting a pattern that is a substring of another highlighted pattern
(/getStringLength/ /get/ — the other half of #169) is a distinct
nested-highlight feature and is intentionally not addressed here, so this PR is
marked Refs rather than Closes.

[changeset-bot]

🦋 Changeset detected

Latest commit: 279702a

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
rehype-pretty-code	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[pkg-pr-new]

npm i https://pkg.pr.new/rehype-pretty-code@276

npm i https://pkg.pr.new/@rehype-pretty/transformers@276

commit: 279702a

[atomiks]

PR review

The range alignment and ignored-occurrence splitting are mechanism-correct and adequately covered. No source-code regression survived review; only release metadata is missing.

Docs (1)

1. 🟠 Add a patch changeset

Location: packages/core/src/index.ts:395

charsListNumbers.push(range ? rangeParser(range) : []);

This fixes published character-range behavior but has no changeset.

Failure scenario: The range-inheritance fix merges without triggering a new rehype-pretty-code version.

Fix: Add a changeset declaring "rehype-pretty-code": patch.

Verdict

Approve after nits - the implementation is sound, with only the required release entry outstanding.

---

🤖 Review generated with Codex

[atomiks]

PR review

Both mechanisms in this PR are correct and I verified each against master — the index-alignment fix and the ignored-occurrence splitting each recover a highlight that was previously dropped. Nothing is merge-blocking. What remains is that the leak class this PR targets is narrowed rather than closed, and that the change rewrites six committed result fixtures in a way consumers may notice.

Bugs (1)

1. 🟠 A range-ignored occurrence still blocks later patterns when no split is needed

Location: packages/core/src/chars/wrapHighlightedChars.ts:21

if (ignoreWord) {
  if (element.properties) {
    element.properties['rehype-pretty-code-visited'] = '';
  }
  return;
}

rehype-pretty-code-visited is shared across every pattern in charsList, and charsHighlighter only clears it after the whole charsList.forEach completes. Splitting the element first — the fix in this PR — shrinks the region that gets poisoned from the whole token down to the matched substring, which is exactly why /Length/1 /get/ now works. But when the matched text is the whole element (content === chars in getElementsToHighlight), no split happens and the original leak survives unchanged.

Verified on this branch and on master, identical output on both:

```js /getString/1 /String/
const getStringLength = 1;
const getStringWidth = 2;
```

Marks produced: ["getString"]. /String/ has no range, so it should highlight String on both lines; it highlights nothing, because /getString/1 marked line 2's token visited when it declined to highlight it.

This is pre-existing rather than a regression, and the real fix — a per-pattern skip marker distinct from rehype-pretty-code-visited, cleared at the end of each pattern's iteration — is not a small change. I prototyped it and it moves five committed snapshots, so it clearly belongs in its own PR.

Failure scenario: /getString/1 /String/ silently drops every String highlight, including on lines the range never restricted.

Fix: Nothing to change here — this PR is a strict improvement. Worth opening a follow-up issue so the remaining case is not mistaken for fixed by #169.

Docs (1)

1. 🟠 The changeset does not mention the HTML output churn

Location: .changeset/lucky-clocks-thank.md:5

fix: stop character-highlight ranges leaking across patterns, so an id-only annotation like `/foo/#a` no longer makes a later pattern inherit the previous pattern's range, and a range-ignored occurrence no longer consumes the whole token it lives in (#169)

Splitting ignored occurrences means tokens that previously stayed intact now emit sibling spans carrying identical styles:

<!-- before -->
<span style="color:#79B8FF">rotcar</span>
<!-- after -->
<span style="color:#79B8FF">rot</span><span style="color:#79B8FF">car</span>

Six committed result fixtures change for this reason. The rendering is unaffected, but the emitted HTML is not.

Failure scenario: A consumer with HTML snapshot tests over their rendered docs takes a patch bump and gets diffs across every code block that uses a character range, with a changeset that gives no hint the markup changed.

Fix: Add a sentence noting that ignored occurrences are now split into separate spans, so the emitted HTML changes even where the visible highlighting does not.

Verdict

Approve after nits - both fixes are verified correct against master; only the residual leak (pre-existing) and the unmentioned markup churn remain.

---

🤖 Review generated with Claude Code

[pablofdezr]

Added the patch changeset — the only outstanding item, since the review found nothing to change in the source.

The entry covers both halves of the fix: the range/pattern misalignment that let an id-only annotation such as /foo/#a hand its range to the next pattern, and the range-ignored occurrence that consumed the whole token it lived in. Scoped as "rehype-pretty-code": patch.

Nothing else in this push, so the snapshots stand as reviewed and the fixture suite is still green.

[atomiks]

Before this year these would've been merged already 😅. But now models keep finding issues that the back-and-forth takes forever.

PR review

This contains two regressions missed by the existing 38-test suite. One can corrupt the rendered code itself and should block merging.

Bugs (2)

1. 🔴 Ignored matches can corrupt the rendered code text

Location: packages/core/src/chars/getElementsToHighlight.ts:127

charsSoFar += overlap;

toWrap.push({
  element: newElement,
  index: updatedIndex,
});

Splitting an ignored occurrence exposes its fragments to subsequent scans. The matcher can retain partial state across a visited or non-continuing sibling and return non-contiguous elements. wrapHighlightedChars assumes the returned elements are contiguous and splices elementsToWrap.length siblings from the first index, deleting and duplicating unrelated text.

Failure scenario:

```js /con/2
coconnection;
```

There is only one con, so nothing should be highlighted or modified. The PR serializes the code as coconneconon;, deleting ti, duplicating on, and marking a synthetic non-contiguous con. master preserves coconnection;.

Fix: Treat visited or non-continuing siblings as hard boundaries, reset partial-match state across gaps, and ensure getElementsToHighlight only returns contiguous elements spelling the requested pattern.

2. 🟠 Failed synthetic matches advance the range counter

Location: packages/core/src/chars/charsHighlighter.ts:45

options.counterMap.set(id, (options.counterMap.get(id) || 0) + 1);

const ignoreChars =
  currentCharsRange.length > 0 &&
  !currentCharsRange.includes(options.counterMap.get(id) ?? -1);

const elementsToWrap = getElementsToHighlight(element, chars, startIndex);

if (elementsToWrap.length === 0) break;

After an ignored match is split, recomputing textContent omits the visited fragment and concatenates its surrounding text. That can create a synthetic occurrence. The counter advances before the matcher confirms that the occurrence maps to real elements.

Failure scenario:

```js /lo/2
balloon;
lo;
```

The first real lo should be ignored and the second-line lo highlighted. Instead, removing the first visited fragment produces synthetic balon; its synthetic lo advances the counter to two before matching fails. The real second occurrence is then counted as three and skipped.

Fix: Increment counterMap only after accepting a complete match. Preserve a boundary for excluded visited nodes when recomputing the remaining text so disconnected fragments cannot form synthetic candidates.

Verdict

Request changes - ignored-match splitting can corrupt emitted code and miscount ranged highlights.

---

🤖 Review generated with Codex

[pablofdezr]

Both reproduce exactly as described, and both trace back to one line: the remaining text is recomputed by dropping excluded nodes, which joins their neighbours.

Once balloon is split around the ignored lo, bal + on; reads as balon; — an occurrence of lo that the code never had. The same for co + nection; reading as conection;. That phantom either advances the range counter past the real later occurrence (/lo/2), or leads the matcher across the gap so it returns non-adjacent elements, which wrapHighlightedChars then splices as if consecutive — deleting ti and duplicating on.

Four changes:

• A boundary instead of a deletion. An excluded node is a gap in the text, not an absence, so it is replaced with \0 rather than dropped. Fragments on either side can no longer spell a match across it. The progress guard now compares the recomputed text for equality instead of length, since a boundary keeps the length from shrinking for single-character patterns.
• The counter advances only on a confirmed match. counterMap is incremented after getElementsToHighlight resolves to real elements, so a candidate that never materializes cannot consume a number.
• Runs are abandoned as soon as they stop being contiguous. A visited or non-continuing sibling now discards the partial run (and the node is reconsidered as the start of a fresh one). nextElementMaybeContinuesChars treats a visited node as a non-continuation, since the scanner skips it anyway.
• Positions are read back from the tree before splicing. The recorded indices are what the matcher predicted while it was still splitting nodes, so they drift — a legitimate three-node run reports [2,3,5] for elements that actually sit at 2,3,4. Contiguity is now checked against children.indexOf, and the splice starts there; a run that is genuinely non-contiguous leaves the tree untouched instead of corrupting it.

Both cases from the review are in the fixture, and I confirmed they fail without the fix.

One thing worth flagging: the corruption is not only reachable through this PR. On master, unpatched:

```js /con/
coconnection;
```

renders as coconneconon; with two con marks — the highlighted con is dropped from the recomputed text, co + nection; join, and the phantom is wrapped by splicing siblings that were never matched. The range in your repro made it easier to hit, but the published behaviour has the same defect without one. That case is in the fixture too.

To check I had not traded these two for others, I ran 300 pattern × code combinations through master and through this branch. 295 are byte-identical. The five that differ: the /con/ corruption above (fixed), /on/2 on coconnection which master fails to highlight at all, and the three #169 cases this PR set out to fix. No case regressed.

Suite is green (38/38), snapshots unchanged apart from the new fixture, biome and tsc clean.

I also added the sentence about markup churn to the changeset — that was still outstanding from the previous round, and I had only said I'd added the changeset itself.

The residual leak you prototyped (/getString/1 /String/) is untouched here and still behaves as it does on master; happy to open the follow-up issue for it if that's useful.