FOP-2918: A surrogate pair is not split by word breaking, and both its units get one bidi level

[plutext]

Addresses FOP-2918, open since 2020 with a patch from kwilkerson attached. This carries that patch's two changes, the issue's own layout test, a second case the issue did not have, and the root cause underneath both.

What goes wrong

A character outside the BMP is two UTF-16 units. TextLayoutManager ends a word where the bidi level changes or, with font-selection-strategy="character-by-character", where the selected font changes. Either could fall between the two units of a pair, and MultiByteFont.mapCharsToGlyphs then rejected the fragment:

java.lang.IllegalArgumentException: ill-formed UTF-16 sequence, contains isolated high surrogate at end of sequence

No PDF is produced.

The commits

1. and 2. The word-splitting guard on the bidi-level branch and the CharUtilities.containsSurrogatePairAt bound (> to >=). These are the same two changes as 2918.patch; they reached this branch by way of the Metanorma fork, whose commits are kept with their author.
2. containsSurrogatePairAt raises the documented IllegalArgumentException for a high surrogate at the end of a sequence, with a test.
3. The same guard on the font-selection branch, which the issue did not cover: an emoji under character-by-character font selection (Aegean600 in the test tree) failed the same way. PDFEncodingTestCase renders it.
4. The root cause. UnicodeBidiAlgorithm.resolveLevels(CharSequence, Direction) leaves a placeholder in the slot of each low surrogate, and its javadoc says both members of a pair come back with the same level. They did not: no rule resolves the placeholder's class, so it stayed at the embedding level while its character took the level of its script. U+10826 (Cypriot, right-to-left) came back as levels 1 and 0, which is why the word was split there in the first place. With the guards alone the word is kept whole and the line's reordering then asserts in InlineRun.split ("heterogeneous inlines not yet supported"). Each placeholder now takes the level of the character before it. SurrogatePairLevelsTestCase holds it.
5. wordbreak_surrogates.xml, the layout test from 2918.patch, unchanged, credited to kwilkerson. It fails on main with the exception above, fails with the guards alone on the assertion, and passes with commit 5.
6. The placeholder's class. Commit 5 copies the level after resolution, which cannot help where the resolution itself is wrong. The placeholder had a class of its own (SURROGATE) that is neither strong nor neutral, so rule N1's look-ahead stopped at it. A neutral outside the BMP inside right-to-left text (U+1F300 between two Hebrew words, in a left-to-right paragraph) therefore never saw the strong text after it, fell to the embedding direction under N2 and resolved to level 0, cutting the right-to-left run in two, where U+263A in the same place resolves to 1. The placeholder now takes the class of the character it belongs to, so the rules see the pair as one character. Outside the BMP Unicode assigns no ES, CS, WS, S, B or explicit-embedding class, so a repeated class resolves as a single one under every rule; commit 5's copy stays as the guarantee the javadoc makes. SurrogatePairLevelsTestCase gains two tests; the neutral one fails with commit 5 alone.

Not addressed here: BidiClass was generated from Unicode data older than 6.1, so U+1F600 and many later characters come back as L (U+1F300, from 6.0, is right). That is why the test uses U+1F300. It is a separate matter, regenerating the table.

Tests

SurrogatePairLevelsTestCase, CharUtilitiesTestCase, PDFEncodingTestCase, and the layout engine test wordbreak_surrogates.xml.

🤖 Generated with Claude Code

[plutext]

Added a seventh commit. Commit 5's level copy fixes a right-to-left character outside the BMP, but not a neutral one: the low surrogate's placeholder had a class of its own, which ended rule N1's run of neutrals, so an emoji (U+1F300 here) between two Hebrew words resolved to the embedding direction and cut the right-to-left run in two, where U+263A resolves with the text. The placeholder now takes its character's class. Two tests added; the neutral one fails on commit 5 alone. Full build (mvn -B package checkstyle:check spotbugs:check) green on main.