Skip to content

icy_engine / icy_parser_core hardening - #197

Merged
mkrueger merged 9 commits into
mkrueger:masterfrom
NuSkooler:pr/engine-hardening
Oct 27, 2026
Merged

mkrueger merged 9 commits into
mkrueger:masterfrom
NuSkooler:pr/engine-hardening

Conversation

@NuSkooler

Copy link
Copy Markdown
Contributor

This PR collects fixes for panics, hangs and unbounded memory use in icy_engine and icy_parser_core. I found them while building enig-term, a browser BBS terminal that compiles icy_engine to WASM, mostly by fuzzing it. On wasm32 a panic or an allocation failure aborts the whole instance, so each of these lost the session. Each fix is its own commit, and all but the last come with a test that fails without it. Valid input behaves as before.

Bound nested CTerm macro expansion

060fed7 stopped a macro from invoking itself, but macros that invoke each other can still multiply their output at every level. With macro 0 printing 1000 bytes, macro 1 invoking macro 0 1000 times and macro 2 invoking macro 1 10 times, CSI 2 * z expands to 10 MB, and each extra level multiplies that again.

Each top-level invocation now gets a budget of 4 * MAX_MACRO_LEN expanded bytes, which nested invocations draw from. Once it runs out, the rest of the expansion is dropped. Depth was already bounded at 64 by the running-macro bits, so there is no separate depth limit. The budget size is a guess at "far more than any real use"; change it if you have a better number.

Test: test_nested_macro_expansion_is_bounded in icy_parser_core/tests/ansi/dcs.rs.

Clamp scroll margins and scroll short lines left and right safely

scroll_left and scroll_right in TextScreen indexed every line of the scroll region as a full row. After a clear the lines are gone, so ESC [ 2 J ESC [ SP @ panicked, and so did any line shorter than the right margin. Missing lines and cells are now treated as blank.

Margins were also stored as sent, so ESC [ 1 ; 9999 r, or a resize, left margins outside the screen for later row and column loops. They are now limited to the screen when set, on set_size/set_width/set_height, and when a saved text window is restored. Like xterm, a bottom or right margin past the screen becomes the last row or column; a margin that starts outside the screen is dropped. The DECSLRM handler already ignored out-of-range values and is unchanged.

Tests: test_horizontal_scroll_handles_cleared_and_short_lines and test_margins_are_clamped_to_the_screen in icy_engine/tests/text_screen.rs.

Reject empty data in BitFont::from_bytes

libyaff accepts an empty document, so BitFont::from_bytes(&[]) returned a blank 8x16 font. An empty CTerm font load (ESC P CTerm:Font:43: ESC \) then made text in that slot disappear. Empty data is now UnknownFontFormat.

Test: from_bytes_rejects_empty_data in fonts/mod.rs.

Keep a DCS-loaded font selected in CSI SP D

Selecting a slot holding a DCS-loaded font found it and reported success, then found no built-in font for the slot and overwrote the state with Failure. The caret did switch fonts, but CSI = 1 n said the selection failed. Failure is now reported only when the slot has no font at all.

Test: test_selecting_a_dcs_loaded_font_slot_succeeds in icy_engine/tests/text_screen.rs.

Check the slot and height of DCS font loads

A CTerm DCS font load went into whatever slot the host named. Slot 0 replaced the font that sets the cell size, and set_font cuts the slot to a u8, so slot 256 did the same. Loads are now accepted only for slots 43-255, above the built-in fonts.

Font heights outside 1-32 rows panic the renderer (glyph.data holds 32 rows, and a height of 0 gives a zero chunk size). The PSF1 parser now caps the height at 32 as the PSF2 parser does. Raw font data is only taken for 1-32 rows per glyph: 256 rows per glyph wrapped the u8 height to 0, and above 32 create_8 would have read the glyphs with the wrong stride, so I rejected those rather than cut them. The DCS handler also rejects any font outside 1-32 rows. MAX_GLYPH_HEIGHT is now exported for that check.

CSI SP D also cut the font number to a u8, so selecting 257 picked the built-in font in slot 1. Numbers above 255 now fail.

Tests: test_dcs_font_load_checks_slot_and_height and test_font_selection_above_255_does_not_wrap in icy_engine/tests/text_screen.rs.

Check inline image dimensions before decoding

MAX_PIXELS was checked only after an inline image had been decoded, so a few bytes claiming a huge image made the decoder allocate the full pixel buffer first. On wasm32 that ran out of memory. The dimensions are now read from the header (ImageReader::into_dimensions for PNM, the JxlDecoder header for JXL, which JxlDecoder::new already parses) and checked before decoding.

I left the decoders' own limits alone. JxlDecoder::new sets its allocation tracker to usize::MAX, so passing image::Limits to it would also bound jxl-oxide's working memory, but I wasn't sure what a large valid JXL needs, so that's your call.

Test: tests/image_memory.rs, its own test binary with a global allocator that records the largest allocation. A PNM header claiming 8000x8000 with no data used to allocate 64 MB before being rejected.

Trim the scrollback when its size shrinks, and skip it at size 0

Every line that scrolls off is rendered to an RGBA strip, about 40 KB per line on an 80 column screen. With a scrollback size of 0 the strip was still rendered and then thrown away, and lowering the size kept everything already stored, since push_chunk removed only one chunk per push. In a long session enig-term reached about 1 GB of WASM memory. Rendering is now skipped at size 0, and the buffer is trimmed whenever the size is set or a chunk is pushed.

Test: test_scrollback_size_limits_stored_chunks in icy_engine/tests/text_screen.rs. The trim part fails without the fix; the size 0 part only checks that nothing is stored, which was already true, since the saving there is the render.

For embedders, a cell-based scrollback or a callback for each line that scrolls off would help more than the pixel strips. enig-term keeps its own cell scrollback now, so a hook would be enough for us. That's a bigger change and not part of this PR.

Drop OSC, DCS and APC strings that exceed the length cap

Once a string reached MAX_STRING_LEN the rest was dropped, but at ST the first 32 MiB were dispatched as if they were the whole string. A large Sixel was drawn cut off, and a CTerm audio upload (APC SyncTERM:C;S;...) was stored as a corrupt file with no error. The parser now remembers that the string overflowed, and at ST reports a MalformedSequence error and drops it. test_unterminated_dcs_payload_is_capped expected the old behaviour and is updated; the test CollectSink now records errors.

Test: test_oversized_apc_and_osc_strings_are_dropped in icy_parser_core/tests/ansi/dcs.rs, plus the updated test_unterminated_dcs_payload_is_capped.

Use web-time for the synchronized output clock

std::time::Instant::now panics on wasm32-unknown-unknown, so ESC [ ? 2026 h aborted a WASM build. terminal_state.rs now uses web_time::Instant, which is std::time::Instant on native targets and performance.now() on wasm32. web-time is already a dependency of icy_engine_scripting and icy_mail.

Test: test_synchronized_output_mode_starts_and_ends covers the mode natively. The panic itself only happens on wasm32, which cargo test doesn't run, so this one has no test that fails without the fix.

cargo test -p icy_engine -p icy_parser_core, cargo fmt --check and cargo check --workspace pass, and clippy shows no new warnings in either crate.

Macros that invoke each other could multiply their output at every
level, so a few KB of macro definitions expanded into gigabytes. Give
each top-level invocation a budget of expanded bytes that nested
invocations draw from, and drop the rest once it runs out.
CSI SP @ and CSI SP A indexed every line of the scroll region as a full
row, so they panicked on lines that were missing after a clear or
shorter than the right margin. Treat missing lines and cells as blank.

Margins were stored as sent, so CSI 1;9999 r or a resize left margins
outside the screen. Limit them to the screen when they are set, and
again on a resize or when a saved text window is restored, dropping a
margin that no longer fits.
libyaff accepts an empty document, so BitFont::from_bytes(&[]) returned
a blank 8x16 font instead of an error. An empty font file, or a CTerm
DCS font load with an empty payload (ESC P CTerm:Font:43: ESC \), then
loaded a font that draws nothing, and text in that slot disappeared.

Return UnknownFontFormat for empty data, before any parser is tried.
Selecting a slot that holds a DCS-loaded font (43 and up) found the
font and reported success, then looked for a built-in font for the same
slot, found none and overwrote the state with Failure. The caret did
switch to the font, but the CSI = 1 n font state report said the
selection failed, so a host could not tell that the font was in use.

Only report failure when the slot has no font at all.
A CTerm DCS font load was stored in whatever slot the host named.
Slot 0 replaced the font that sets the cell size, and set_font cut the
slot to a u8, so slot 256 did the same. Only accept slots 43-255, above
the built-in fonts.

Font heights outside 1-32 rows panic the renderer. The PSF1 parser now
caps the height at 32 like the PSF2 parser, raw font data is only taken
for 1-32 rows per glyph (256 rows wrapped the height to 0), and the DCS
handler rejects any font outside that range.

CSI SP D also cut the font number to a u8, so selecting 257 picked the
built-in font in slot 1. Numbers above 255 now fail.
MAX_PIXELS was checked only after an inline image had been decoded, so
a few bytes claiming a huge image made the decoder allocate the full
pixel buffer first. Read the dimensions from the header and reject the
image before decoding it.
Every line that scrolls off is rendered to an RGBA strip for the
scrollback, about 40 KB per line on an 80 column screen. With a
scrollback size of 0 the strip was still rendered and then thrown
away, and lowering the size kept everything already stored, since
push_chunk removed only one chunk per push.

Skip rendering when the size is 0, and trim to the size whenever it is
set or a chunk is pushed.
Once a string reached MAX_STRING_LEN the rest was dropped, but at ST
the first 32 MiB were still dispatched as if they were the whole
string. A large Sixel was drawn cut off, and a CTerm audio upload was
stored as a corrupt file with no error.

Remember that the string overflowed, and at ST report a
MalformedSequence error and drop it instead of dispatching it.
std::time::Instant::now panics on wasm32-unknown-unknown, so a host
sending CSI ? 2026 h aborted a WASM build of icy_engine. web_time's
Instant is std::time::Instant on native targets and uses
performance.now() on wasm32, so native behaviour is unchanged.
@mkrueger
mkrueger merged commit 745f252 into mkrueger:master Oct 27, 2026
1 check failed
mkrueger added a commit that referenced this pull request Oct 27, 2026
Add allocation regression coverage for rejected nested invocations and budget reset between top-level calls. Follow up on #197.
NuSkooler added a commit to NuSkooler/enig-term that referenced this pull request Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants