Skip to content

kernel: update LZ4 compressor to upstream 1.10.0 - #501

Merged
dormancygrace merged 3 commits into
mainfrom
codex/lz4-compressor-110
Oct 27, 2026
Merged

dormancygrace merged 3 commits into
mainfrom
codex/lz4-compressor-110

Conversation

@dormancygrace

@dormancygrace dormancygrace commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Update only the kernel LZ4 compressor to upstream 1.10.0. Preserve the public kernel API, workspace size and alignment, module name, and six exported symbols. Leave the decoder and LZ4HC unchanged.

A private compression core retains the upstream algorithm, with thin wrappers adapting it to Linux. Allocating convenience APIs are omitted. The smaller zram hash table will be a separate PR.

Validation:

  • 170,114 compression cases covering alignments, bounded output, guard pages, and compatibility with the existing kernel decoder; ASan/UBSan.
  • Public API tests against an independent liblz4 decoder: dictionaries, zram-style state cloning, saveDict, and streams exceeding 2 GiB; normal and ASan/UBSan runs.
  • Linux 7.3 module build with production C906 compiler flags; the existing six exports are preserved.
  • Zero-fuzz patch application, kernel configuration gate, and platform configuration/build tests.

The native API test is included in CI. The final integration has not yet been benchmarked on the device; this PR makes no measured performance claim.

@dormancygrace dormancygrace changed the title kernel: обновить компрессор LZ4 до 1.10.0 kernel: update LZ4 compressor to upstream 1.10.0 Oct 26, 2026
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Build outputs

Compared with the build of main at 2571baa.

1 changed, 0 added, 0 removed of 570 outputs.

directory files
changed lib lib/modules/7.3.0-nanokvm-os-r3/kernel/lib/lz4/lz4_compress.ko
Hashes: outputs-sha256 artifact of this run.

@dormancygrace dormancygrace left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The compressor update looks sound.

  • The patch applies with zero fuzz to 7.3, and the PR's compressor test passes.
  • 200,000 random round-trips through the existing kernel decoder all matched.
  • The exported API is the same six symbols, and the licence headers are consistent with the deleted file.

Two comments, plus a numbering note: #484 and #492 also add 0052-*/0053-* kernel patches. build.sh applies names in sorted order, so all of them still apply, but the LZ4, riscv and cifs patches would interleave by filename. The PR that lands second should take the next free numbers. This stack already continues to 0055 in #515.

Comment thread platform/kernel/0054-lz4-update-compressor-1.10.patch
Comment thread platform/kernel/0052-lz4-update-compressor-1.10.patch Outdated

@dormancygrace dormancygrace left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review confirmed the invalid-workspace NULL dereference. Fixed it in 9cf2a34 and retired the legacy candidate generator for the new source layout. Public symbols/workspace layout remain unchanged. Normal and ASan/UBSan API/oracle tests pass; Linux 7.3 LZ4 patch-stack application and platform build/config tests pass. Compression/streaming bodies were compared to upstream v1.10.0 after namespace normalization. This is a runtime correctness correction, so the PR remains open.

@dormancygrace dormancygrace left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-checked at 9cf2a34. Both earlier points are fixed:

  • Workspace check: both ext-state entry points now check LZ4_initStream() before using the result. A NULL or misaligned workspace returns 0, and the workspace, output and *srcSizePtr are left untouched. invalid_workspace() covers NULL and all seven misaligned offsets for fast, default and destSize. #503's LZ4_compress_fast_4k has the same guard now.
  • Bench tool: variants.py and build.py fail with an explicit message on the 1.10 layout before creating any output, and test-lz4-compressor.py checks that.

One thing to do before merge: #484 has landed, so main now has 0052-riscv-native-scalar-memmove.patch and 0053-riscv-native-scalar-usercopy.patch. This PR adds a second 0052, and #503 a second 0053. The two touch different files (arch/riscv/lib/memmove.S versus lib/lz4/*), so sorted application should still work, but the numbering no longer says what order is intended. Please renumber the LZ4 stack to the next free numbers (0054 here, then #503, #505 and #515 in order).

The build check of this head started three seconds after #484 merged (22:42:49 against 22:42:46), so it is unclear whether a tree with both 0052s has been built. The CI run after renumbering will settle it. With that, this looks ready to me.

@dormancygrace dormancygrace left a comment •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the numbering follow-up in review #5660210602 at d12e7d7. The compressor patch is now 0054-lz4-update-compressor-1.10.patch, after the merged RISC-V 0052/0053 patches; its contents are byte-for-byte unchanged. The test reference is updated and current main is integrated. Clean Linux 7.3 patch/Kconfig, platform build and compressor oracle/API tests pass, including ASan/UBSan. The dependent LZ4 patches are renumbered 0055, 0056 and 0057 with their references updated and the complete chain tested. No unresolved findings; the main merge tree exactly matches this validated head.

@dormancygrace
dormancygrace merged commit 845ad75 into main Oct 27, 2026
15 checks passed
@dormancygrace
dormancygrace deleted the codex/lz4-compressor-110 branch October 27, 2026 00:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant