Skip to content

fix(extract): recover nested firmware filesystems with native decoders - #108

Merged
adi0x90 merged 2 commits into
masterfrom
fix/recursive-extraction
Sep 25, 2026
Merged

adi0x90 merged 2 commits into
masterfrom
fix/recursive-extraction

Conversation

@adi0x90

@adi0x90 adi0x90 commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Summary

Firmware extraction could stop after carving an image without recovering its root filesystem. Nested containers, older SquashFS layouts, and legacy LZMA blocks also left supported-looking inputs undecoded.

This adds recursive native recovery and explicit recovery outcomes. For example, the WNAP320 vendor ZIP now traverses ZIP → TAR → big-endian SquashFS 3.1 and recovers 938 regular files and 197 symlinks on case-sensitive storage.

Changes

  • Separate container traversal, archive handlers, filesystem decoders, output handling, and CLI reporting. SquashFS 2/3/4 readers support both byte orders and zlib, LZMA, and XZ; header, inode, codec, and cached metadata logic remain independent.
  • Bound input size, decoded output, entries, nesting, artifacts, and caches. Stage output before publication, validate paths, defer symlink creation, and reject filename collisions on case-insensitive storage.
  • Add per-artifact lineage, offsets, decoder outcomes, counts, timings, and failure details. Extend manifests and JSON with artifacts and recovery_status while preserving existing fields.
  • Continue fallback when recovery is incomplete or only intermediate artifacts were carved. Empty directories and directory names alone no longer establish rootfs recovery. Explicit --extractor native stays native.
  • Preserve the original filesystem layout and add regression coverage for nested archives, legacy formats, malformed input, limits, links, and fallback decisions.

Verification

  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace — 1,996 passed, 0 failed, 1 ignored local-fixture test
  • CLI extraction integration suite — 47 passed
  • cargo test -p firmware-analysis-toolkit --test test_kernel_profile_consistency — 4 passed
  • python3 scripts/build-runtime-data-bundle.py --check — 33 files verified
  • GitHub packaging checks — all five platform builds and archive smoke tests passed
  • Installer regression suite — 16 passed
  • Documentation links and GitHub Markdown rendering checked

Validation was repeated after integration with current master. Local firmware validation recovered rootfs trees from all 11 saved inputs and both tested vendor ZIP entry paths. Across the 11 images, 21,351 regular-file contents and symlink targets matched independent reference extraction with no missing or differing entries. ZIP entry paths reproduced the corresponding raw-input inventories.

The four previously failing raw inputs completed in 0.23–0.67 seconds per optimized CLI run on macOS arm64 with case-sensitive APFS. These are single-run measurements, not a controlled benchmark. Vendor firmware and extracted trees are not included in this PR.

Notes

  • Case-distinct names require case-sensitive output storage. WNAP320 contains both xt_MARK.ko and xt_mark.ko; a collision now reports partial recovery and does not publish that rootfs tree.
  • SquashFS LZO/LZ4/Zstd and TAR hard links/sparse entries remain unsupported. Device nodes are counted and omitted; ownership, timestamps, and extended attributes are not restored.
  • files_only and partial may still exit successfully for compatibility. Automation should inspect recovery_status and rootfs.
  • This change verifies extraction only. Guest staging, kernel availability, boot, networking, and service behavior remain outside this PR's validation.

Follow nested containers and decode legacy SquashFS layouts with bounded native readers. Separate traversal, format parsing, staged output, and artifact reporting so filesystem support can grow without expanding CLI orchestration.

Keep fallback available for incomplete recovery, reject filename collisions, and distinguish recovered rootfs trees from intermediate evidence. Add regression coverage and document limits; local validation recovered 11 firmware inputs and two ZIP entry paths with matching file contents and symlink targets.
Keep the native recovery changes alongside upstream retry handling and installation fixes. Resolve the changelog conflict by retaining both sets of additions and fixes.

Validation: 1,996 workspace tests passed with one ignored fixture; formatting, clippy, kernel-profile, runtime-data, and 16 installer checks passed. Rebuilt firmware extraction recovered 11 images and two ZIP inputs; 21,351 file contents and symlink targets matched reference extraction.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant