diff --git a/CHANGELOG.md b/CHANGELOG.md index c0734af..4457155 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,16 @@ ## [Unreleased] +### Added + +- An Arch Linux and Omarchy installation guide with tested prerequisites, optional tools, and extraction verification steps. + +### Fixed + +- Healthy bootloader tmux sessions are reused on tmux versions that sanitize control characters in display formats. +- Binwalk runs that exit without scanning any files are retried up to twice within the original timeout, preserving failed attempts and reporting a failure if scanning never starts. +- Automatic native dependency installation recognizes Omarchy and other distributions declaring `ID_LIKE=arch`, while preserving explicitly supported distribution recipes. + ## [2.0.0] First regular release of the Rust-based FAT CLI. diff --git a/INSTALL.md b/INSTALL.md index 0435281..fdc73c5 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -45,6 +45,109 @@ Install it in a separate directory, run `fat doctor`, and create a project with Use the `fat` commands in the [quick start](README.md#quick-start); the FAT 1.x `setup.sh`, `fat.py`, and `fat.config` instructions apply to the older release. Keep existing FAT 1.x workspaces until you have recreated the projects you need in FAT 2. +## Arch Linux and Omarchy + +Use a regular account with `sudo` access. This sequence was validated on Omarchy 4.0.4 (x86-64), using an 8 GB VM and two build jobs. A clean Arch installation was not separately tested. + +### Install FAT and the default extraction engine + +Install the prerequisites with a full system upgrade, then initialize Rust. If you already installed Rust through rustup, omit `rustup` from the package list below: + +```bash +sudo pacman -Syu --needed git rustup python +rustup default stable +rustc --version # must be 1.90 or newer +``` + +The installer requires Rust and Python to be present; `--install-system-deps` does not bootstrap them. + +Clone FAT, or use your existing checkout, and run the installer as your regular user: + +```bash +git clone https://github.com/attify/firmware-analysis-toolkit.git +cd firmware-analysis-toolkit +./scripts/install.sh --install-system-deps --jobs 2 +export PATH="$HOME/.local/bin:$HOME/.cargo/bin:$PATH" +``` + +This installs FAT, its runtime data, and pinned Binwalk 3.1.0 when no usable Binwalk is already available. Use `--jobs 1` if the build runs short of memory. Add the `PATH` export to your shell's startup file to keep it in future sessions. + +The installer uses Arch packages when `/etc/os-release` identifies Arch/Manjaro directly or an otherwise unrecognized distribution declares `arch` in `ID_LIKE`, as Omarchy does. Explicitly supported distribution IDs take precedence. + +For an older FAT checkout that rejects Omarchy, install the native packages manually and rerun without `--install-system-deps`: + +```bash +sudo pacman -S --needed base-devel pkgconf fontconfig freetype2 e2fsprogs 7zip squashfs-tools +./scripts/install.sh --jobs 2 +``` + +`7zip` is the current Arch package name; the installer's `p7zip` request resolves through its compatibility alias on the tested host. + +Verify the installed CLI and data: + +```bash +fat --version +fat data verify +fat doctor --strict +``` + +Binwalk 3.1.0 can occasionally exit before scanning its input. FAT retries that specific zero-file result up to twice, keeps incomplete output and logs beside `work/binwalk.log`, and reports failure if all attempts scan zero files. + +The installer also extracts a synthetic ZIP and verifies the recovered contents. To repeat that check from the checkout: + +```bash +python3 scripts/smoke-test-extraction.py --fat "$HOME/.local/bin/fat" +``` + +### Add the tools your workflow needs + +Each group is optional; the default extraction setup above does not need all of them. + +| Workflow | Arch packages or setup | +| --- | --- | +| Binary triage and `fat decompile` | `sudo pacman -S --needed radare2 r2ghidra` | +| Ghidra headless analysis and taint normalization | `sudo pacman -S --needed ghidra jdk21-openjdk`, then `export GHIDRA_HOME=/opt/ghidra` | +| ARM and MIPS system emulation | `sudo pacman -S --needed qemu-system-arm qemu-system-mips` | +| User-mode emulation and debugging | `sudo pacman -S --needed qemu-user gdb` | +| Bootloader terminal sessions | `sudo pacman -S --needed tmux` | +| Python tool environments | `sudo pacman -S --needed python-pip python-pipx` | +| Container backends | See the Docker steps below | +| Binary taint analysis | Follow [Optional angr setup](#optional-angr-setup) | +| Joern CPG analysis | Follow the [upstream installation guide](https://docs.joern.io/installation/); put its `joern` and C/C++ frontend commands on `PATH` | + +`r2ghidra` supplies the radare2 decompiler plugin used by `fat decompile`. Installing standalone Ghidra does not install that plugin. The Ghidra package above was tested with JDK 21; follow the package's Java requirement if a later release changes it. + +For Docker, install and start the service: + +```bash +sudo pacman -S --needed docker +sudo systemctl start docker +sudo docker run --rm hello-world +``` + +FAT must also be able to contact Docker as the user running FAT. Configure rootless Docker or your intended Docker access method using the [upstream post-installation guide](https://docs.docker.com/engine/install/linux-postinstall/), then verify `docker info` without `sudo`. Membership in the `docker` group grants root-level privileges. Installing Docker or QEMU alone does not configure a FirmAE, Firmadyne, or EMUX backend; `fat doctor` reports missing bundles and recipes separately. + +For native unblob, install the Python application and common extractors: + +```bash +sudo pacman -S --needed python-pipx erofs-utils lz4 lzop zstd partclone unarchiver android-tools +pipx install unblob +pipx install jefferson +pipx install ubi-reader +unblob --show-external-dependencies +``` + +The base packages and these additions still do not supply every unblob extractor. In particular, its SquashFS handler needs `sasquatch` even when `unsquashfs` is installed. Treat missing entries in the dependency report as unavailable format support. Follow [unblob's extractor installation instructions](https://unblob.org/installation/#install-extractors) for the remaining formats, or use its upstream Docker distribution: + +```bash +docker pull ghcr.io/onekey-sec/unblob:latest +docker run --rm --network none ghcr.io/onekey-sec/unblob:latest --show-external-dependencies +``` + +The container supplied all reported extractors and recovered a test SquashFS file on the validated host. Follow the [upstream container usage instructions](https://unblob.org/installation/#docker-image) to mount your input and output directories. This is a separate unblob invocation; it does not add those extractors to the host's native unblob installation. + +`fat doctor --strict` requires a usable extraction engine. It does not verify every unblob extractor, the angr Python environment, or a successful firmware boot. The synthetic extraction check above tests a real extraction operation. + ## Installer options The source installer requires Rust 1.90+, Python 3, and a C compiler. The default extraction profile also needs Binwalk's native build dependencies; use `--install-system-deps` to install the supported host packages. @@ -75,13 +178,16 @@ FAT orchestrates external tools. Install only those needed for your workflow; `f | Tool | Needed for | Typical installation | | --- | --- | --- | | binwalk | Firmware extraction | `./scripts/install.sh --profile extraction` installs pinned Rust Binwalk 3.1.0 | -| unblob | Alternative extraction | `pipx install unblob` | +| unblob | Alternative extraction | `pipx install unblob`, plus its [external extractors](https://unblob.org/installation/#install-extractors) | | debugfs | ext filesystem extraction and ext2 repair | `brew install e2fsprogs` or `apt install e2fsprogs` | | radare2 | Binary triage and symbol resolution | `brew install radare2` or `apt install radare2` | -| Ghidra | Decompilation and taint normalization | Install from [ghidra-sre.org](https://ghidra-sre.org/) and set `GHIDRA_HOME` | +| r2ghidra | `fat decompile` | Install the [radare2 plugin](https://github.com/radareorg/r2ghidra); Arch: `pacman -S radare2 r2ghidra` | +| Ghidra | Headless analysis and taint normalization | Install from [ghidra-sre.org](https://ghidra-sre.org/) and set `GHIDRA_HOME` | | Joern | CPG proof stage | Install from [joern.io](https://joern.io/) | | QEMU | System-mode emulation | `brew install qemu` or `apt install qemu-system` | | Docker | Containerized emulation backends | Follow the [Docker installation guide](https://docs.docker.com/get-docker/) | +| GDB | Debugger integration | `gdb` or `gdb-multiarch`, with support for the target architecture | +| tmux | Bootloader emulation sessions | `pacman -S tmux`, `apt install tmux`, or `brew install tmux` | | Python 3 | Discovery runners | `apt install python3` | | Python 3.10+ and angr | Optional binary taint engine | Isolated environment in [Optional angr setup](#optional-angr-setup) | @@ -98,6 +204,14 @@ export FAT_PYTHON="$HOME/.local/share/fat-angr-venv/bin/python" Keep `FAT_PYTHON` set in shells running FAT. On Debian/Ubuntu, install `python3-venv` if environment creation is unavailable. `fat doctor --strict` checks extraction-engine readiness; it does not validate the optional angr environment. The import command above checks that environment without running firmware analysis. +For angr releases that provide the optional Unicorn extra, install it in the same environment if you need native execution acceleration: + +```bash +"$FAT_PYTHON" -m pip install 'angr[unicorn]' +``` + +This resolved the optional Unicorn loading warning with angr 10.0.0 on the tested Omarchy host. Normal SSH login shells and non-interactive SSH commands can load different startup files; export `FAT_PYTHON` explicitly in scripts or one-shot SSH commands instead of assuming an interactive shell configuration was loaded. + ## macOS ext2 builder image `FAT_EXT2_BUILDER_IMAGE` may point to an operator-provided image pinned by digest. FAT does not pull that image or install packages at runtime. The builder disables networking, mounts firmware source read-only, enables `no-new-privileges`, and uses five narrow capabilities. diff --git a/README.md b/README.md index f37bb53..0351206 100644 --- a/README.md +++ b/README.md @@ -36,13 +36,15 @@ fat data install --archive ./dist/fat-data.zip fat doctor ``` -One-Shot Installer: +On Arch Linux or Omarchy, start with the [Arch installation steps](INSTALL.md#arch-linux-and-omarchy) for prerequisite packages and optional tools. + +One-Shot Installer for supported hosts: ```bash ./scripts/install.sh --install-system-deps ``` -Refer [INSTALL.md](INSTALL.md) for more customization in the installation process. +See [INSTALL.md](INSTALL.md) for installer options and optional tools. ## Quick Command Ref. diff --git a/crates/fat_analyze/src/image_measurements.rs b/crates/fat_analyze/src/image_measurements.rs index ac3007c..3116039 100644 --- a/crates/fat_analyze/src/image_measurements.rs +++ b/crates/fat_analyze/src/image_measurements.rs @@ -47,7 +47,7 @@ pub fn measure_repetition(bytes: &[u8]) -> RepetitionMeasurements { }; result.total_block_count = bytes.len() / BLOCK; let mut seen = HashSet::new(); - for block in bytes.chunks_exact(BLOCK) { + for block in bytes.as_chunks::().0 { if !seen.insert(block) { result.duplicate_block_count += 1; } @@ -85,7 +85,7 @@ pub fn measure_repetition(bytes: &[u8]) -> RepetitionMeasurements { .map(|n| n as usize) .unwrap_or(bytes.len()); let mut uniform_seen = HashSet::new(); - for block in bytes[..unit].chunks_exact(BLOCK) { + for block in bytes[..unit].as_chunks::().0 { if block.iter().all(|b| *b == block[0]) && !uniform_seen.insert(block[0]) { result.duplicate_uniform_blocks += 1; } diff --git a/crates/fat_analyze/src/mcu_code.rs b/crates/fat_analyze/src/mcu_code.rs index dcd8948..6df46fa 100644 --- a/crates/fat_analyze/src/mcu_code.rs +++ b/crates/fat_analyze/src/mcu_code.rs @@ -548,9 +548,11 @@ fn string_reference( Some(("ascii", String::from_utf8(tail[..end].to_vec()).ok()?)) } else { let units: Vec = tail - .chunks_exact(2) + .as_chunks::<2>() + .0 + .iter() .take(MAX_STRING_UNITS + 1) - .map(|pair| u16::from_le_bytes([pair[0], pair[1]])) + .map(|pair| u16::from_le_bytes(*pair)) .collect(); units .iter() diff --git a/crates/fat_analyze/tests/mcu_detect.rs b/crates/fat_analyze/tests/mcu_detect.rs index 21daa82..73d5ae1 100644 --- a/crates/fat_analyze/tests/mcu_detect.rs +++ b/crates/fat_analyze/tests/mcu_detect.rs @@ -543,7 +543,7 @@ fn regression_fast_vector_count_stops_before_executable_bytes() { } // Plausible-looking pointer data after the first mapped handler is code/data, // not another hundred IRQ vectors. - for word in bytes[0x104..].chunks_exact_mut(4) { + for word in bytes[0x104..].as_chunks_mut::<4>().0 { word.copy_from_slice(&(base + 0x901).to_le_bytes()); } let profile = detect_cortex_m_ivt(&bytes).expect("boot ROM vectors"); diff --git a/crates/fat_analyze/tests/mcu_vectors.rs b/crates/fat_analyze/tests/mcu_vectors.rs index c76a4cf..78894ef 100644 --- a/crates/fat_analyze/tests/mcu_vectors.rs +++ b/crates/fat_analyze/tests/mcu_vectors.rs @@ -17,7 +17,7 @@ fn table(base: u32) -> Vec { // Nonzero reserved fields must not terminate the table or become handlers. put(&mut bytes, 7, 0xdead_beef); put(&mut bytes, 8, base + 0x81); - for word in bytes[0x80..].chunks_exact_mut(4) { + for word in bytes[0x80..].as_chunks_mut::<4>().0 { word.copy_from_slice(&(base + 0x101).to_le_bytes()); } bytes diff --git a/crates/fat_bootloader/src/qemu.rs b/crates/fat_bootloader/src/qemu.rs index 6098eb0..67eeb5e 100644 --- a/crates/fat_bootloader/src/qemu.rs +++ b/crates/fat_bootloader/src/qemu.rs @@ -942,7 +942,7 @@ fn session_needs_restart(launch: &BootloaderQemuLaunch) -> Result Result>() + .join("; "); + let failure_note = if failure_details.is_empty() { + String::new() + } else { + format!("; {failure_details}") + }; + let install_note = if attempted_tools + .iter() + .any(|outcome| outcome.detail.as_deref() == Some("not installed")) + { + ", install the missing tools" + } else { + "" + }; return Err(format!( - "firmware extraction failed: neither binwalk nor unblob completed successfully{timeout_note}; inspect {} and {}, install the missing tools, run `fat doctor`, then retry `fat extract --force {}`", + "firmware extraction failed: neither binwalk nor unblob completed successfully{timeout_note}{failure_note}; inspect {} and {}{install_note}, run `fat doctor`, then retry `fat extract --force {}`", project_dir.join("work/binwalk.log").display(), project_dir.join("work/unblob.log").display(), project_dir.display(), diff --git a/crates/fat_extract/src/command.rs b/crates/fat_extract/src/command.rs index e3a2168..bb14ea4 100644 --- a/crates/fat_extract/src/command.rs +++ b/crates/fat_extract/src/command.rs @@ -93,7 +93,16 @@ where }; if timed_out { - let seconds = timeout.map(|timeout| timeout.as_secs()).unwrap_or_default(); + // A remaining retry deadline can be fractional (for example 7.999s + // of an 8s budget). Round up rather than claiming it expired a second + // early or reporting a positive subsecond timeout as zero seconds. + let seconds = timeout + .map(|timeout| { + timeout + .as_secs() + .saturating_add(u64::from(timeout.subsec_nanos() != 0)) + }) + .unwrap_or_default(); append_line( log_path, &format!("timeout after {seconds}s; process group {process_group_id} killed"), diff --git a/crates/fat_extract/src/extractors.rs b/crates/fat_extract/src/extractors.rs index 4241ed5..5155ddb 100644 --- a/crates/fat_extract/src/extractors.rs +++ b/crates/fat_extract/src/extractors.rs @@ -1,5 +1,6 @@ use std::ffi::OsString; use std::fs; +use std::io::{self, Read, Seek, SeekFrom}; use std::path::{Path, PathBuf}; use std::time::{Duration, Instant}; @@ -396,21 +397,55 @@ impl Extractor for ExternalExtractor { .iter() .map(|arg| arg.to_string_lossy().into_owned()) .collect(); - let outcome = run_logged_command( - self.id, - invocation, - &request.work_dir, - &request.log_path, - request.timeout, - on_tick, - ); + let mut attempts = 0; + let outcome = loop { + let elapsed = started.elapsed(); + let remaining = request.timeout.map(|limit| limit.saturating_sub(elapsed)); + if remaining == Some(Duration::ZERO) { + break Ok(CommandOutcome::TimedOut); + } + attempts += 1; + let outcome = run_logged_command( + self.id, + invocation.clone(), + &request.work_dir, + &request.log_path, + remaining, + &mut |tick| on_tick(elapsed + tick), + ); + // Binwalk 3 can exit successfully before its queued worker starts. + // Its explicit zero-file summary means the input was never scanned; + // a completed scan with no signatures is a different, valid result. + if self.id != "binwalk" + || !matches!(outcome, Ok(CommandOutcome::Succeeded)) + || !binwalk_analyzed_zero_files(&request.log_path) + { + break outcome; + } + // A fresh cwd avoids the input-symlink collision in Binwalk 3.1.0. + // Keep incomplete output outside the extraction tree so it cannot + // be mistaken for evidence from the completed scan. + if let Err(err) = preserve_binwalk_attempt(request, attempts) { + break Err(err); + } + if attempts == 3 { + return ExtractionOutcome::new(self.id, ExtractStatus::Failed) + .with_detail("Binwalk analyzed zero files in all 3 attempts") + .with_duration(started.elapsed()) + .with_args(rendered_args); + } + }; let duration = started.elapsed(); match outcome { Ok(CommandOutcome::Succeeded) => { let evidence = CarvedEvidence::survey(&request.work_dir); ExtractionOutcome::new(self.id, ExtractStatus::Succeeded) - .with_detail(evidence.summary()) + .with_detail(if attempts > 1 { + format!("{} after retry ({} attempts)", evidence.summary(), attempts) + } else { + evidence.summary().to_owned() + }) .with_evidence(evidence) .with_duration(duration) .with_args(rendered_args) @@ -435,3 +470,53 @@ impl Extractor for ExternalExtractor { } } } + +/// Read only the log tail: extractor output can be much larger than the input. +fn binwalk_analyzed_zero_files(log_path: &Path) -> bool { + let read_tail = || -> io::Result> { + let mut log = fs::File::open(log_path)?; + let start = log.metadata()?.len().saturating_sub(4096); + log.seek(SeekFrom::Start(start))?; + let mut tail = Vec::new(); + log.take(4096).read_to_end(&mut tail)?; + Ok(tail) + }; + read_tail().is_ok_and(|tail| { + String::from_utf8_lossy(&tail) + .lines() + .any(|line| line.trim().starts_with("Analyzed 0 files for ")) + }) +} + +fn preserve_binwalk_attempt(request: &ExtractionRequest, attempt: usize) -> io::Result<()> { + let nested_log = request.log_path.strip_prefix(&request.work_dir).ok(); + let archive_base = if nested_log.is_some() { + &request.work_dir + } else { + &request.log_path + }; + let mut sequence = attempt; + let (output, log) = loop { + let output = archive_base.with_extension(format!("attempt-{sequence}")); + let log = archive_base.with_extension(format!("attempt-{sequence}.log")); + if !output.try_exists()? && !log.try_exists()? { + break (output, log); + } + sequence += 1; + }; + fs::rename(&request.work_dir, &output)?; + let moved_log = nested_log.map(|relative| output.join(relative)); + fs::rename(moved_log.as_deref().unwrap_or(&request.log_path), &log)?; + fs::create_dir_all(&request.work_dir)?; + if let Some(parent) = request.log_path.parent() { + fs::create_dir_all(parent)?; + } + // Keep the primary log useful even when the final attempt was incomplete. + fs::write( + &request.log_path, + format!( + "Binwalk analyzed zero files; incomplete output and log preserved at {}\n", + log.display() + ), + ) +} diff --git a/scripts/install.sh b/scripts/install.sh index fad0f14..9eebb10 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -151,6 +151,28 @@ if [[ -z "$jobs" ]]; then fi [[ "$jobs" =~ ^[1-9][0-9]*$ ]] || die "--jobs must be a positive integer" +linux_dependency_id() { + # Keep absent release fields independent of the caller's environment. The + # override lets installer tests use a fixture without changing the host. + local ID="" ID_LIKE="" + local os_release="${FAT_INSTALL_OS_RELEASE:-/etc/os-release}" + if [[ -r "$os_release" ]]; then + # shellcheck disable=SC1090 + source "$os_release" + fi + case "${ID:-}" in + fedora|rhel|centos|ubuntu|debian|arch|manjaro) ;; + *) + # Derivatives such as Omarchy declare compatibility through ID_LIKE. + # Preserve explicit recipes and match a whole whitespace-separated token. + if [[ "${ID_LIKE:-}" =~ (^|[[:space:]])arch($|[[:space:]]) ]]; then + ID=arch + fi + ;; + esac + printf '%s\n' "${ID:-}" +} + install_build_dependencies() { if [[ "$(uname -s)" == "Darwin" ]]; then if ! command_available cc; then @@ -159,11 +181,7 @@ install_build_dependencies() { return fi - if [[ -r /etc/os-release ]]; then - # shellcheck disable=SC1091 - source /etc/os-release - fi - case "${ID:-}" in + case "$(linux_dependency_id)" in fedora|rhel|centos) run sudo dnf install -y gcc make ;; @@ -187,11 +205,7 @@ install_extraction_dependencies() { return fi - if [[ -r /etc/os-release ]]; then - # shellcheck disable=SC1091 - source /etc/os-release - fi - case "${ID:-}" in + case "$(linux_dependency_id)" in fedora) run sudo dnf install -y \ gcc-c++ pkgconf-pkg-config fontconfig-devel freetype-devel \ diff --git a/tests/integration/test_bootloader_emulate.rs b/tests/integration/test_bootloader_emulate.rs index 6ab1424..f81554a 100644 --- a/tests/integration/test_bootloader_emulate.rs +++ b/tests/integration/test_bootloader_emulate.rs @@ -257,6 +257,44 @@ fn bootloader_emulate_restarts_stale_tmux_sessions() { #[test] fn bootloader_emulate_reuses_a_healthy_tmux_session() { + assert_healthy_tmux_session_is_reused(); +} + +#[test] +fn bootloader_emulate_reuses_healthy_sessions_when_tmux_sanitizes_control_characters() { + if let Some(mut command) = test_subprocess::isolated_test( + "bootloader_emulate_reuses_healthy_sessions_when_tmux_sanitizes_control_characters", + ) { + let path = std::env::var_os("PATH").expect("PATH"); + let tmux = std::env::split_paths(&path) + .map(|directory| directory.join("tmux")) + .find(|binary| binary.is_file()) + .expect("tmux on PATH"); + let shim_dir = tempdir().expect("tmux shim dir"); + let shim = shim_dir.path().join("tmux"); + // tmux 3.7c replaces literal control characters in display formats with + // underscores. Model that behavior while retaining a real tmux session + // so this regression also runs against older installed tmux versions. + write_file( + &shim, + b"#!/bin/bash\nargs=()\nfor argument in \"$@\"; do\n args+=(\"${argument//$'\\t'/_}\")\ndone\nexec \"$FAT_TEST_REAL_TMUX\" \"${args[@]}\"\n", + ); + std::fs::set_permissions(&shim, std::fs::Permissions::from_mode(0o755)) + .expect("chmod tmux shim"); + command.env("FAT_TEST_REAL_TMUX", tmux).env( + "PATH", + std::env::join_paths( + std::iter::once(shim_dir.path().to_path_buf()).chain(std::env::split_paths(&path)), + ) + .expect("shim PATH"), + ); + test_subprocess::assert_success(&mut command); + return; + } + assert_healthy_tmux_session_is_reused(); +} + +fn assert_healthy_tmux_session_is_reused() { let project = prepared_bootloader_project(); let fake_dir = tempdir().expect("fake runtime dir"); let quoted_dir = fake_dir.path().join("quoted\"runtime"); diff --git a/tests/integration/test_data_cli.rs b/tests/integration/test_data_cli.rs index c9618f5..0e91f96 100644 --- a/tests/integration/test_data_cli.rs +++ b/tests/integration/test_data_cli.rs @@ -1,3 +1,6 @@ +#[path = "../support/subprocess.rs"] +mod test_subprocess; + use std::fs; use std::io::Write; use std::path::Path; @@ -124,6 +127,14 @@ fn data_install_status_verify_and_list_manage_a_versioned_tree() { #[test] fn data_status_resolves_and_verifies_executable_relative_share_data() { + // Keep writable executable-copy handles out of concurrently spawned test + // children, which can otherwise make exec fail with ETXTBSY on Linux. + if let Some(mut command) = test_subprocess::isolated_test( + "data_status_resolves_and_verifies_executable_relative_share_data", + ) { + test_subprocess::assert_success(&mut command); + return; + } let prefix = tempfile::tempdir().unwrap(); let isolated_home = tempfile::tempdir().unwrap(); let bin_dir = prefix.path().join("bin"); @@ -181,6 +192,12 @@ fn data_status_resolves_and_verifies_executable_relative_share_data() { #[test] fn data_verify_does_not_fall_back_past_corrupt_executable_relative_data() { + if let Some(mut command) = test_subprocess::isolated_test( + "data_verify_does_not_fall_back_past_corrupt_executable_relative_data", + ) { + test_subprocess::assert_success(&mut command); + return; + } let prefix = tempfile::tempdir().unwrap(); let isolated_home = tempfile::tempdir().unwrap(); let bin_dir = prefix.path().join("bin"); @@ -217,6 +234,12 @@ fn data_verify_does_not_fall_back_past_corrupt_executable_relative_data() { #[test] fn stale_executable_relative_active_record_is_broken_without_managed_fallback() { + if let Some(mut command) = test_subprocess::isolated_test( + "stale_executable_relative_active_record_is_broken_without_managed_fallback", + ) { + test_subprocess::assert_success(&mut command); + return; + } let prefix = tempfile::tempdir().unwrap(); let isolated_home = tempfile::tempdir().unwrap(); let bin_dir = prefix.path().join("bin"); @@ -318,6 +341,12 @@ fn data_verify_uses_the_authoritative_environment_tree() { #[test] fn data_status_does_not_fall_back_past_a_configured_broken_environment_root() { + if let Some(mut command) = test_subprocess::isolated_test( + "data_status_does_not_fall_back_past_a_configured_broken_environment_root", + ) { + test_subprocess::assert_success(&mut command); + return; + } let prefix = tempfile::tempdir().unwrap(); let configured = tempfile::tempdir().unwrap(); let bin_dir = prefix.path().join("bin"); diff --git a/tests/integration/test_extract_cli.rs b/tests/integration/test_extract_cli.rs index 0e38bfb..8a7213c 100644 --- a/tests/integration/test_extract_cli.rs +++ b/tests/integration/test_extract_cli.rs @@ -1677,6 +1677,10 @@ fn fat_extract_fails_without_any_available_extraction_engine() { String::from_utf8_lossy(&output.stdout), String::from_utf8_lossy(&output.stderr) ); + assert!( + String::from_utf8_lossy(&output.stderr).contains("install the missing tools"), + "missing engines should retain installation advice: {output:?}" + ); let project_dir = only_project_dir(workspace.path()); assert!( !project_dir.join("work/extraction-manifest.json").exists(), @@ -1712,6 +1716,10 @@ fn fat_extract_fails_when_every_extraction_engine_exits_nonzero() { String::from_utf8_lossy(&output.stdout), String::from_utf8_lossy(&output.stderr) ); + assert!( + !String::from_utf8_lossy(&output.stderr).contains("install the missing tools"), + "installed engines that fail should direct users to their logs: {output:?}" + ); let project_dir = only_project_dir(workspace.path()); assert!( !project_dir.join("work/extraction-manifest.json").exists(), @@ -1719,6 +1727,56 @@ fn fat_extract_fails_when_every_extraction_engine_exits_nonzero() { ); } +#[test] +fn fat_extract_reports_binwalk_retry_exhaustion_without_missing_tool_advice() { + let workspace = tempdir().expect("workspace"); + let fake_bin_dir = tempdir().expect("fake bin dir"); + let firmware_path = workspace.path().join("zero-scan.bin"); + fs::write(&firmware_path, b"firmware-bytes").expect("firmware file"); + write_script( + &fake_bin_dir.path().join("binwalk"), + "#!/bin/sh\nprintf 'incomplete\\n' > partial.txt\nprintf 'Analyzed 0 files for 85 file signatures (187 magic patterns) in 5.0 milliseconds\\n'\n", + ); + + let output = Command::new(env!("CARGO_BIN_EXE_fat")) + .args([ + "extract", + firmware_path.to_str().expect("firmware path"), + "--extractor", + "binwalk", + ]) + .current_dir(workspace.path()) + // unblob is absent but was not selected, so it must not trigger advice. + .env("PATH", fake_bin_dir.path()) + .output() + .expect("fat extract runs"); + + assert!(!output.status.success(), "{output:?}"); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!(stderr.contains("firmware extraction failed:"), "{stderr}"); + assert!( + stderr.contains("binwalk: Binwalk analyzed zero files in all 3 attempts"), + "the terminal error must report why the installed engine failed: {stderr}" + ); + assert!(stderr.contains("work/binwalk.log"), "{stderr}"); + assert!(stderr.contains("work/unblob.log"), "{stderr}"); + assert!(!stderr.contains("install the missing tools"), "{stderr}"); + let project_dir = only_project_dir(workspace.path()); + assert_eq!(project_status(&project_dir), ProjectStatus::Error); + assert!(!project_dir.join("work/extraction-manifest.json").exists()); + assert!(!project_dir.join("work/extractions").exists()); + for attempt in 1..=3 { + let archived = project_dir.join(format!("work/binwalk.attempt-{attempt}")); + assert_eq!( + fs::read_to_string(archived.join("partial.txt")).expect("preserved output"), + "incomplete\n" + ); + assert!(project_dir + .join(format!("work/binwalk.attempt-{attempt}.log")) + .is_file()); + } +} + #[test] fn fat_extract_rejects_zero_exit_engines_that_recover_no_evidence() { let workspace = tempdir().expect("workspace"); diff --git a/tests/integration/test_extraction_strategy.rs b/tests/integration/test_extraction_strategy.rs index 8402c33..b8cdb20 100644 --- a/tests/integration/test_extraction_strategy.rs +++ b/tests/integration/test_extraction_strategy.rs @@ -263,3 +263,188 @@ fn selection_parsing_accepts_known_engines_and_rejects_the_rest() { ); assert!(ExtractorSelection::parse("unblob", &known).is_err()); } + +#[cfg(unix)] +#[path = "../support/subprocess.rs"] +mod subprocess; + +#[cfg(unix)] +mod binwalk_recovery { + use super::*; + use fat_extract::extractors::ExternalExtractor; + use std::fs; + use std::os::unix::fs::PermissionsExt; + + fn check(test: &str, mode: &str, timeout: Option) { + if let Some(mut child) = subprocess::isolated_test(test) { + let temp = tempdir().unwrap(); + let bin = temp.path().join("bin"); + fs::create_dir(&bin).unwrap(); + let script = bin.join("binwalk"); + fs::write( + &script, + r#"#!/bin/sh +count_file="$FAT_BINWALK_TEST_ROOT/count" +n=0 +if [ -f "$count_file" ]; then read -r n < "$count_file"; fi +n=$((n + 1)) +printf '%s\n' "$n" > "$count_file" +case "$FAT_BINWALK_TEST_MODE" in + failed) exit 2 ;; + empty) printf 'Analyzed 1 file for 85 file signatures\n'; exit 0 ;; + timeout) sleep 3 ;; + once|repeated|nested) + if [ "$((n % 2))" -eq 0 ]; then + test ! -e partial.txt || exit 3 + mkdir -p rootfs/etc + printf 'recovered\n' > rootfs/etc/marker + printf 'Analyzed 1 file for 85 file signatures\n' + exit 0 + fi ;; +esac +printf 'incomplete\n' > partial.txt +printf 'Analyzed 0 files for 85 file signatures (187 magic patterns) in 5.0 milliseconds\n' +"#, + ) + .unwrap(); + fs::set_permissions(&script, fs::Permissions::from_mode(0o755)).unwrap(); + let paths = std::iter::once(bin) + .chain(std::env::split_paths( + &std::env::var_os("PATH").unwrap_or_default(), + )) + .collect::>(); + child + .env("PATH", std::env::join_paths(paths).unwrap()) + .env("FAT_BINWALK_TEST_ROOT", temp.path()) + .env("FAT_BINWALK_TEST_MODE", mode); + subprocess::assert_success(&mut child); + return; + } + let root = PathBuf::from(std::env::var_os("FAT_BINWALK_TEST_ROOT").unwrap()); + let firmware = root.join("firmware.bin"); + fs::write(&firmware, b"fixture").unwrap(); + let request = ExtractionRequest { + firmware, + work_dir: root.join("extractions/binwalk"), + log_path: if mode == "nested" { + root.join("extractions/binwalk/binwalk.log") + } else { + root.join("binwalk.log") + }, + timeout, + }; + let result = ExternalExtractor::binwalk().extract(&request, &mut |_| {}); + let count = fs::read_to_string(root.join("count")).unwrap(); + match mode { + "once" | "repeated" | "nested" => { + assert_eq!(count.trim(), "2", "zero scans must be retried"); + assert_eq!(result.status, ExtractStatus::Succeeded); + assert_eq!( + fs::read_to_string(request.work_dir.join("rootfs/etc/marker")).unwrap(), + "recovered\n" + ); + assert!(!request.work_dir.join("partial.txt").exists()); + let previous = if mode == "nested" { + root.join("extractions/binwalk.attempt-1") + } else { + root.join("binwalk.attempt-1") + }; + assert_eq!( + fs::read_to_string(previous.join("partial.txt")).unwrap(), + "incomplete\n" + ); + assert!(fs::read_to_string(previous.with_extension("attempt-1.log")) + .unwrap() + .contains("Analyzed 0 files")); + assert!(result.detail.unwrap().contains("retry")); + if mode == "repeated" { + fs::remove_dir_all(&request.work_dir).unwrap(); + let next = ExternalExtractor::binwalk().extract(&request, &mut |_| {}); + assert_eq!(next.status, ExtractStatus::Succeeded); + assert!(previous.join("partial.txt").is_file()); + assert!(root.join("binwalk.attempt-2/partial.txt").is_file()); + } + } + "persistent" => { + assert_eq!(result.status, ExtractStatus::Failed); + assert_eq!(count.trim(), "3", "retry budget must be bounded"); + assert!(result.detail.unwrap().contains("analyzed zero files")); + assert_eq!(fs::read_dir(&request.work_dir).unwrap().count(), 0); + assert!(root.join("binwalk.attempt-3/partial.txt").is_file()); + } + "timeout" => { + assert_eq!(result.status, ExtractStatus::TimedOut); + assert_eq!(count.trim(), "2", "retry must share the original deadline"); + assert!(result.duration < Duration::from_secs(6)); + } + "empty" => { + assert_eq!(result.status, ExtractStatus::Succeeded); + assert_eq!(count.trim(), "1", "a scan with no signatures is valid"); + } + "failed" => { + assert_eq!(result.status, ExtractStatus::Failed); + assert_eq!(count.trim(), "1", "other errors must not be retried"); + } + _ => unreachable!(), + } + } + + #[test] + fn recovers_zero_scan_with_fresh_output_and_preserved_evidence() { + check( + "binwalk_recovery::recovers_zero_scan_with_fresh_output_and_preserved_evidence", + "once", + None, + ); + } + #[test] + fn retries_with_log_inside_output_directory() { + check( + "binwalk_recovery::retries_with_log_inside_output_directory", + "nested", + None, + ); + } + + #[test] + fn preserves_previous_runs_when_retrying_again() { + check( + "binwalk_recovery::preserves_previous_runs_when_retrying_again", + "repeated", + None, + ); + } + + #[test] + fn fails_after_three_zero_scans() { + check( + "binwalk_recovery::fails_after_three_zero_scans", + "persistent", + None, + ); + } + #[test] + fn keeps_one_deadline_across_retries() { + check( + "binwalk_recovery::keeps_one_deadline_across_retries", + "timeout", + Some(Duration::from_secs(5)), + ); + } + #[test] + fn accepts_completed_scan_without_signatures() { + check( + "binwalk_recovery::accepts_completed_scan_without_signatures", + "empty", + None, + ); + } + #[test] + fn does_not_retry_other_failures() { + check( + "binwalk_recovery::does_not_retry_other_failures", + "failed", + None, + ); + } +} diff --git a/tests/integration/test_managed_linux_vm_runtime.rs b/tests/integration/test_managed_linux_vm_runtime.rs index 0f7c83e..d403113 100644 --- a/tests/integration/test_managed_linux_vm_runtime.rs +++ b/tests/integration/test_managed_linux_vm_runtime.rs @@ -634,15 +634,22 @@ fn firmadyne_failed_stop_is_nonzero_retriable_and_then_idempotent() { } #[test] -fn fat_doctor_reports_native_arm64_firmae_recipe_checks_when_configured() { +fn fat_doctor_reports_native_firmae_recipe_checks_when_configured() { let path_dir = tempdir().expect("path dir"); let bundle_dir = tempdir().expect("bundle dir"); let upstream_dir = tempdir().expect("upstream dir"); let helper_python = path_dir.path().join("firmae-host-python"); + let native_arch = match std::env::consts::ARCH { + "aarch64" => "arm64", + "x86_64" => "amd64", + other => other, + }; make_executable( path_dir.path().join("docker"), - "#!/bin/sh\nif [ \"$1\" = \"version\" ]; then printf 'arm64\\n'; exit 0; fi\nexit 1\n", + &format!( + "#!/bin/sh\nif [ \"$1\" = \"version\" ]; then printf '{native_arch}\\n'; exit 0; fi\nexit 1\n" + ), ); make_executable(helper_python.clone(), "#!/bin/sh\nexit 0\n"); std::fs::create_dir_all(upstream_dir.path().join("database")).expect("database dir"); @@ -682,7 +689,9 @@ fn fat_doctor_reports_native_arm64_firmae_recipe_checks_when_configured() { let stdout = String::from_utf8_lossy(&output.stdout); assert!(stdout.contains("FirmAE [firmae]: available"), "{stdout}"); assert!( - stdout.contains("firmae recipe docker-server-arch: pass (arm64 (native))"), + stdout.contains(&format!( + "firmae recipe docker-server-arch: pass ({native_arch} (native))" + )), "{stdout}" ); assert!( diff --git a/tests/release/test_install_script.py b/tests/release/test_install_script.py index 7b74ee7..fbc91c7 100755 --- a/tests/release/test_install_script.py +++ b/tests/release/test_install_script.py @@ -174,6 +174,158 @@ def run_installer(self, *arguments: str) -> subprocess.CompletedProcess[str]: check=False, ) + def configure_linux_release(self, contents: str | None) -> None: + release = self.root / "fixture os-release" + if contents is not None: + release.write_text(contents, encoding="utf-8") + self.environment["FAT_INSTALL_OS_RELEASE"] = str(release) + write_executable(self.fake_bin / "uname", "#!/bin/sh\nprintf 'Linux\\n'\n") + # Never delegate to a real package manager, even if dispatch is wrong. + write_executable( + self.fake_bin / "sudo", + f""" + #!{sys.executable} + import os + from pathlib import Path + import sys + + with Path(os.environ["FAKE_INSTALL_LOG"]).open("a", encoding="utf-8") as output: + output.write("sudo " + " ".join(sys.argv[1:]) + "\\n") + """, + ) + + def test_omarchy_uses_arch_dependencies_for_both_profiles(self) -> None: + self.configure_linux_release("ID=omarchy\nID_LIKE=arch\n") + for profile in ("core", "extraction"): + with self.subTest(profile=profile): + self.log.unlink(missing_ok=True) + result = self.run_installer( + "--profile", profile, "--install-system-deps", + "--prefix", str(self.root / profile), "--jobs", "1", + ) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + packages = [ + line for line in self.log.read_text().splitlines() + if line.startswith("sudo ") + ] + expected = ["sudo pacman -S --needed base-devel"] + if profile == "extraction": + expected.append( + "sudo pacman -S --needed fontconfig freetype2 " + "e2fsprogs p7zip squashfs-tools" + ) + self.assertEqual(packages, expected) + + def test_arch_family_matches_only_complete_whitespace_separated_tokens(self) -> None: + for family, supported in ( + ("arch other", True), + ("other arch another", True), + ("other\tarch\n", True), + ("archlinux", False), + ("notarch", False), + ("arch-other", False), + ("debian ubuntu", False), + ): + with self.subTest(family=family): + self.configure_linux_release(f'ID=derivative\nID_LIKE="{family}"\n') + result = self.run_installer("--install-system-deps", "--dry-run") + if supported: + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn("sudo pacman -S --needed base-devel", result.stdout) + self.assertIn("e2fsprogs p7zip squashfs-tools", result.stdout) + else: + self.assertNotEqual(result.returncode, 0) + self.assertIn("unsupported OS", result.stderr) + self.assertFalse(self.log.exists(), "dry-run must not execute commands") + + def test_supported_ids_take_precedence_over_arch_family(self) -> None: + recipes = { + "fedora": [ + "sudo dnf install -y gcc make", + "sudo dnf install -y gcc-c++ pkgconf-pkg-config fontconfig-devel " + "freetype-devel e2fsprogs 7zip lz4 zstd squashfs-tools util-linux-script", + ], + "rhel": [ + "sudo dnf install -y gcc make", + "sudo dnf install -y gcc-c++ pkgconf-pkg-config fontconfig-devel " + "freetype-devel e2fsprogs p7zip p7zip-plugins squashfs-tools util-linux-script", + ], + "ubuntu": [ + "sudo apt-get update", + "sudo apt-get install -y build-essential", + "sudo apt-get update", + "sudo apt-get install -y g++ pkg-config libfontconfig1-dev " + "libfreetype6-dev e2fsprogs p7zip-full squashfs-tools", + ], + "arch": [ + "sudo pacman -S --needed base-devel", + "sudo pacman -S --needed fontconfig freetype2 e2fsprogs p7zip squashfs-tools", + ], + } + recipes["centos"] = recipes["rhel"] + recipes["debian"] = recipes["ubuntu"] + recipes["manjaro"] = recipes["arch"] + for distro, expected in recipes.items(): + with self.subTest(distro=distro): + self.configure_linux_release(f"ID={distro}\nID_LIKE=arch\n") + self.log.unlink(missing_ok=True) + result = self.run_installer( + "--install-system-deps", "--prefix", str(self.root / distro), + "--jobs", "1", + ) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + packages = [ + line for line in self.log.read_text().splitlines() + if line.startswith("sudo ") + ] + self.assertEqual(packages, expected) + + def test_missing_release_fields_do_not_use_inherited_environment(self) -> None: + self.environment.update({"ID": "arch", "ID_LIKE": "arch"}) + for contents in ("", "ID=unknown\n", "ID_LIKE=notarch\n"): + with self.subTest(contents=contents): + self.configure_linux_release(contents) + result = self.run_installer("--install-system-deps") + self.assertNotEqual(result.returncode, 0) + self.assertIn("rerun without --install-system-deps", result.stderr) + self.assertNotIn("unbound variable", result.stderr) + self.assertFalse(self.log.exists(), "unsupported OS must fail before mutation") + + self.configure_linux_release("ID_LIKE=arch\n") + result = self.run_installer("--install-system-deps", "--dry-run") + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn("sudo pacman -S --needed base-devel", result.stdout) + + def test_missing_release_file_fails_before_installing(self) -> None: + self.environment.update({"ID": "arch", "ID_LIKE": "arch"}) + self.configure_linux_release(None) + result = self.run_installer("--install-system-deps") + self.assertNotEqual(result.returncode, 0) + self.assertIn("unsupported OS", result.stderr) + self.assertIn("rerun without --install-system-deps", result.stderr) + self.assertNotIn("unbound variable", result.stderr) + self.assertFalse(self.log.exists()) + + def test_omarchy_system_deps_dry_run_does_not_mutate(self) -> None: + self.configure_linux_release("ID=omarchy\nID_LIKE=arch\n") + self.environment["TMPDIR"] = str(self.root) + scratch = self.root / "fat-install.DRY_RUN" + scratch.mkdir() + marker = scratch / "keep.txt" + marker.write_text("existing data") + for profile in ("core", "extraction"): + with self.subTest(profile=profile): + prefix = self.root / profile + result = self.run_installer( + "--profile", profile, "--install-system-deps", + "--prefix", str(prefix), "--dry-run", + ) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn("sudo pacman -S --needed base-devel", result.stdout) + self.assertEqual(marker.read_text(), "existing data") + self.assertFalse(prefix.exists()) + self.assertFalse(self.log.exists()) + def test_extraction_profile_installs_pinned_binwalk_and_is_idempotent(self) -> None: prefix = self.root / "extraction-prefix" first = self.run_installer(