Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,16 @@

## [Unreleased]

### Added

- An Arch Linux and Omarchy installation guide with tested prerequisites, optional tools, and extraction verification steps.

### Fixed

- Healthy bootloader tmux sessions are reused on tmux versions that sanitize control characters in display formats.
- Binwalk runs that exit without scanning any files are retried up to twice within the original timeout, preserving failed attempts and reporting a failure if scanning never starts.
- Automatic native dependency installation recognizes Omarchy and other distributions declaring `ID_LIKE=arch`, while preserving explicitly supported distribution recipes.

## [2.0.0]

First regular release of the Rust-based FAT CLI.
Expand Down
118 changes: 116 additions & 2 deletions INSTALL.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,109 @@ Install it in a separate directory, run `fat doctor`, and create a project with
Use the `fat` commands in the [quick start](README.md#quick-start); the FAT 1.x `setup.sh`, `fat.py`, and `fat.config` instructions apply to the older release.
Keep existing FAT 1.x workspaces until you have recreated the projects you need in FAT 2.

## Arch Linux and Omarchy

Use a regular account with `sudo` access. This sequence was validated on Omarchy 4.0.4 (x86-64), using an 8 GB VM and two build jobs. A clean Arch installation was not separately tested.

### Install FAT and the default extraction engine

Install the prerequisites with a full system upgrade, then initialize Rust. If you already installed Rust through rustup, omit `rustup` from the package list below:

```bash
sudo pacman -Syu --needed git rustup python
rustup default stable
rustc --version # must be 1.90 or newer
```

The installer requires Rust and Python to be present; `--install-system-deps` does not bootstrap them.

Clone FAT, or use your existing checkout, and run the installer as your regular user:

```bash
git clone https://github.com/attify/firmware-analysis-toolkit.git
cd firmware-analysis-toolkit
./scripts/install.sh --install-system-deps --jobs 2
export PATH="$HOME/.local/bin:$HOME/.cargo/bin:$PATH"
```

This installs FAT, its runtime data, and pinned Binwalk 3.1.0 when no usable Binwalk is already available. Use `--jobs 1` if the build runs short of memory. Add the `PATH` export to your shell's startup file to keep it in future sessions.

The installer uses Arch packages when `/etc/os-release` identifies Arch/Manjaro directly or an otherwise unrecognized distribution declares `arch` in `ID_LIKE`, as Omarchy does. Explicitly supported distribution IDs take precedence.

For an older FAT checkout that rejects Omarchy, install the native packages manually and rerun without `--install-system-deps`:

```bash
sudo pacman -S --needed base-devel pkgconf fontconfig freetype2 e2fsprogs 7zip squashfs-tools
./scripts/install.sh --jobs 2
```

`7zip` is the current Arch package name; the installer's `p7zip` request resolves through its compatibility alias on the tested host.

Verify the installed CLI and data:

```bash
fat --version
fat data verify
fat doctor --strict
```

Binwalk 3.1.0 can occasionally exit before scanning its input. FAT retries that specific zero-file result up to twice, keeps incomplete output and logs beside `work/binwalk.log`, and reports failure if all attempts scan zero files.

The installer also extracts a synthetic ZIP and verifies the recovered contents. To repeat that check from the checkout:

```bash
python3 scripts/smoke-test-extraction.py --fat "$HOME/.local/bin/fat"
```

### Add the tools your workflow needs

Each group is optional; the default extraction setup above does not need all of them.

| Workflow | Arch packages or setup |
| --- | --- |
| Binary triage and `fat decompile` | `sudo pacman -S --needed radare2 r2ghidra` |
| Ghidra headless analysis and taint normalization | `sudo pacman -S --needed ghidra jdk21-openjdk`, then `export GHIDRA_HOME=/opt/ghidra` |
| ARM and MIPS system emulation | `sudo pacman -S --needed qemu-system-arm qemu-system-mips` |
| User-mode emulation and debugging | `sudo pacman -S --needed qemu-user gdb` |
| Bootloader terminal sessions | `sudo pacman -S --needed tmux` |
| Python tool environments | `sudo pacman -S --needed python-pip python-pipx` |
| Container backends | See the Docker steps below |
| Binary taint analysis | Follow [Optional angr setup](#optional-angr-setup) |
| Joern CPG analysis | Follow the [upstream installation guide](https://docs.joern.io/installation/); put its `joern` and C/C++ frontend commands on `PATH` |

`r2ghidra` supplies the radare2 decompiler plugin used by `fat decompile`. Installing standalone Ghidra does not install that plugin. The Ghidra package above was tested with JDK 21; follow the package's Java requirement if a later release changes it.

For Docker, install and start the service:

```bash
sudo pacman -S --needed docker
sudo systemctl start docker
sudo docker run --rm hello-world
```

FAT must also be able to contact Docker as the user running FAT. Configure rootless Docker or your intended Docker access method using the [upstream post-installation guide](https://docs.docker.com/engine/install/linux-postinstall/), then verify `docker info` without `sudo`. Membership in the `docker` group grants root-level privileges. Installing Docker or QEMU alone does not configure a FirmAE, Firmadyne, or EMUX backend; `fat doctor` reports missing bundles and recipes separately.

For native unblob, install the Python application and common extractors:

```bash
sudo pacman -S --needed python-pipx erofs-utils lz4 lzop zstd partclone unarchiver android-tools
pipx install unblob
pipx install jefferson
pipx install ubi-reader
unblob --show-external-dependencies
```

The base packages and these additions still do not supply every unblob extractor. In particular, its SquashFS handler needs `sasquatch` even when `unsquashfs` is installed. Treat missing entries in the dependency report as unavailable format support. Follow [unblob's extractor installation instructions](https://unblob.org/installation/#install-extractors) for the remaining formats, or use its upstream Docker distribution:

```bash
docker pull ghcr.io/onekey-sec/unblob:latest
docker run --rm --network none ghcr.io/onekey-sec/unblob:latest --show-external-dependencies
```

The container supplied all reported extractors and recovered a test SquashFS file on the validated host. Follow the [upstream container usage instructions](https://unblob.org/installation/#docker-image) to mount your input and output directories. This is a separate unblob invocation; it does not add those extractors to the host's native unblob installation.

`fat doctor --strict` requires a usable extraction engine. It does not verify every unblob extractor, the angr Python environment, or a successful firmware boot. The synthetic extraction check above tests a real extraction operation.

## Installer options

The source installer requires Rust 1.90+, Python 3, and a C compiler. The default extraction profile also needs Binwalk's native build dependencies; use `--install-system-deps` to install the supported host packages.
Expand Down Expand Up @@ -75,13 +178,16 @@ FAT orchestrates external tools. Install only those needed for your workflow; `f
| Tool | Needed for | Typical installation |
| --- | --- | --- |
| binwalk | Firmware extraction | `./scripts/install.sh --profile extraction` installs pinned Rust Binwalk 3.1.0 |
| unblob | Alternative extraction | `pipx install unblob` |
| unblob | Alternative extraction | `pipx install unblob`, plus its [external extractors](https://unblob.org/installation/#install-extractors) |
| debugfs | ext filesystem extraction and ext2 repair | `brew install e2fsprogs` or `apt install e2fsprogs` |
| radare2 | Binary triage and symbol resolution | `brew install radare2` or `apt install radare2` |
| Ghidra | Decompilation and taint normalization | Install from [ghidra-sre.org](https://ghidra-sre.org/) and set `GHIDRA_HOME` |
| r2ghidra | `fat decompile` | Install the [radare2 plugin](https://github.com/radareorg/r2ghidra); Arch: `pacman -S radare2 r2ghidra` |
| Ghidra | Headless analysis and taint normalization | Install from [ghidra-sre.org](https://ghidra-sre.org/) and set `GHIDRA_HOME` |
| Joern | CPG proof stage | Install from [joern.io](https://joern.io/) |
| QEMU | System-mode emulation | `brew install qemu` or `apt install qemu-system` |
| Docker | Containerized emulation backends | Follow the [Docker installation guide](https://docs.docker.com/get-docker/) |
| GDB | Debugger integration | `gdb` or `gdb-multiarch`, with support for the target architecture |
| tmux | Bootloader emulation sessions | `pacman -S tmux`, `apt install tmux`, or `brew install tmux` |
| Python 3 | Discovery runners | `apt install python3` |
| Python 3.10+ and angr | Optional binary taint engine | Isolated environment in [Optional angr setup](#optional-angr-setup) |

Expand All @@ -98,6 +204,14 @@ export FAT_PYTHON="$HOME/.local/share/fat-angr-venv/bin/python"

Keep `FAT_PYTHON` set in shells running FAT. On Debian/Ubuntu, install `python3-venv` if environment creation is unavailable. `fat doctor --strict` checks extraction-engine readiness; it does not validate the optional angr environment. The import command above checks that environment without running firmware analysis.

For angr releases that provide the optional Unicorn extra, install it in the same environment if you need native execution acceleration:

```bash
"$FAT_PYTHON" -m pip install 'angr[unicorn]'
```

This resolved the optional Unicorn loading warning with angr 10.0.0 on the tested Omarchy host. Normal SSH login shells and non-interactive SSH commands can load different startup files; export `FAT_PYTHON` explicitly in scripts or one-shot SSH commands instead of assuming an interactive shell configuration was loaded.

## macOS ext2 builder image

`FAT_EXT2_BUILDER_IMAGE` may point to an operator-provided image pinned by digest. FAT does not pull that image or install packages at runtime. The builder disables networking, mounts firmware source read-only, enables `no-new-privileges`, and uses five narrow capabilities.
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,13 +36,15 @@ fat data install --archive ./dist/fat-data.zip
fat doctor
```

One-Shot Installer:
On Arch Linux or Omarchy, start with the [Arch installation steps](INSTALL.md#arch-linux-and-omarchy) for prerequisite packages and optional tools.

One-Shot Installer for supported hosts:

```bash
./scripts/install.sh --install-system-deps
```

Refer [INSTALL.md](INSTALL.md) for more customization in the installation process.
See [INSTALL.md](INSTALL.md) for installer options and optional tools.

## Quick Command Ref.

Expand Down
4 changes: 2 additions & 2 deletions crates/fat_analyze/src/image_measurements.rs
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ pub fn measure_repetition(bytes: &[u8]) -> RepetitionMeasurements {
};
result.total_block_count = bytes.len() / BLOCK;
let mut seen = HashSet::new();
for block in bytes.chunks_exact(BLOCK) {
for block in bytes.as_chunks::<BLOCK>().0 {
if !seen.insert(block) {
result.duplicate_block_count += 1;
}
Expand Down Expand Up @@ -85,7 +85,7 @@ pub fn measure_repetition(bytes: &[u8]) -> RepetitionMeasurements {
.map(|n| n as usize)
.unwrap_or(bytes.len());
let mut uniform_seen = HashSet::new();
for block in bytes[..unit].chunks_exact(BLOCK) {
for block in bytes[..unit].as_chunks::<BLOCK>().0 {
if block.iter().all(|b| *b == block[0]) && !uniform_seen.insert(block[0]) {
result.duplicate_uniform_blocks += 1;
}
Expand Down
6 changes: 4 additions & 2 deletions crates/fat_analyze/src/mcu_code.rs
Original file line number Diff line number Diff line change
Expand Up @@ -548,9 +548,11 @@ fn string_reference(
Some(("ascii", String::from_utf8(tail[..end].to_vec()).ok()?))
} else {
let units: Vec<u16> = tail
.chunks_exact(2)
.as_chunks::<2>()
.0
.iter()
.take(MAX_STRING_UNITS + 1)
.map(|pair| u16::from_le_bytes([pair[0], pair[1]]))
.map(|pair| u16::from_le_bytes(*pair))
.collect();
units
.iter()
Expand Down
2 changes: 1 addition & 1 deletion crates/fat_analyze/tests/mcu_detect.rs
Original file line number Diff line number Diff line change
Expand Up @@ -543,7 +543,7 @@ fn regression_fast_vector_count_stops_before_executable_bytes() {
}
// Plausible-looking pointer data after the first mapped handler is code/data,
// not another hundred IRQ vectors.
for word in bytes[0x104..].chunks_exact_mut(4) {
for word in bytes[0x104..].as_chunks_mut::<4>().0 {
word.copy_from_slice(&(base + 0x901).to_le_bytes());
}
let profile = detect_cortex_m_ivt(&bytes).expect("boot ROM vectors");
Expand Down
2 changes: 1 addition & 1 deletion crates/fat_analyze/tests/mcu_vectors.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ fn table(base: u32) -> Vec<u8> {
// Nonzero reserved fields must not terminate the table or become handlers.
put(&mut bytes, 7, 0xdead_beef);
put(&mut bytes, 8, base + 0x81);
for word in bytes[0x80..].chunks_exact_mut(4) {
for word in bytes[0x80..].as_chunks_mut::<4>().0 {
word.copy_from_slice(&(base + 0x101).to_le_bytes());
}
bytes
Expand Down
6 changes: 4 additions & 2 deletions crates/fat_bootloader/src/qemu.rs
Original file line number Diff line number Diff line change
Expand Up @@ -942,15 +942,17 @@ fn session_needs_restart(launch: &BootloaderQemuLaunch) -> Result<bool, Bootload
"-p",
"-t",
&launch.tmux_session,
"#{pane_dead}\t#{@fat_launch_command_hex}",
"#{pane_dead}|#{@fat_launch_command_hex}",
])
.output()?;
if !output.status.success() {
return Ok(true);
}

let state = String::from_utf8_lossy(&output.stdout);
let (pane_dead, command_identity) = state.trim_end().split_once('\t').unwrap_or(("1", ""));
// tmux may sanitize literal control characters in display formats. A
// printable separator is unambiguous because the command identity is hex.
let (pane_dead, command_identity) = state.trim_end().split_once('|').unwrap_or(("1", ""));

Ok(pane_dead == "1" || command_identity != launch_command_identity(&launch.qemu_command))
}
Expand Down
25 changes: 24 additions & 1 deletion crates/fat_cli/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3690,8 +3690,31 @@ fn cmd_extract(
timed_out.join(" and ")
)
};
let failure_details = attempted_tools
.iter()
.filter_map(|outcome| {
outcome
.detail
.as_ref()
.map(|detail| format!("{}: {detail}", outcome.extractor))
})
.collect::<Vec<_>>()
.join("; ");
let failure_note = if failure_details.is_empty() {
String::new()
} else {
format!("; {failure_details}")
};
let install_note = if attempted_tools
.iter()
.any(|outcome| outcome.detail.as_deref() == Some("not installed"))
{
", install the missing tools"
} else {
""
};
return Err(format!(
"firmware extraction failed: neither binwalk nor unblob completed successfully{timeout_note}; inspect {} and {}, install the missing tools, run `fat doctor`, then retry `fat extract --force {}`",
"firmware extraction failed: neither binwalk nor unblob completed successfully{timeout_note}{failure_note}; inspect {} and {}{install_note}, run `fat doctor`, then retry `fat extract --force {}`",
project_dir.join("work/binwalk.log").display(),
project_dir.join("work/unblob.log").display(),
project_dir.display(),
Expand Down
11 changes: 10 additions & 1 deletion crates/fat_extract/src/command.rs
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,16 @@ where
};

if timed_out {
let seconds = timeout.map(|timeout| timeout.as_secs()).unwrap_or_default();
// A remaining retry deadline can be fractional (for example 7.999s
// of an 8s budget). Round up rather than claiming it expired a second
// early or reporting a positive subsecond timeout as zero seconds.
let seconds = timeout
.map(|timeout| {
timeout
.as_secs()
.saturating_add(u64::from(timeout.subsec_nanos() != 0))
})
.unwrap_or_default();
append_line(
log_path,
&format!("timeout after {seconds}s; process group {process_group_id} killed"),
Expand Down
Loading
Loading