Benchmark: ForgeZero beats Ninja by 6.59x on 1000 ASM modules

[alexvoste]

Performance Benchmark: ForgeZero vs Ninja on 1000 ASM modules

This benchmark evaluates the scheduling and orchestration efficiency of ForgeZero compared to the industry-standard Ninja build system. The test repository consists of 1000 independent assembly modules, compiled and linked in a parallel environment utilizing 8 threads (-j 8).

The results demonstrate a substantial performance advantage for ForgeZero on clean builds, specifically under resource-constrained hardware conditions.

Hyperfine Benchmark Results

Benchmark 1: ../fzt -dir . -out fz_out -j 8
  Time (mean ± σ):      308.7 ms ±  15.4 ms    [User: 246.5 ms, System: 301.0 ms]
  Range (min … max):    279.0 ms … 331.3 ms    10 runs

Benchmark 2: ninja
  Time (mean ± σ):     2.035 s ±  0.129 s    [User: 8.759 s, System: 4.150 s]
  Range (min … max):   1.946 s …  2.366 s    10 runs

Summary
  ../fzt -dir . -out fz_out -j 8 
    6.59 ± 0.53 times faster than ninja

Analysis of Architectural Differences

• System Overhead: Ninja accumulated 4.150 seconds of System time, indicating significant operating system overhead during process creation (fork/exec) for 1000 individual compiler invocations. ForgeZero completed the same task with only 301.0 ms of System time due to its lightweight runtime orchestration.
• CPU Utilization: Under thermal and power limits of a mobile processor (Core i5-10210U), Ninja's process management led to CPU throttling, inflating User time to 8.759 seconds. ForgeZero maintained optimal resource allocation, completing the execution in 308.7 ms.

System Configuration

OS: Arch Linux x86_64
Host: Latitude 5410
Kernel: 7.0.9-zen2-1-zen
Shell: zsh 5.9
Terminal: alacritty
CPU: Intel i5-10210U (8) @ 1.700GHz
GPU: Intel CometLake-U GT2 [UHD Graphics]