CRUX competitor: BitNet (bitnet.cpp) — build ternary 1.58-bit inference (TQ2_0 kernels, bitnet arch, LUT/W2A8 borrow list) #5375

CRUX competitive review: microsoft/BitNet (bitnet.cpp) vs aprender. What to build, borrow and copy

Peer: microsoft/BitNet @ 0b341e5 (MIT). It is a llama.cpp fork plus Python glue, and it runs 1.58-bit ternary inference (I2_S / TL1 / TL2 CPU kernels and a W2A8 CUDA GEMV).
Subject: aprender origin/main (2026-10-15).

Where aprender stands today (measured)

• Formats: TQ1_0/TQ2_0 are named and sized only: crates/aprender-quant/src/ggml_type.rs:200-202, crates/aprender-serve/src/gguf/ggml_type_table.rs:217-228.
  • On CPU, fused_matmul_into.rs:33-96 has no TQ arm, so a ternary tensor fails with UnsupportedOperation.
  • On GPU, the capability contract has gpu_supported: false.
• Kernels: there is no ternary dequant, no ternary dot or matmul (AVX2, AVX-512, NEON, PTX or wgpu), and no lookup-table GEMV.
• Architecture: there is no bitnet family in contracts/model-families/. Nothing references relu2 (squared ReLU) or attn_sub_norm/ffn_sub_norm (subln).
• Activations: activation quant is per block only (Q8_0/Q8_K). BitNet's a8 needs per-token absmax int8.
• Conversion: apr quantize/convert have no absmean ternary quantizer.

Result: aprender cannot load or run any of the 17 HF checkpoints bitnet.cpp supports, including microsoft/BitNet-b1.58-2B-4T, the Falcon3-1.58bit family, Llama3-8B-1.58 and the BitNet embedding 0.6B/270M models.

BUILD (no peer code worth taking; do it natively)

1. bitnet model-family contract and forward pass:

   • squared ReLU FFN
   • sub-layer norms (attn_sub_norm, ffn_sub_norm)
   • per-tensor fp32 weight scale
   • per-token absmax int8 activation quant

   First target: microsoft/BitNet-b1.58-2B-4T.

2. TQ2_0 / TQ1_0 as the on-disk ternary format, with dequant plus a fused TQ2_0 × Q8_K dot on scalar, AVX2, AVX-512 and NEON in aprender-compute.

   • Use upstream-ggml types, not I2_S. bitnet.cpp's I2_S block layout depends on the CPU (128 on x86, 64 on ARM, src/ggml-bitnet-mad.cpp:11-15) while its Python packer hard-codes 128. So an I2_S GGUF is likely not portable across ISAs, and I2_S exists only in their fork.
   • Do read I2_S GGUFs at import time and repack them to TQ2_0, so their published GGUFs load.

3. apr quantize -s tq2_0: absmean ternary quantization from bf16 safetensors, so we don't depend on their converter chain.

BORROW (take the idea, re-implement in Rust)

4. The I2_S multiply-add kernel shape: unpack 2-bit weights and run an int8 dot (maddubs, or vdotq on ARM dotprod).
   • Variants: 1x1, 1x4, 1xN and Nx1 (ggml-bitnet-mad.cpp:198-1055).
   • Tiling is in include/gemm-config.h.
   • This is the simple, fast baseline and should land first.
5. TL1/TL2 lookup-table GEMV for decode:
   • TL1 builds a per-activation-pair 16-entry int8 table and looks it up with NEON vqtbl1q_s8.
   • TL2 packs 3 weights into a 4-bit index plus a sign bit (~1.67 bpw) and uses AVX2 pshufb with sign-xor.
   • Their version generates a C header per model shape from a hard-coded shape dictionary (utils/codegen_tl{1,2}.py), which raises NotImplementedError for unknown models. Use const generics and runtime tile selection instead, and keep BM/BK/bm as a tunable table (like their kernel_config.ini) rather than baked-in headers.
6. W2A8 CUDA GEMV (gpu/bitnet_kernels.cu):
   • __dp4a over int32 words, each holding 16 two-bit values, with 16x32 block permutation (gpu/pack_weight.py).
   • Port it as PTX in aprender-gpu/src/kernels/quantize/ next to the q4k dp4a kernels.
   • Theirs supports only M=1 and 10 fixed N×K shapes on sm_80. Ours must be shape-generic.

COPY (cheap, take the pattern nearly as-is)

7. Dummy-model generator (utils/generate-dummy-bitnet-model.py, --model-size 125M…100B): random ternary weights at a given scale, for kernel and throughput benches without downloads.
   • This is how they produce the "100B on one CPU at 5–7 tok/s" figure.
   • Add it as a test fixture or bench helper, e.g. apr bench --synthetic bitnet:<size>.
8. Their HF model list as a conformance set (setup_env.py:13-64): every repo there should apr import and apr run once this lands.
9. Perplexity and MTEB parity rows:
   • Their I2_S embedding is 0.7180 vs 0.7188 for the original on MTEB (docs/bitnet-embeddings-i2s-guide.md).
   • These become falsifiable acceptance numbers for apr ppl / apr embed.

Do NOT copy

• Python subprocess glue and the pinned personal llama.cpp fork: their FAQ Q1 is a build-break workaround.
• Generating per-model kernels at build time.
• The non-portable I2_S layout.
• --use-pretuned: it is a no-op, because codegen runs unconditionally after the pretuned files are copied (setup_env.py:168-206).
• e2e_benchmark.py / setup_env.py run_command: it calls sys.exit(1) unconditionally, so the benchmark exits 1 on success.
• No tests and no CI upstream: do not inherit their speed claims (2.37–6.17× on x86, 1.37–5.07× on ARM). They point to arXiv 2410.16144 and are not reproduced in the repo.

Acceptance (falsifiable; same host, same threads, measured, not quoted)

• apr run hf://microsoft/BitNet-b1.58-2B-4T generates coherent text on x86 (AVX2 and AVX-512) and on ARM (NEON).
• apr ppl on 2B-4T is within 1% of bitnet.cpp i2_s llama-perplexity on the same text.
• Decode tok/s at 1 thread and at N threads ≥ bitnet.cpp (e2e_benchmark.py -n 128 -p 512) on the same box, with numbers recorded in an evidence receipt.
• A ternary kernel falsification test: a planted wrong sign or scale must fail the dot-product parity test against a scalar reference.
• Capability contract rows: TQ2_0 is cpu_supported: true. GPU stays false until item 6 lands.

Same-night artifact CRUX (apr 0.67.0, local build, audit.sh self-test 20/20 armed)

• C7: apr convert, apr rm and apr serve change state but advertise no preview flag in their own --help. Cost: there is no way to learn what apr convert would do without letting it run.
• M1: the root Makefile has no help: target.

Suggested order: 1 → 2 → 4 → 3 → 8 → 5 → 6, with 7 and 9 alongside as test infrastructure.

🤖 Generated with Claude Code