ARCH: shared attention/FFN building blocks exist but 0% adopted — new architectures still cost ~1,600 lines each #3422

tl;dr

#3418 named the quant-dispatch duplication (30 files, no single source of truth) as
one reason a new model architecture is expensive to add. It is not the only reason,
and on its own it will not get aprender to "add Qwen3-Next MoE the way llama.cpp adds
it" — llama.cpp's qwen3moe.cpp is 179 lines because architectures are graphs
assembled from shared, quant-agnostic building blocks (build_attn, build_moe_ffn,
build_norm, ggml_rope_ext). This ticket is about that layer in aprender.

The good news, measured just now: aprender already has a version of this layer —
crates/aprender-serve/src/gguf/inference/forward/attention.rs (25 functions:
parallel_multihead_attention_gpu, standard_softmax, online_softmax,
tiled_single_head_attention, reshape_for_parallel_heads, …) and ffn_block.rs
(24 functions). This is not a from-scratch design problem.

The bad news: adoption is zero for the two newest, largest architectures.

Evidence (2026-09-17, origin/main @ 1d7dcc5e9)

$ for f in gguf/inference/forward/{single,forward_qwen35,forward_qwen3_moe,forward_qwen3_moe_gpu,forward_qwen3_moe_traced,forward_cached,gemma_dispatch,ffn_block,batch,batched,encoder_decoder}.rs; do
    p="crates/aprender-serve/src/$f"; [ -f "$p" ] && echo "$(grep -vc '^\s*//\|^\s*$' "$p") $f"
  done | sort -rn
939 gguf/inference/forward/forward_qwen35.rs
633 gguf/inference/forward/ffn_block.rs
328 gguf/inference/forward/forward_qwen3_moe_traced.rs
321 gguf/inference/forward/forward_qwen3_moe.rs
314 gguf/inference/forward/forward_cached.rs
284 gguf/inference/forward/batched.rs
167 gguf/inference/forward/encoder_decoder.rs
66  gguf/inference/forward/forward_qwen3_moe_gpu.rs
32  gguf/inference/forward/gemma_dispatch.rs

$ grep -cE 'standard_single_head_attention|parallel_multihead_attention|parallel_batched_qk_scores|standard_softmax|online_softmax|tiled_single_head_attention|reshape_for_parallel_heads' \
    gguf/inference/forward/forward_qwen35.rs gguf/inference/forward/forward_qwen3_moe.rs
forward_qwen35.rs:0
forward_qwen3_moe.rs:0

$ grep -cE 'ffn_block::|adaptive_ffn::' gguf/inference/forward/forward_qwen35.rs gguf/inference/forward/forward_qwen3_moe.rs
forward_qwen35.rs:0
forward_qwen3_moe.rs:0

Neither of the two newest architecture forwards calls a single function from either
shared module. Both reimplement their own attention inline (forward_attention at
forward_qwen35.rs:896) with their own scaling, softmax, and head-splitting logic.

Also measured, and this is the part #3418 doesn't cover: RoPE is implemented
independently in ~25 files, attention math in 20+, RMSNorm in 38
(grep -rl 'fn.*rmsnorm' crates/aprender-serve/src = 38). Quant-type dispatch (#3418)
is one axis of the duplication; forward-pass logic is a second, larger axis, and it's
the one that actually determines how many lines a new architecture costs.

Not all of it is avoidable duplication. forward_qwen35.rs also has legitimately
novel operators for the Qwen3.5/Next hybrid architecture — delta_rule_recurrence,
causal_conv1d, gated_rmsnorm, apply_partial_neox_rope (gated DeltaNet linear
attention interleaved with full attention) — that have no equivalent anywhere in the
shared modules, because nothing else in the tree needs them yet. llama.cpp had to add
graph ops for the same thing. The claim here is narrower: of that ~1,600-line surface,
some fraction is genuinely new math (unavoidable, needs a slot to plug into) and some
fraction is reimplementation of attention/softmax/head-reshaping that already exists
elsewhere in the same crate (avoidable, and today has zero enforcement stopping it).

Per-backend duplication compounds it. forward_qwen3_moe.rs (CPU, 321 lines) and
forward_qwen3_moe_gpu.rs (GPU, 66 lines) are separate entry points per architecture,
not one graph two backends walk — the same multiplication #3418 described for quant
dispatch, one layer up.

Relationship to #3418 / PP-QUANT-001

#3418 (quant-type dispatch, tracked as PP-QUANT-001, epic #3421) is a prerequisite,
not a substitute: a shared attn_block/moe_ffn_block composition layer needs to be
quant-agnostic to be reusable across architectures, which needs the quant dispatch
table PP-QUANT-001 is building. This ticket is the layer above it.

Ask

A dedicated design + implementation cycle (companion to PP-QUANT-001, own spec) whose
exit criterion is measurable: a new dense or MoE architecture (the next real test case
being Qwen3-Next / Qwen3.8-Flash, already sitting untested per #3418) lands as a small
loader-mapping file plus a forward composed from shared blocks — not a
~1,600-line reimplementation — while the Qwen2.5-Coder GPU-parity measurement
(docs/BEATS.md) shows no regression.

Related

• ARCH: quant-type dispatch is duplicated across ~30 files with no single source of truth — propose a dedicated consolidation release #3418 / PP-QUANT-001 (epic EPIC: PP-QUANT-001 — quant-type dispatch, Phase 0/1 (0.69 train) #3421) — prerequisite: quant-agnostic dispatch table
• P0: dense Qwen3 (Qwen3ForCausalLM) has no current proof it runs — last verification is 7 months stale and CI never covers it #3413 — dense Qwen3 proof-of-life gap, downstream of the same forward-path duplication
• crates/aprender-serve/src/gguf/inference/forward/attention.rs,
  ffn_block.rs — the existing, under-adopted shared layer