[Bizuayeu]

What this changes

One paragraph at the end of the GLM recipe's "Prompt chunk size": at TP=3 with TF_GLM_ROCE_MULTI=1, TF_GLM_CHUNK_SOLO=4096 gains a little under half of what it gains on NCCL, at the run's own settings, and where the time goes. Documentation only.

• Issue: Refs GLM CUDA at TP=3: under TF_GLM_CHUNK_SOLO=4096, TF_GLM_ROCE_MULTI costs a cold 128K prompt about 7.5%, not 1 to 2% #1994, which has the measurements and the profile.
• Stacked on: main (this head is on ad98b602c and merges clean with d8b80bdd8).
• atc overlap --json: against main d8b80bdd8, no open pull request found on these lines. glm cuda: a prompt step takes two chunks while no other stream decodes, each one's exchanges under the other's sites #1423 changes the same section about 30 lines above, in another sentence, and the two merge without a conflict.

Checks on this head

• Head commit: 8ec088ee5 (the review's wording fix, amended into the one commit)
• Lean checker, python3 tools/zig/lean_check.py, on the base and on this head: 424 and 424, no new findings.
• Tests run, with the command and pass, fail and skip counts (Linux x86_64 under WSL2): zig build test 487/487 steps (6,463 of 6,509 tests, 46 skipped). No failures.
• CI: green, all seven checks on this head.
• Not run, and why: the CUDA and Metal steps, as no code changes.

Receipt

n/a: documentation only. The paragraph's numbers are #1994's run 1: main 9e8c368b0, three GB10s at TP=3, Bizuayeu/GLM-5.3-Flash-XSFP4 at 88736f91 with its MTP drafts, PARALLEL=8, TF_GLM_KV_SHARE=32, TENSORFOLD_PREFILL_SHARE=0.7, two fresh servers per arm, each cold prompt's first token from tools/bench_cache.py --require-cold. Where the time goes is from its run 2, with the profiler, and the last sentence from its run 3 (#1994 (comment)).

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency. (No new test: documentation only.)
• Comments and docstrings are one line. No measurements or history in the source. (The measurements are in the recipe, as its other rows are.)
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.

[ashhart]

Thanks @Bizuayeu. An independent review on current main (ef9fa70):
Verdict: LAND after one wording fix, once the numbers are checked against #1994. It adds one paragraph to the GLM
CUDA recipe.

Applying it: cleanly on main ef9fa70. The lean check stays at 424. The author line is @Bizuayeu's noreply.

Private names: none. The diff and commit name only "three GB10s", public env variables and issue numbers. There's
no hostname, address, path or person.

The references are right.

• TF_GLM_ROCE_MULTI=1 is glm cuda: at three ranks or more, TF_GLM_ROCE_MULTI=1 sends the small all-gathers over fabric/tp's multirank RoCE #1524's (CHANGELOG: off by default, three ranks or more, NCCL stays the default because a
  cold prompt was 1.2 to 1.6% slower with the wire).
• TF_GLM_CHUNK_SOLO=4096 is the recipe's existing solo setting (cuda_prompt_solo.zig, cuda_ranks.zig:19).

The receipts: #1994's run isn't in our inputs, so I can't check the paragraph's four numbers against it. The lead
should, on the issue. The PR body says they come from #1994's run 1 (main 9e8c368, --parallel 8,
TF_GLM_KV_SHARE=32, TENSORFOLD_PREFILL_SHARE=0.7, two fresh servers an arm, cold first tokens from
tools/bench_cache.py --require-cold). The attribution to the side stream's exchanges is from run 2's profile.

Changes:

1. "About half" overstates it a little. By its own numbers, 3.2 to 3.4% against 7.3 to 8.4% is 38% to 47% of the
   NCCL gain. Say "a little under half", or give the range.
2. Name the run's settings. The table just above is at --parallel 4 --context 524288, TF_KV_PAGES=on. This
   paragraph's numbers are at --parallel 8 with TF_GLM_KV_SHARE=32 and TENSORFOLD_PREFILL_SHARE=0.7, so they
   aren't on the same footing as the table's TP=3 column (7.2% to 8.5% there). One clause with those settings avoids a
   wrong comparison.

Run needed: none (docs only).

[Bizuayeu]

"About half" overstates it a little. […] Name the run's settings. (review)

Both are in 8ec088ee5, still one commit: "a little under half", and the run's settings in one clause, --parallel 8 with TF_GLM_KV_SHARE=32 and TENSORFOLD_PREFILL_SHARE=0.7, "not the table's settings". I also changed the last sentence to what #1994's run 3 supports: the wire's extra time "grows when solo doubles a pass", no longer "slow more than their size grows", since cutting each exchange to a 2,048-row pass's size takes back only 42% of it.

For the check against #1994: the ranges are its run 1 rows, solo with the wire −3.2, −3.2 and −3.4% at 8K, 32K and 128K, and without it −7.3, −8.4 and −8.3%.

A line on TF_GLM_OVERLAP_PIECES=6 will come in a PR of its own once I've measured it without the profiler.

[ashhart]

Thank you @Bizuayeu. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

[ashhart]

Thanks @Bizuayeu! This is in TensorFold 1.2.2, out now: https://github.com/ashhart/TensorFold/releases/tag/v1.2.2