refactor(examples): group per-backend material under its backend dire…

[Xiaoming-AMD]

Why

Every backend already owns a top-level directory named after the framework field in its experiment YAML, but model studies and walkthroughs for those backends sat as siblings of the backends. So examples/ mixed backends with Megatron-only content, and nothing told you which backend deepseek-v4/ or moe_package/ belonged to.

What moved

Megatron-specific material now lives under examples/megatron/:

New location	Was
megatron/models/deepseek-v4/	examples/deepseek-v4/
megatron/models/kimi-k3/	examples/models/kimi-k3/
megatron/guides/customer_package/	examples/customer_package/
megatron/guides/moe_package/	examples/moe_package/
megatron/guides/odc/	examples/odc/
megatron/guides/offline_tune/	examples/offline_tune/
megatron/guides/tuning_agent/	examples/agents/tuning_agent/

mlperf/ and hardware_configs/ deliberately stay at the top level: MLPerf submissions span Megatron, Megatron-Bridge and diffusion, and the hardware configs feed primus projection rather than any one backend. examples/README.md gains a Directory Layout section stating the rule.

Backend directories keep their top-level position, which is what makes this safe: the two places that resolve a backend directory by name at runtime — examples/scripts/prepare_experiment.py building examples/<framework>/prepare.py, and SCAN_ROOTS in primus/core/config/schema_check.py scanning examples/<backend>/configs — are untouched, as are the from examples.scripts.utils import ... imports.

Beyond renaming

Ten scripts derive the repo root from their own location, e.g.

REPO="$(cd "${HERE}/../../.." && pwd)"

They are two levels deeper now, so those ../ chains were extended and each one was verified to still resolve to the repo root.

.gitignore (the negation that un-ignores the projection site's JSON), the deploy-backend-gap-dashboard path filter, tools/backend_gap_report/pages-sections.json, and the HipBLASLt stage-2 call in runner/helpers/hooks/train/pretrain/prepare_experiment.sh all hard-code moved paths and were updated too.

Verification

• All 105 references to the moved paths were rewritten; zero occurrences of the old paths remain.
• A checker resolved all 609 literal examples/... references in tracked files against main and against this branch. Unresolved paths went from 61 to 59 — no new breakage, and two fewer because the tuning-agent docs now point at target_cluster_mi355x_4nodes.yaml, the only cluster file that exists (they referenced target_cluster.yaml and ..._2nodes.yaml, neither of which was ever added). The remaining 59 are pre-existing and untouched: synthetic fixtures in test_config_schema_check.py, upstream paths such as Megatron-LM's examples/multimodal/, and shell strings the scanner truncates at +-concatenation.
• pre-commit run passes on all 98 changed files, including shellcheck.
• pytest tests/runner/test_docs_cli_examples.py tests/unit_tests/configs/test_config_schema_check.py — 64 passed.

One unrelated fix, needed to get CI green

test_documented_script_invocations_exist already fails on main (verified on a clean checkout): docs/02-user-guide/jax-maxtext-training.md runs bash tools/fetch_primus.sh, which is not in this repository. That command runs after cd MAD, so it belongs to the external ROCm/MAD checkout — it is allow-listed here with that rationale. #999 carries the identical fix, so if that merges first this becomes a trivial duplicate-line conflict.

Note for #999

examples/maxdiffusion/prepare.py should be deleted there: that PR migrates the per-backend prepare.py files into runner/helpers/hooks/train/pretrain/, but this one survived and still does from examples.scripts.utils import ..., a module the same PR deletes. Nothing references it and runner/helpers/hooks/train/pretrain/maxdiffusion/prepare.py supersedes it.

Reviewing

git show --find-renames (or ?w=1 with rename detection) keeps this to the ~40 content lines; the rest is pure movement.

[Copilot]

🔵 Needs a closer look

The newly added RCCL AVG workaround can produce incorrect results with async_op=True, and the new offline tuning script has unguarded parsing paths that can crash at runtime.

Pull request overview

This PR reorganizes examples/ content so that backend-specific models and guides live under their owning backend directory (not as siblings), while updating hard-coded paths and documentation references to match the new layout.

Changes:

• Move Megatron-specific studies/guides under examples/megatron/ and update in-repo references accordingly.
• Update scripts that derive the repo root from their own path to account for the deeper directory nesting.
• Adjust ancillary wiring (dashboard config, GitHub Actions path filters, .gitignore, docs/tests allowlists) to point at the moved projection and guide assets.

File summaries

File	Description
tools/backend_gap_report/pages-sections.json	Updates projection site source path for the DeepSeek-V4 dashboard section.
tests/runner/test_docs_cli_examples.py	Allow-lists an external MAD-only script and updates moved-path allowlist entry.
runner/helpers/hooks/train/pretrain/prepare_experiment.sh	Updates offline-tune script path for the Megatron prepare hook.
README.md	Updates the DeepSeek-V4 examples link to the new Megatron location.
primus/configs/models/megatron/kimi_k3_curve.yaml	Updates comment reference to the moved Kimi-K3 run script.
primus/agents/tuning_agent/README.md	Updates example --target-cluster path to the moved tuning-agent guide location.
primus/agents/tuning_agent/cli.py	Updates usage docstring to reference the moved target_cluster YAML.
examples/README.md	Adds a “Directory Layout” section documenting the new per-backend organization and updates an offline_tune link.
examples/megatron/models/kimi-k3/run_kimi_k3_curve_pretrain_mi355x.sh	Updates embedded path references after relocation under examples/megatron/models/.
examples/megatron/models/kimi-k3/run_kimi_k3_8L_official_pretrain_mi355x.sh	Updates usage comment to the new script path.
examples/megatron/models/deepseek-v4/run_v4_4layer_sft.sh	Fixes repo-root resolution after moving the script deeper under examples/megatron/models/.
examples/megatron/models/deepseek-v4/run_dsv4_projection_1gpu.sh	Updates repo-root resolution and PYTHONPATH references for moved projection assets.
examples/megatron/models/deepseek-v4/run_deepseek_v4_pro_muon_local.sh	Updates repo-root resolution and PYTHONPATH references for moved RCCL workaround assets.
examples/megatron/models/deepseek-v4/run_deepseek_v4_flash.sh	Updates embedded path references and repo-root resolution after relocation.
examples/megatron/models/deepseek-v4/rccl_avg_workaround/sitecustomize.py	Adds runtime workarounds (RCCL AVG reduction + primus_turbo import stub) for certain single-GPU containers.
examples/megatron/models/deepseek-v4/rccl_avg_workaround/.gitignore	Ignores bytecode artifacts for the new workaround module.
examples/megatron/models/deepseek-v4/projection/tools/v4_flops.py	Adds an analytic FLOPs model used to align projection TFLOP/s with Megatron reporting.
examples/megatron/models/deepseek-v4/projection/tools/kernel_module_map.py	Adds kernel/stack-to-module and flop-class mapping logic for projection attribution.
examples/megatron/models/deepseek-v4/projection/tools/gen_mock_data.py	Updates usage string for the moved projection tooling path.
examples/megatron/models/deepseek-v4/projection/site/index.html	Updates displayed “Generated from …” path to the moved projection directory.
examples/megatron/models/deepseek-v4/projection/script/deepseek_v4_layer_trace-projection.sh	Updates comments/usage references to the moved projection pipeline path.
examples/megatron/models/deepseek-v4/projection/README.md	Updates directory layout and command examples to the moved projection path.
examples/megatron/models/deepseek-v4/projection/design/07-iteration-timeline.md	Adds/updates design documentation for the projection site’s iteration timeline view.
examples/megatron/models/deepseek-v4/projection/design/06-calibration.md	Updates calibration instructions to use the moved projection scripts/tools.
examples/megatron/models/deepseek-v4/projection/design/05-deployment.md	Updates deployment docs and workflow path filter references for the moved site.
examples/megatron/models/deepseek-v4/projection/design/04-projection-math.md	Adds/updates the authoritative projection math derivation documentation.
examples/megatron/models/deepseek-v4/projection/design/03-json-schema.md	Adds/updates the breakdown JSON schema documentation for the projection site.
examples/megatron/models/deepseek-v4/projection/design/02-assumptions.md	Adds/updates the assumptions document used by the projection methodology.
examples/megatron/models/deepseek-v4/projection/design/01-overview.md	Adds/updates the projection methodology overview documentation.
examples/megatron/models/deepseek-v4/gfx942/run_full_4k_thd_multinode.sh	Updates usage comments and repo-root resolution after relocation.
examples/megatron/models/deepseek-v4/gfx942/run_full_4k_multinode.sh	Updates usage comments, repo-root resolution, and delegate script path.
examples/megatron/models/deepseek-v4/gfx942/run_full_128k_thd_multinode.sh	Updates usage comments after relocation.
examples/megatron/models/deepseek-v4/gfx942/run_128k_thd_packed.sh	Updates usage comments and repo-root resolution after relocation.
examples/megatron/models/deepseek-v4/gfx942/run_128k_dense_hca_csa.sh	Updates usage comments, repo-root resolution, and README path after relocation.
examples/megatron/models/deepseek-v4/gfx942/README.md	Updates the primary invocation example to the moved script path.
examples/megatron/models/deepseek-v4/gfx942/README_thd_packed.md	Updates referenced script paths for moved THD packed recipes.
examples/megatron/models/deepseek-v4/gfx942/README_full_4k_multinode.md	Updates referenced script paths for moved multinode recipe.
examples/megatron/models/deepseek-v4/gfx942/prepare_sft_data.py	Adds data-prep helper for long-form SFT jsonl used by gfx942 recipes.
examples/megatron/models/deepseek-v4/gfx942/prepare_packed_data.py	Adds packer that bins variable-length SFT samples into fixed-size THD bins.
examples/megatron/models/deepseek-v4/benchmark/bench_v4_attention.py	Updates run instructions to the moved benchmark path.
examples/megatron/models/deepseek-v4/benchmark/bench_v4_attention_results.md	Updates reproduction command path to the moved benchmark script.
examples/megatron/guides/tuning_agent/target_cluster_mi355x_4nodes.yaml	Adds a concrete tuning-agent target cluster YAML under the Megatron guides tree.
examples/megatron/guides/offline_tune/README.md	Adds offline HipBLASLt tuning walkthrough under the Megatron guides tree.
examples/megatron/guides/offline_tune/offline_tune_gemm.py	Adds an offline hipblaslt-bench tuning helper script.
examples/megatron/guides/moe_package/start_training_qwen_30B_a3B.sh	Updates repo-root resolution after moving the MoE package guide deeper.
examples/megatron/guides/moe_package/start_training_dsv2_lite.sh	Updates comments and repo-root resolution for the moved MoE package guide script.
examples/megatron/guides/moe_package/run_qwen_30B_a3B_pretrain_mi355x.sh	Adds a Qwen pretrain launcher under the moved MoE package guide directory.
examples/megatron/guides/moe_package/run_qwen_235B_a22B_pretrain_mi355x.sh	Adds a Qwen 235B pretrain launcher under the moved MoE package guide directory.
examples/megatron/guides/moe_package/run_minimax_m2.5_pretrain_mi355x_pretrain_mi355x.sh	Adds a MiniMax M2.5 pretrain launcher under the moved MoE package guide directory.
examples/megatron/guides/moe_package/run_minimax_m2.5_4layers_proxy.sh	Adds a small proxy launcher for MiniMax M2.5 under the moved MoE package guide directory.
examples/megatron/guides/moe_package/run_gpt_oss_120B_mi355x.sh	Adds/updates a GPT-OSS 120B launcher under the moved MoE package guide directory.
examples/megatron/guides/moe_package/run_glm5_pretrain_mi355x.sh	Adds a GLM-5 launcher under the moved MoE package guide directory.
examples/megatron/guides/moe_package/run_glm5_4layers_proxy.sh	Adds a GLM-5 4-layer proxy launcher under the moved MoE package guide directory.
examples/megatron/guides/moe_package/run_deepseek_v3_pretrain_mi355x.sh	Adds a DeepSeek-V3 launcher under the moved MoE package guide directory.
examples/megatron/guides/moe_package/run_deepseek_v3_4layers_proxy_pretrain_mi355x.sh	Adds a DeepSeek-V3 4-layer proxy launcher under the moved MoE package guide directory.
examples/megatron/guides/moe_package/run_deepseek_v2_pretrain_mi355x.sh	Updates EXP path to the moved MoE-package configs directory.
examples/megatron/guides/moe_package/run_deepseek_v2_lite_pretrain_mi355x.sh	Updates EXP path to the moved MoE-package configs directory.
examples/megatron/guides/moe_package/README.md	Updates links to moved MoE package run scripts.
examples/megatron/guides/moe_package/configs/MI355X/deepseek_v2-pretrain-baseline.yaml	Adds a baseline DeepSeek-V2 experiment YAML under the moved MoE-package guide.
examples/megatron/guides/moe_package/configs/MI355X/deepseek_v2_lite-pretrain-baseline.yaml	Adds a baseline DeepSeek-V2-Lite experiment YAML under the moved MoE-package guide.
examples/megatron/guides/customer_package/run_qwen3_30b_a3b_pretrain_mi355x.sh	Adds a customer package launcher script under the moved Megatron guides tree.
examples/megatron/guides/customer_package/run_qwen_8b_pretrain_mi355x.sh	Adds a customer package launcher script under the moved Megatron guides tree.
examples/megatron/guides/customer_package/run_llama3.1_8b_pretrain_mi355x.sh	Adds a customer package launcher script under the moved Megatron guides tree.
examples/megatron/guides/customer_package/run_llama3.1_70b_pretrain_mi355x.sh	Adds a customer package launcher script under the moved Megatron guides tree.
examples/megatron/configs/MI355X/qwen3_30B_A3B-BF16-sft-packed.yaml	Updates embedded reference to the moved MoE-package helper script.
examples/megatron/configs/MI355X/kimi_k3-BF16-8L-official.yaml	Updates embedded reference to the moved Kimi-K3 launcher script.
docs/07-technical-blogs/README.md	Updates the DeepSeek-V4 blog’s “In this repo” link to the moved examples location.
docs/04-technical-guides/performance-tuning.md	Updates offline_tune references to the moved Megatron guide location.
docs/04-technical-guides/multi-node-networking.md	Updates customer_package path reference to the moved Megatron guide location.
docs/04-technical-guides/moe-training.md	Updates MoE-package paths throughout the guide to match the moved structure.
docs/02-user-guide/tuning-agent.md	Updates tuning-agent target_cluster example path and link to the moved YAML.
docs/02-user-guide/pretraining.md	Updates offline-tune tooling path reference to the moved Megatron guide location.
.gitignore	Updates “un-ignore” exceptions for the moved DeepSeek-V4 projection site JSON data directory.
.github/workflows/deploy-backend-gap-dashboard.yml	Updates workflow path filters to the moved DeepSeek-V4 projection site directory.

Review details

• Files reviewed: 44/98 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.