diff --git a/README.md b/README.md index e5cf530..57c779f 100644 --- a/README.md +++ b/README.md @@ -38,6 +38,11 @@ does not replace packages in the core Multi-TurboQuant environment. The separate `mtq-godzilla-gigatoken` workflow can also prepare, build, and qualify a revision-pinned Godzilla llama.cpp runtime with native Gigatoken tokenization without modifying an existing checkout. +For the exact Godzilla `09214b160` compatibility baseline, +`mtq-godzilla-compose` can create a separate hash-tracked source tree with an +off-by-default, request-gated PFlash policy and bounded whole-idle-slot +KVFlash residency. It does not claim arbitrary KV-page restore or remove +Godzilla's existing TriAttention/KVarN conflict. ```bash git clone https://github.com/aivrar/multi-turboquant @@ -382,6 +387,46 @@ The validated server is under `build-gigatoken-/bin` (and `bin/Release` for multi-config Windows builds). The default build may serve the API without an embedded browser UI; launch it with a local GGUF model path. +### Exact Godzilla PFlash/KVFlash composition + +`mtq-godzilla-compose` creates a new source tree pinned to Godzilla commit +`09214b160b402011359f0ef9d5fa8f8be1112e85`, applies exact fail-closed source +edits, records hashes for every changed runtime file, and builds only +`llama-server`. It refuses existing destinations and arbitrary Godzilla +checkouts. + +```bash +# Read-only plan +mtq-godzilla-compose plan /opt/godzilla-composed + +# Prepare, build, and verify the CPU server +mtq-godzilla-compose all /opt/godzilla-composed \ + --backend cpu --max-jobs 2 --yes + +# Separate CUDA build with the matching toolkit +mtq-godzilla-compose build /opt/godzilla-composed \ + --backend cuda --cuda-toolkit /usr/local/cuda-12.6 --max-jobs 2 --yes +``` + +Both additions are disabled by default. Start with `--pflash` to permit the +feature, then set `"pflash": true` only on a plain completion request that has +passed workload-specific quality checks. Chat, multimodal, embedding, rerank, +and parallel-parent requests are bypassed. Prefix and suffix tokens are +protected, while the eligible middle is thinned deterministically. + +`--kvflash-pages N --kvflash-page-tokens 256` sets a token-accounted LRU budget +for complete idle-slot KV state. This is a useful, composable server-residency +tier; it is not the research fork's unfinished arbitrary-page restore path and +does not claim disk restore or prefill skipping. `/props` reports the exact +composition profile and tier, while application and eviction events are sent +to the server log. + +The overlay leaves DFlash/DDTree, KVarN, and TriAttention internals unchanged. +PFlash may be used before either KVarN or TriAttention, and whole-slot KVFlash +does not change their cache representation. Godzilla still rejects +TriAttention plus KVarN, and SpecLA remains outside this profile because it is +a specialized linear-attention runtime rather than a stackable add-on. + ### Godzilla KVarN and DFlash KVarN is available through `atomicmilkshake/godzilla-llama.cpp`, not upstream @@ -607,7 +652,9 @@ builds, and import validation. Constrained research sources, training systems, and separate serving runtimes instead receive repository-specific setup contracts. This includes Lucebox, ChunkLlama, RaBitQCache, ScoPE, DuoAttention, IceCache, the PFlash/KVFlash llama.cpp fork, and the guarded -Resonance-JetLong composition. In particular, current Maru checkouts are +Resonance-JetLong composition. The separate `godzilla_composition` catalog +entry documents the exact pinned overlay rather than treating the HawgAuto +fork as an installable add-on. In particular, current Maru checkouts are recognized through `pyproject.toml`/`setup.py` and the `maru_resource_manager`/`maru_server` packages—no root `CMakeLists.txt` is required. Maru remains guided-only because its upstream installer expects a @@ -734,7 +781,8 @@ The browser UI now has two focused views: issue #43 projects alongside the earlier add-ons, domvox, the CUDA weight-share source, and separate llama.cpp forks without importing or executing source code; guided and blocked profiles are not made installable - by discovery. The pinned Godzilla + Gigatoken source preparation + by discovery. Prepared exact-commit composition trees are recognized by their + manifest and routed through the hash-bounded inspector. The pinned Godzilla + Gigatoken source preparation and build remains an explicit `mtq-godzilla-gigatoken` CLI operation. Settings and form defaults persist in @@ -758,7 +806,7 @@ multi_turboquant/ methods/ 5 method families, all with encode/decode kernels/triton/ Attention backend, vectorized encode, dispatch calibration/ Weight-norm analysis, TriAttention adapters, parity wrapper - integration/ llama.cpp flags, Godzilla/Gigatoken builder, weight sharing, vLLM patch + integration/ llama.cpp flags, pinned Godzilla builders, weight sharing, vLLM patch benchmark/ Head-to-head comparison, perplexity, VRAM profiling ``` @@ -798,6 +846,7 @@ missing: | Gigatoken Python tokenizer accelerator | [marcelroed/gigatoken](https://github.com/marcelroed/gigatoken) | | Gigatoken llama.cpp runtime integration lineage | [chynggi/gigatoken-llama.cpp](https://github.com/chynggi/gigatoken-llama.cpp) | | Issue #43 optimization research and source contracts | [JetSpec](https://github.com/hao-ai-lab/JetSpec), [Lucebox](https://github.com/Luce-Org/lucebox), [Proxima](https://github.com/Tenosra/Proxima), [Jet-Long](https://github.com/jet-ai-projects/jet-long), [ChunkLlama](https://github.com/HKUNLP/ChunkLlama), [RaBitQCache](https://github.com/Sakuraaa0/RaBitQCache), [ScoPE](https://github.com/oncemoe/ScoPE), [DuoAttention](https://github.com/mit-han-lab/duo-attention), [IceCache](https://github.com/yuzhenmao/IceCache), and [PFlash/KVFlash llama.cpp](https://github.com/HawgAuto/llama.cpp-dflash-pflash-kvflash) | +| Exact Godzilla PFlash/KVFlash composition request | [Godzilla llama.cpp](https://github.com/atomicmilkshake/godzilla-llama.cpp), [PFlash/KVFlash fork](https://github.com/HawgAuto/llama.cpp-dflash-pflash-kvflash), and issue [#44](https://github.com/aivrar/multi-turboquant/issues/44) | We reimplemented the Python-native algorithms in Python. Godzilla/KVarN support is a command-generation, source-inspection, and preparation-workflow @@ -831,6 +880,7 @@ contracts and do not imply runtime compatibility. | Requested Debian 12/13 hardening, deeper diagnostics, the exact Godzilla `09214b160` compatibility profile, domvox/Gigatoken support, and reviewed CUDA weight-share source handling | [@jawadala](https://github.com/jawadala) | Issue [#40](https://github.com/aivrar/multi-turboquant/issues/40) | | Reported a domvox calibration launch under an interpreter without Torch, prompting compatible-environment discovery, fail-closed final preflight, exact-environment conversion, and detailed redacted failure bundles | [@jawadala](https://github.com/jawadala) | Issue [#42](https://github.com/aivrar/multi-turboquant/issues/42) | | Proposed the JetSpec, Lucebox, Proxima, Jet-Long, ChunkLlama, RaBitQCache, ScoPE, DuoAttention, IceCache, PFlash/KVFlash, and Resonance-JetLong review, prompting pinned source profiles, read-only discovery contracts, runtime capability scanning, and fail-closed composition metadata | [@jawadala](https://github.com/jawadala) | Issue [#43](https://github.com/aivrar/multi-turboquant/issues/43) | +| Requested a safe PFlash/KVFlash composition path for the exact Godzilla `09214b160` baseline, prompting the pinned overlay, request and runtime guardrails, and build verification | [@jawadala](https://github.com/jawadala) | Issue [#44](https://github.com/aivrar/multi-turboquant/issues/44) | | ForgeAttention — fused MLX kernels for Apple Silicon (`multi_turboquant/kernels/metal/`): packed-3-bit fused QK, tiled SV, flash decode, sparse SV with phase-1/2 early exit, per-head attention budget calibration | [@user-23xyz](https://github.com/user-23xyz) | PR [#1](https://github.com/aivrar/multi-turboquant/pull/1) · sibling project [user-23xyz/forgeattention](https://github.com/user-23xyz/forgeattention) | Thank you to [@jawadala](https://github.com/jawadala) for the sustained issue @@ -839,7 +889,8 @@ Godzilla/KVarN support, context-extension tooling, optimization catalog, isolated dependency system, practical UI workflow, and the official and domvox TriAttention calibration paths, including the interpreter-path correction, parity-checked Gigatoken option, and the broader issue #43 research -catalog with explicit safety boundaries. +catalog with explicit safety boundaries, including the exact-commit composition +workflow prompted by issue #44. The Metal path is community-maintained — the maintainer does not have Apple Silicon hardware, so issues specific to MLX/Metal should tag the contributor for context. diff --git a/docs/context-extension.md b/docs/context-extension.md index e05c76e..2d0f1a0 100644 --- a/docs/context-extension.md +++ b/docs/context-extension.md @@ -140,6 +140,25 @@ runtime workflow ports the reviewed tokenizer changes onto either exact Godzilla v0.3.7 or `09214b160` source profile and runs differential plus legacy tokenizer tests. Neither path changes the KVarN/TriAttention guardrail. +Issue #44 adds a separate exact-commit composition workflow. Running +`mtq-godzilla-compose` against a new destination prepares Godzilla +`09214b160b402011359f0ef9d5fa8f8be1112e85` with two deliberately bounded +server additions: + +- PFlash is off by default, requires startup and per-request opt-in, protects + prompt edges, and bypasses chat, multimodal, embedding, rerank, and + parallel-parent requests. +- KVFlash retains complete idle-slot KV states under a token-accounted LRU + budget. It does not implement or claim arbitrary-page restore, disk restore, + or prefill skipping. + +Because the KVFlash tier never edits individual cache entries, it stays outside +KVarN's representation and TriAttention's pruning domain. PFlash can precede +either path, but it remains a lossy prefill policy requiring model/workload +quality validation. DFlash/DDTree remains available. TriAttention plus KVarN +is still rejected, and SpecLA is not included because it is a different +linear-attention runtime. + ## Sources - llama.cpp server documentation: diff --git a/docs/manual.md b/docs/manual.md index 408e7e9..41b55c4 100644 --- a/docs/manual.md +++ b/docs/manual.md @@ -917,6 +917,45 @@ optional and can be supplied with `--fixture-dir`; missing optional fixtures are not registered as failures. `verify` reruns the suites and checks the resulting `llama-server --version` output. +### Exact Godzilla PFlash/KVFlash composition + +`mtq-godzilla-compose` is a separate source-preparation workflow for the exact +Godzilla `09214b160b402011359f0ef9d5fa8f8be1112e85` baseline. It creates a new +destination, applies exact-anchor edits, records hashes for all adapted runtime +files, builds only `llama-server`, and verifies that the resulting binary still +advertises Godzilla DDTree/TriAttention together with the new PFlash/KVFlash +surface. Existing destinations and arbitrary checkouts are rejected. + +```bash +mtq-godzilla-compose plan /opt/godzilla-composed +mtq-godzilla-compose all /opt/godzilla-composed \ + --backend cpu --max-jobs 2 --yes +mtq-godzilla-compose build /opt/godzilla-composed \ + --backend cuda --cuda-toolkit /usr/local/cuda-12.6 --max-jobs 2 --yes +mtq-godzilla-compose verify /opt/godzilla-composed --backend cpu +``` + +PFlash has two gates: the server must start with `--pflash`, and an eligible +plain completion request must contain `"pflash": true`. It remains off for +chat, multimodal, embedding, rerank, and parallel-parent requests, preserves a +protected prefix and suffix, and does not operate below +`--pflash-min-tokens`. `--pflash-keep-ratio` is restricted to `(0, 1]`. +Because prompt thinning is lossy, validate exact-task bypass, retrieval, code, +perplexity, and task quality before enabling it for a route. + +KVFlash in this profile is deliberately a whole-idle-slot LRU residency tier: +`--kvflash-pages N --kvflash-page-tokens 256` defines its token-accounted +budget. It does not manipulate KVarN pages or TriAttention entries and does not +claim arbitrary-page restore, disk restore, or prefill skipping. `/props` +reports the configuration and truthful capability limits; PFlash applications +and KVFlash evictions are logged. + +DFlash/DDTree can remain active with the overlay. PFlash may precede either +KVarN or TriAttention, and the whole-slot residency tier can retain either +representation. TriAttention plus KVarN remains rejected by Godzilla itself. +SpecLA is excluded because its linear-attention runtime is not a generic +Godzilla add-on. + ### CUDA weight-share launch wrapper For Linux x86-64 + CUDA multi-process serving, Multi-TurboQuant recognizes and @@ -1556,6 +1595,8 @@ multi_turboquant/ llamacpp_args.py Generate llama.cpp CLI flags godzilla_gigatoken.py Pinned combined-source preparation, build, and verification godzilla_gigatoken_cli.py Confirmed runtime workflow CLI + godzilla_composition.py Exact-commit PFlash/KVFlash overlay and compatibility contract + godzilla_composition_cli.py Confirmed composition preparation/build CLI weight_share.py CUDA LD_PRELOAD launch wrapper vllm_patch.py Monkeypatch vLLM for all methods bridge_adapter.py Adapter for Llama_TQ bridge apps @@ -1627,6 +1668,11 @@ multi_turboquant.integration.plan_godzilla_gigatoken(target, action="prepare") multi_turboquant.integration.prepare_godzilla_gigatoken(plan, confirmed=True) multi_turboquant.integration.build_godzilla_gigatoken(plan, confirmed=True) multi_turboquant.integration.verify_godzilla_gigatoken(plan) +multi_turboquant.integration.validate_godzilla_composition(config) +multi_turboquant.integration.plan_godzilla_composition(target, action="prepare") +multi_turboquant.integration.prepare_godzilla_composition(plan, confirmed=True) +multi_turboquant.integration.build_godzilla_composition(plan, confirmed=True) +multi_turboquant.integration.verify_godzilla_composition(plan) multi_turboquant.integration.patch_vllm(config) multi_turboquant.integration.is_vllm_patched() multi_turboquant.integration.BridgeAdapter(config) @@ -1706,6 +1752,7 @@ these authors for the mathematical ideas and research: | Debian 12/13 support, exact Godzilla `09214b160` profile, domvox/Gigatoken integration, diagnostics, and reviewed CUDA weight-share source workflow | jawadala / issue #40 | Community contribution | | Compatible TriAttention interpreter discovery, fail-closed dependency checks, exact-environment domvox conversion, and detailed calibration failure reports | jawadala / issue #42 | Community contribution | | JetSpec, Lucebox, Proxima, Jet-Long, ChunkLlama, RaBitQCache, ScoPE, DuoAttention, IceCache, PFlash/KVFlash, and Resonance-JetLong review; pinned source profiles, source contracts, scanner coverage, and fail-closed composition | jawadala / issue #43 | Community contribution | +| Exact-commit Godzilla PFlash/KVFlash composition workflow and safe pairing boundaries | jawadala / issue #44 | Community contribution | We reimplemented the Python-native algorithms in Python under a unified API. Godzilla/KVarN support is a command-generation, source-inspection, and preparation-workflow integration; context-extension support is a llama.cpp command-generation and capability-scanning integration only. This repository does not bundle Godzilla, BeeLlama, KVarN, Resonance RoPE, LongRoPE, domvox, Gigatoken, CUDA weight sharing, the issue #43 research projects, or llama.cpp source trees; installable workflows use reviewed revisions in isolated environments, while guided entries remain read-only contracts. Credit goes to the upstream authors for the technical work, and thank you to @jawadala for the sustained issue reports and concrete suggestions that identified the Godzilla/KVarN integration target, context-extension/UI scanner work, optional dependency workflow, consolidated UI workspace, official and domvox calibration paths, parity-checked Gigatoken options, and the broader fail-closed optimization review. diff --git a/docs/optimizations.md b/docs/optimizations.md index 8795e8f..0ce09e3 100644 --- a/docs/optimizations.md +++ b/docs/optimizations.md @@ -99,6 +99,7 @@ a separately implemented and validated LMCache SERDE. | DuoAttention | Guided research source | Model-specific head patterns and multiple native CUDA components require separate qualification | | IceCache | Guided research source | Requires the separate M-DCI artifact, native builds, OpenBLAS/OpenMP, and a high-core-count CPU workflow | | PFlash/KVFlash llama.cpp | Guided separate runtime | Hawg33 fork is a separate llama.cpp family, not an arbitrary patch for Godzilla | +| Godzilla composition | Experimental pinned native workflow | Exact `09214b160` overlay with request-gated PFlash and complete-idle-slot KVFlash residency | | Resonance-JetLong | Native backend required | Reviewed only for JetLong `yarn`/`jetlong_freq`; Qwen3 remains a gated research hypothesis | ## Composition rules @@ -311,6 +312,7 @@ eligible for the reviewed profile. | `duoattention` | Blocked | Model-specific head patterns and native CUDA stack | Requires exact-model artifact and runtime qualification | | `icecache` | Blocked | M-DCI artifact, C++/CUDA builds, OpenBLAS/OpenMP, and CPU-heavy indexing | Requires upstream prerequisites and independent evaluation | | `pflash_llamacpp` | Blocked | Separate Hawg33 llama.cpp runtime family | Inspect/build the fork separately; do not apply it as a Godzilla patch | +| `godzilla_composition` | Guided | Exact Godzilla `09214b160` source overlay | Use `mtq-godzilla-compose`; PFlash is lossy and KVFlash is limited to complete idle slots | | `resonance_jetlong` | Blocked | Research composition requiring a trained compatible artifact and long-context gates | Planner metadata only; no automatic training or serving claim | Blocked rows are catalog records, not failed installations. They intentionally @@ -349,6 +351,24 @@ tree, verifies all source revisions and hashes, builds with `LLAMA_GIGATOKEN=ON`, and runs both the differential and legacy tokenizer suites. It does not patch a selected or arbitrary checkout. +The Hawg33 PFlash/KVFlash catalog record remains blocked as a separate upstream +runtime family. The narrower `godzilla_composition` workflow is distinct: it +creates a new tree from exact Godzilla commit +`09214b160b402011359f0ef9d5fa8f8be1112e85` and applies a reviewed local +overlay. PFlash is disabled unless both `--pflash` and per-request +`"pflash": true` are present, and it bypasses chat, multimodal, embedding, +rerank, and parallel-parent requests. KVFlash is implemented only as an LRU +budget over complete idle slots. Arbitrary KV pages, disk restore, and prefill +skip remain explicitly unsupported. + +This narrower residency boundary avoids modifying KVarN's representation or +TriAttention's intra-sequence pruning. DFlash/DDTree can remain enabled, and +PFlash may run before either KVarN or TriAttention. The upstream +TriAttention/KVarN conflict is unchanged, and SpecLA is excluded because it is +a specialized linear-attention runtime. Use `mtq-godzilla-compose plan` before +preparation, then run CPU/CUDA build qualification and workload-specific +PFlash quality plus KVFlash concurrency/soak tests. + ### CUDA weight-share source The source picker recognizes `pontostroy/cuda-llm-weight-share` only when its diff --git a/docs/ui-workspace.md b/docs/ui-workspace.md index 8aa8791..d1cf123 100644 --- a/docs/ui-workspace.md +++ b/docs/ui-workspace.md @@ -199,13 +199,17 @@ picker can also inspect local folders for guided or blocked entries including Maru, Speculative Prefill, RocketKV, Lexico, AdaDecode, Resonance YaRN, Lucebox, ChunkLlama, RaBitQCache, ScoPE, DuoAttention, IceCache, and the PFlash/KVFlash llama.cpp fork, plus domvox TriAttention and the reviewed CUDA -weight-share source. Resonance-JetLong is instead a planner-only composition +weight-share source. A prepared `mtq-godzilla-compose` tree is recognized from +its manifest and checked with the hash-bounded composition inspector rather +than labeled as a generic Godzilla checkout. Resonance-JetLong is instead a planner-only composition record because it does not have a standalone source checkout. The picker also recognizes the separate `chynggi/gigatoken-llama.cpp` checkout as an informational experimental Windows x64/Linux x86_64 runtime fork. Discovery does not make it a Godzilla build or compile it. Use `mtq-godzilla-gigatoken` separately to create and qualify either pinned Godzilla v0.3.7 or the exact `09214b160` compatibility -profile; the command refuses arbitrary or existing target trees. Renamed +profile. Use `mtq-godzilla-compose` for the separate exact-`09214b160` +request-gated PFlash and complete-idle-slot KVFlash overlay. Both commands +refuse arbitrary or existing target trees. Renamed Godzilla trees are recognized by `scripts/godzilla-paths.ps1`. FlashAttention inspection checks the expected source markers and reports version and Git remote metadata when available. @@ -243,7 +247,9 @@ If the chosen script belongs to the other calibration mode, the planner now identifies the official/domvox mismatch and names the appropriate mode. The UI does not configure or compile a Godzilla CMake project automatically. Use Godzilla's documented process for ordinary checkouts, or the explicit -`mtq-godzilla-gigatoken` workflow for its pinned combined runtime. TriAttention remains experimental and +`mtq-godzilla-gigatoken` workflow for its pinned tokenizer runtime, or +`mtq-godzilla-compose` for the bounded PFlash/KVFlash profile. These workflows +remain CLI-only and do not patch a selected UI source folder. TriAttention remains experimental and model-specific. Existing `.triattention` output is reused only after its v1 header, dimensions, sampled indices, numeric arrays, and exact file length pass validation. diff --git a/multi_turboquant/integration/__init__.py b/multi_turboquant/integration/__init__.py index 1c8d990..1e01a2e 100644 --- a/multi_turboquant/integration/__init__.py +++ b/multi_turboquant/integration/__init__.py @@ -61,6 +61,17 @@ verify_godzilla_gigatoken, get_godzilla_source_profile, ) +from .godzilla_composition import ( + COMPOSITION_PROFILE, + GodzillaComposition, + GodzillaCompositionPlan, + build_godzilla_composition, + inspect_godzilla_composition, + plan_godzilla_composition, + prepare_godzilla_composition, + validate_godzilla_composition, + verify_godzilla_composition, +) __all__ = [ "patch_vllm", @@ -106,4 +117,13 @@ "prepare_godzilla_gigatoken", "verify_godzilla_gigatoken", "get_godzilla_source_profile", + "COMPOSITION_PROFILE", + "GodzillaComposition", + "GodzillaCompositionPlan", + "build_godzilla_composition", + "inspect_godzilla_composition", + "plan_godzilla_composition", + "prepare_godzilla_composition", + "validate_godzilla_composition", + "verify_godzilla_composition", ] diff --git a/multi_turboquant/integration/godzilla_composition.py b/multi_turboquant/integration/godzilla_composition.py new file mode 100644 index 0000000..467a450 --- /dev/null +++ b/multi_turboquant/integration/godzilla_composition.py @@ -0,0 +1,550 @@ +# SPDX-License-Identifier: MIT +"""Pinned PFlash/KVFlash composition overlay for the reviewed Godzilla tree. + +The overlay is intentionally conservative. PFlash is a request-gated, +deterministic prompt thinning policy. KVFlash is a server-level LRU residency +budget for complete idle slots; it does not claim the unfinished arbitrary-page +restore path from the research fork. Neither feature changes KVarN's cache +representation or TriAttention's intra-sequence eviction policy. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import platform +import shutil +import stat +import tempfile +from dataclasses import dataclass +from pathlib import Path +from typing import Sequence + +from .godzilla_gigatoken import ( + GODZILLA_COMPAT_COMMIT, + GODZILLA_URL, + RuntimeIssue, + _checkout_godzilla_source, + _git_revision, + _replace_once, + _resolve_cuda_compiler, + _run_checked, + get_godzilla_source_profile, +) + + +COMPOSITION_SCHEMA = 1 +COMPOSITION_PROFILE = "godzilla-09214b160-pflash-kvflash-v1" +MANIFEST_NAME = ".mtq-godzilla-composition.json" +_RUNTIME_FILES = ( + "common/common.h", + "common/arg.cpp", + "tools/server/server-task.h", + "tools/server/server-task.cpp", + "tools/server/server-context.cpp", +) +_EXPECTED_RUNTIME_SHA256 = { + "common/common.h": "1a7ab495602bd662a58d96afe90187c2d81ac5815c8fc818456226093d2b2695", + "common/arg.cpp": "9aa5ee48029fdece1e40dac129065bac2ae7d460f432dc6e9161ad36e0f72886", + "tools/server/server-task.h": "c2c65b8b65740eab74df7d16c6f0379a61c14184cab029984ea80db5f85bf6f2", + "tools/server/server-task.cpp": "334c6f8e5963bf400eeadba81a53caadf006b5096f09a963d26ccebd235f18fc", + "tools/server/server-context.cpp": "46009558c7a3a1c29b056cb1c287d61408abc429207fd9fe55bfb8a00a5a23ce", +} + + +@dataclass(frozen=True) +class GodzillaComposition: + pflash: bool = True + kvflash: bool = True + dflash: bool = True + ddtree: bool = True + triattention: bool = False + kvarn: bool = False + spec_la: bool = False + + +def validate_godzilla_composition(config: GodzillaComposition) -> tuple[RuntimeIssue, ...]: + """Return the fail-closed compatibility contract for the pinned overlay.""" + issues: list[RuntimeIssue] = [] + if config.triattention and config.kvarn: + issues.append(RuntimeIssue( + "error", + "triattention_kvarn_conflict", + "Godzilla 09214b160 rejects TriAttention with KVarN; KVarN-aware pruning is not implemented.", + )) + if config.spec_la: + issues.append(RuntimeIssue( + "error", + "specla_not_available", + "SpecLA is a specialized linear-attention runtime, not a stackable add-on for this Godzilla tree.", + )) + if config.pflash and config.triattention: + issues.append(RuntimeIssue( + "warning", + "two_stage_token_selection", + "PFlash changes the prompt before TriAttention scores retained KV entries; qualify quality for the chosen model and workload.", + )) + if config.kvflash: + issues.append(RuntimeIssue( + "info", + "kvflash_slot_tier", + "KVFlash is limited to LRU residency of complete idle slots; arbitrary KV pages, prefill skipping, and disk restore are not claimed.", + )) + return tuple(issues) + + +@dataclass(frozen=True) +class GodzillaCompositionPlan: + action: str + target: Path + build_dir: Path + backend: str + max_jobs: int + generator: str | None + cuda_compiler: Path | None + commands: tuple[tuple[str, ...], ...] + issues: tuple[RuntimeIssue, ...] + + @property + def ready(self) -> bool: + return not any(issue.severity == "error" for issue in self.issues) + + def to_dict(self) -> dict[str, object]: + return { + "action": self.action, + "target": str(self.target), + "build_dir": str(self.build_dir), + "backend": self.backend, + "max_jobs": self.max_jobs, + "generator": self.generator, + "cuda_compiler": str(self.cuda_compiler) if self.cuda_compiler else None, + "commands": [list(command) for command in self.commands], + "issues": [issue.to_dict() for issue in self.issues], + "ready": self.ready, + "profile": COMPOSITION_PROFILE, + "godzilla_commit": GODZILLA_COMPAT_COMMIT, + } + + +def _supported_platform() -> bool: + return platform.system().lower() in {"windows", "linux"} and platform.machine().lower() in { + "amd64", "x86_64" + } + + +def plan_godzilla_composition( + target: str | Path, + *, + action: str = "prepare", + backend: str = "cpu", + max_jobs: int = 2, + generator: str | None = None, + cuda_toolkit: str | Path | None = None, +) -> GodzillaCompositionPlan: + target_path = Path(target).expanduser().resolve() + normalized_action = action.strip().lower() + normalized_backend = backend.strip().lower() + build_dir = target_path / f"build-mtq-composition-{normalized_backend}" + cuda_compiler = _resolve_cuda_compiler(cuda_toolkit) if normalized_backend == "cuda" else None + issues = list(validate_godzilla_composition(GodzillaComposition())) + commands: list[tuple[str, ...]] = [] + + if normalized_action not in {"prepare", "build", "all", "verify"}: + issues.append(RuntimeIssue("error", "invalid_action", "Action must be prepare, build, all, or verify.")) + if normalized_backend not in {"cpu", "cuda"}: + issues.append(RuntimeIssue("error", "invalid_backend", "Backend must be cpu or cuda.")) + if max_jobs < 1: + issues.append(RuntimeIssue("error", "invalid_max_jobs", "max_jobs must be at least 1.")) + if not _supported_platform(): + issues.append(RuntimeIssue("error", "unsupported_platform", "Only Windows/Linux x86-64 is qualified.")) + if normalized_backend == "cuda" and cuda_compiler is None: + issues.append(RuntimeIssue("error", "cuda_compiler_missing", "CUDA builds require nvcc on PATH or --cuda-toolkit.")) + + if normalized_action in {"prepare", "all"}: + if target_path.exists(): + issues.append(RuntimeIssue("error", "target_exists", "Preparation requires a new destination.")) + if shutil.which("git") is None: + issues.append(RuntimeIssue("error", "git_missing", "git is required.")) + commands.extend(( + ("git", "init", str(target_path)), + ("git", "fetch", "--depth", "1", "origin", GODZILLA_COMPAT_COMMIT), + ("git", "checkout", "--detach", "FETCH_HEAD"), + )) + if normalized_action in {"build", "verify"}: + inspection = inspect_godzilla_composition(target_path) + if not inspection["valid"]: + issues.append(RuntimeIssue("error", "invalid_source", "; ".join(inspection["issues"]))) + if normalized_action in {"build", "all"}: + if shutil.which("cmake") is None: + issues.append(RuntimeIssue("error", "cmake_missing", "CMake is required.")) + configure = [ + "cmake", "-S", str(target_path), "-B", str(build_dir), + "-DLLAMA_BUILD_SERVER=ON", "-DLLAMA_BUILD_TESTS=OFF", "-DLLAMA_BUILD_UI=OFF", + "-DLLAMA_CURL=OFF", f"-DGGML_CUDA={'ON' if normalized_backend == 'cuda' else 'OFF'}", + "-DGGML_NATIVE=OFF", "-DGGML_CCACHE=OFF", + ] + if generator: + configure[5:5] = ["-G", generator] + if cuda_compiler is not None: + configure.append(f"-DCMAKE_CUDA_COMPILER={cuda_compiler}") + commands.extend((tuple(configure), ( + "cmake", "--build", str(build_dir), "--config", "Release", + "--target", "llama-server", "-j", str(max_jobs), + ))) + return GodzillaCompositionPlan( + normalized_action, target_path, build_dir, normalized_backend, max_jobs, generator, cuda_compiler, + tuple(commands), tuple(issues), + ) + + +def _apply_overlay(root: Path) -> None: + header_anchor = " int32_t cache_ram_mib = 8192; // -1 = no limit, 0 - disable, 1 = 1 MiB, etc.\n" + _replace_once(root / "common/common.h", header_anchor, header_anchor + """ + + // Multi-TurboQuant composition overlay (off by default) + bool pflash_enabled = false; // allow request-gated prompt thinning + float pflash_keep_ratio = 0.75f; // fraction retained when a request sets pflash=true + int32_t pflash_min_tokens = 4096; // do not thin shorter prompts + int32_t pflash_prefix_tokens = 256; // protected leading tokens + int32_t pflash_suffix_tokens = 128; // protected trailing tokens + int32_t kvflash_pages = 0; // complete-idle-slot residency budget in pages + int32_t kvflash_page_tokens = 256; // accounting granularity; not arbitrary-page restore +""") + + decl_anchor = "static void common_params_speculative_normalize(common_params & params);\n" + _replace_once(root / "common/arg.cpp", decl_anchor, decl_anchor + "static void common_params_composition_normalize(common_params & params);\n") + call_anchor = " common_params_speculative_normalize(params);\n" + _replace_once(root / "common/arg.cpp", call_anchor, call_anchor + " common_params_composition_normalize(params);\n") + + arg_anchor = """ add_opt(common_arg( + {"--cache-idle-slots"}, + {"--no-cache-idle-slots"}, + "save and clear idle slots on new task (default: enabled, requires unified KV and cache-ram)", + [](common_params & params, bool value) { + params.cache_idle_slots = value; + } + ).set_env("LLAMA_ARG_CACHE_IDLE_SLOTS").set_examples({LLAMA_EXAMPLE_SERVER})); +""" + arg_block = """ + add_opt(common_arg( + {"--pflash"}, + "enable request-gated PFlash prompt thinning (requests must also set pflash=true)", + [](common_params & params) { params.pflash_enabled = true; } + ).set_examples({LLAMA_EXAMPLE_SERVER})); + add_opt(common_arg( + {"--pflash-keep-ratio"}, "RATIO", + "fraction of prompt tokens retained by PFlash (default: 0.75)", + [](common_params & params, const std::string & value) { params.pflash_keep_ratio = std::stof(value); } + ).set_examples({LLAMA_EXAMPLE_SERVER})); + add_opt(common_arg( + {"--pflash-min-tokens"}, "N", + "minimum prompt length eligible for PFlash (default: 4096)", + [](common_params & params, int value) { params.pflash_min_tokens = value; } + ).set_examples({LLAMA_EXAMPLE_SERVER})); + add_opt(common_arg( + {"--kvflash-pages"}, "N", + "retain up to N pages of complete idle-slot KV state using LRU (0 disables)", + [](common_params & params, int value) { params.kvflash_pages = value; } + ).set_examples({LLAMA_EXAMPLE_SERVER})); + add_opt(common_arg( + {"--kvflash-page-tokens"}, "N", + "KVFlash accounting page size in tokens (default: 256)", + [](common_params & params, int value) { params.kvflash_page_tokens = value; } + ).set_examples({LLAMA_EXAMPLE_SERVER})); +""" + _replace_once(root / "common/arg.cpp", arg_anchor, arg_anchor + arg_block) + + normalize_anchor = "static void common_params_speculative_normalize(common_params & params) {\n" + normalize_block = """static void common_params_composition_normalize(common_params & params) { + if (params.pflash_keep_ratio <= 0.0f || params.pflash_keep_ratio > 1.0f) { + throw std::invalid_argument("--pflash-keep-ratio must be in (0, 1]"); + } + if (params.pflash_min_tokens < 1 || params.pflash_prefix_tokens < 0 || params.pflash_suffix_tokens < 0) { + throw std::invalid_argument("PFlash token limits must be non-negative and min-tokens must be positive"); + } + if (params.kvflash_pages < 0 || params.kvflash_page_tokens < 1) { + throw std::invalid_argument("--kvflash-pages must be >= 0 and --kvflash-page-tokens must be >= 1"); + } + if (params.kvflash_pages > 0 && params.cache_idle_slots) { + LOG_WRN("warning: KVFlash residency supersedes --cache-idle-slots; retaining bounded idle slots instead\\n"); + params.cache_idle_slots = false; + } +} + +""" + _replace_once(root / "common/arg.cpp", normalize_anchor, normalize_block + normalize_anchor) + + task_header_anchor = " bool return_progress = false;\n" + _replace_once(root / "tools/server/server-task.h", task_header_anchor, task_header_anchor + " bool pflash = false; // explicit per-request opt-in\n") + task_parse_anchor = " params.post_sampling_probs = json_value(data, \"post_sampling_probs\", defaults.post_sampling_probs);\n" + _replace_once(root / "tools/server/server-task.cpp", task_parse_anchor, task_parse_anchor + " params.pflash = json_value(data, \"pflash\", false);\n") + + process_anchor = " void process_single_task(server_task && task) {\n" + helpers = r''' bool maybe_apply_pflash(server_task & task) { + if (!params_base.pflash_enabled || !task.params.pflash || task.is_parent() || + (task.type != SERVER_TASK_TYPE_COMPLETION && task.type != SERVER_TASK_TYPE_INFILL) || + (task.params.res_type != TASK_RESPONSE_TYPE_NONE && task.params.res_type != TASK_RESPONSE_TYPE_OAI_CMPL) || + task.tokens.has_media() || task.n_tokens() < params_base.pflash_min_tokens) { + return false; + } + const size_t before = task.tokens.size(); + const size_t prefix = std::min(before, (size_t) params_base.pflash_prefix_tokens); + const size_t suffix = std::min(before - prefix, (size_t) params_base.pflash_suffix_tokens); + const size_t target = std::max(prefix + suffix, + (size_t) std::ceil(before * params_base.pflash_keep_ratio)); + if (target >= before) { + return false; + } + const size_t middle_source = before - prefix - suffix; + const size_t middle_keep = target - prefix - suffix; + llama_tokens selected; + selected.reserve(target); + for (size_t i = 0; i < prefix; ++i) { + selected.push_back(task.tokens[i]); + } + for (size_t i = 0; i < middle_keep; ++i) { + selected.push_back(task.tokens[prefix + (i * middle_source) / middle_keep]); + } + for (size_t i = before - suffix; i < before; ++i) { + selected.push_back(task.tokens[i]); + } + task.tokens = server_tokens(selected, false); + task.params.n_keep = std::min(task.params.n_keep, (int32_t) selected.size()); + SRV_INF("PFlash request %d: retained %zu/%zu tokens (explicit opt-in)\n", + task.id, selected.size(), before); + return true; + } + + void enforce_kvflash_residency() { + if (params_base.kvflash_pages <= 0) { + return; + } + const int64_t budget = (int64_t) params_base.kvflash_pages * params_base.kvflash_page_tokens; + int64_t resident = 0; + for (const auto & slot : slots) { + if (!slot.is_processing()) { + resident += slot.prompt.n_tokens(); + } + } + while (resident > budget) { + server_slot * victim = nullptr; + for (auto & slot : slots) { + if (!slot.is_processing() && slot.prompt.n_tokens() > 0 && + (!victim || slot.t_last_used < victim->t_last_used)) { + victim = &slot; + } + } + if (!victim) { + break; + } + const int64_t removed = victim->prompt.n_tokens(); + slot_clear_hybrid_safe(*victim, "KVFlash LRU residency eviction"); + resident -= removed; + SRV_INF("KVFlash LRU eviction: slot=%d removed=%" PRId64 " resident=%" PRId64 "/%" PRId64 " tokens\n", + victim->id, removed, resident, budget); + } + } + +''' + _replace_once(root / "tools/server/server-context.cpp", process_anchor, helpers + process_anchor) + + tokenize_anchor = """ if (task.cli) { + if (!tokenize_cli_input(task)) { + break; + } + } + + const int id_slot = task.id_slot; +""" + _replace_once(root / "tools/server/server-context.cpp", tokenize_anchor, tokenize_anchor.replace( + "\n const int id_slot", "\n maybe_apply_pflash(task);\n\n const int id_slot" + )) + + idle_anchor = """ if (params_base.cache_idle_slots) { + for (auto & s : slots) { + if (!s.is_processing()) { + if (params_base.cache_ram_mib != 0 && prompt_cache) { + slot_save_and_clear(s); + } else { + slot_clear_hybrid_safe(s, "before new task"); + } + } + } + } +""" + idle_new = """ if (params_base.kvflash_pages > 0) { + enforce_kvflash_residency(); + } else if (params_base.cache_idle_slots) { + for (auto & s : slots) { + if (!s.is_processing()) { + if (params_base.cache_ram_mib != 0 && prompt_cache) { + slot_save_and_clear(s); + } else { + slot_clear_hybrid_safe(s, "before new task"); + } + } + } + } +""" + _replace_once(root / "tools/server/server-context.cpp", idle_anchor, idle_new) + + props_anchor = " { \"cors_proxy_enabled\", params.ui_mcp_proxy || params.webui_mcp_proxy },\n" + props_block = """ { "mtq_composition", json { + {"profile", "godzilla-09214b160-pflash-kvflash-v1"}, + {"pflash", json { + {"enabled", params.pflash_enabled}, + {"request_gated", true}, + {"telemetry", "server_log"}, + }}, + {"kvflash", json { + {"tier", "complete_idle_slots"}, + {"pages", params.kvflash_pages}, + {"page_tokens", params.kvflash_page_tokens}, + {"telemetry", "server_log"}, + {"arbitrary_page_restore", false}, + {"prefill_skip", false}, + }}, + } }, +""" + _replace_once(root / "tools/server/server-context.cpp", props_anchor, props_anchor + props_block) + + +def _file_hashes(root: Path) -> dict[str, str]: + return {name: hashlib.sha256((root / name).read_bytes()).hexdigest() for name in _RUNTIME_FILES} + + +def _git_source_state(root: Path) -> tuple[set[str], set[str]] | None: + try: + changed_result = _run_checked( + ("git", "diff", "--name-only", "HEAD", "--"), cwd=root, capture=True, + ) + untracked_result = _run_checked( + ("git", "ls-files", "--others", "--exclude-standard"), cwd=root, capture=True, + ) + except (OSError, RuntimeError): + return None + changed = set(changed_result.stdout.decode("utf-8", errors="replace").splitlines()) + untracked = set(untracked_result.stdout.decode("utf-8", errors="replace").splitlines()) + return changed, untracked + + +def _remove_tree(path: Path) -> None: + def retry(function, value, _error): + os.chmod(value, stat.S_IWRITE | stat.S_IREAD) + function(value) + shutil.rmtree(path, onerror=retry) + + +def prepare_godzilla_composition(plan: GodzillaCompositionPlan, *, confirmed: bool = False) -> dict[str, object]: + if plan.action not in {"prepare", "all"} or not plan.ready: + raise RuntimeError("A ready prepare/all plan is required") + if not confirmed: + raise RuntimeError("Preparation was not confirmed") + plan.target.parent.mkdir(parents=True, exist_ok=True) + work = Path(tempfile.mkdtemp(prefix=f".{plan.target.name}.mtq-", dir=plan.target.parent)).resolve() + try: + _checkout_godzilla_source(work, get_godzilla_source_profile("09214b160")) + if _git_revision(work) != GODZILLA_COMPAT_COMMIT: + raise RuntimeError("Pinned Godzilla revision did not match after checkout") + _apply_overlay(work) + runtime_hashes = _file_hashes(work) + if runtime_hashes != _EXPECTED_RUNTIME_SHA256: + raise RuntimeError("Generated overlay hashes do not match the reviewed exact-commit profile") + manifest = { + "schema": COMPOSITION_SCHEMA, + "profile": COMPOSITION_PROFILE, + "godzilla": {"url": GODZILLA_URL, "commit": GODZILLA_COMPAT_COMMIT}, + "capabilities": { + "pflash": "request-gated deterministic prompt thinning", + "kvflash": "complete idle-slot LRU residency", + "arbitrary_kv_page_restore": False, + "prefill_skip": False, + }, + "runtime_files_sha256": runtime_hashes, + } + (work / MANIFEST_NAME).write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8", newline="\n") + os.replace(work, plan.target) + except BaseException: + if work.exists(): + _remove_tree(work) + raise + return inspect_godzilla_composition(plan.target) + + +def inspect_godzilla_composition(target: str | Path) -> dict[str, object]: + root = Path(target).expanduser().resolve() + issues: list[str] = [] + manifest_path = root / MANIFEST_NAME + if not manifest_path.is_file(): + return {"valid": False, "target": str(root), "issues": [f"missing {MANIFEST_NAME}"]} + try: + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + except (OSError, ValueError) as exc: + return {"valid": False, "target": str(root), "issues": [f"invalid manifest: {exc}"]} + if manifest.get("schema") != COMPOSITION_SCHEMA or manifest.get("profile") != COMPOSITION_PROFILE: + issues.append("manifest schema/profile mismatch") + if _git_revision(root) != GODZILLA_COMPAT_COMMIT: + issues.append("Godzilla revision mismatch") + expected = manifest.get("runtime_files_sha256", {}) + if not isinstance(expected, dict): + issues.append("manifest runtime hash table is invalid") + expected = {} + try: + actual = _file_hashes(root) + except OSError as exc: + issues.append(str(exc)) + actual = {} + for name in _RUNTIME_FILES: + if expected.get(name) != actual.get(name): + issues.append(f"runtime file changed: {name}") + if _EXPECTED_RUNTIME_SHA256.get(name) != actual.get(name): + issues.append(f"runtime file does not match reviewed overlay: {name}") + source_state = _git_source_state(root) + if source_state is None: + issues.append("unable to inspect the prepared Git source state") + else: + changed, untracked = source_state + if changed != set(_RUNTIME_FILES): + issues.append("tracked source changes do not match the reviewed overlay allowlist") + unexpected = untracked - {MANIFEST_NAME} + if unexpected: + issues.append("unexpected untracked source files: " + ", ".join(sorted(unexpected))) + return {"valid": not issues, "target": str(root), "profile": COMPOSITION_PROFILE, "issues": issues} + + +def build_godzilla_composition(plan: GodzillaCompositionPlan, *, confirmed: bool = False) -> dict[str, object]: + if plan.action not in {"build", "all"} or not plan.ready: + raise RuntimeError("A ready build/all plan is required") + if not confirmed: + raise RuntimeError("Build was not confirmed") + for command in plan.commands[-2:]: + _run_checked(command) + return verify_godzilla_composition(plan) + + +def _find_server(build_dir: Path) -> Path | None: + candidates = sorted(build_dir.glob("**/llama-server.exe")) + sorted(build_dir.glob("**/llama-server")) + return next((path for path in candidates if path.is_file()), None) + + +def verify_godzilla_composition(plan: GodzillaCompositionPlan) -> dict[str, object]: + inspection = inspect_godzilla_composition(plan.target) + server = _find_server(plan.build_dir) + issues = list(inspection["issues"]) + help_text = "" + if server is None: + issues.append(f"llama-server was not found under {plan.build_dir}") + else: + result = _run_checked((str(server), "--help"), capture=True) + help_text = ((result.stdout or b"") + (result.stderr or b"")).decode("utf-8", errors="replace") + for flag in ("--pflash", "--kvflash-pages", "--spec-branch-budget", "--triattention-stats"): + if flag not in help_text: + issues.append(f"built server help is missing {flag}") + return { + "valid": not issues, + "target": str(plan.target), + "build_dir": str(plan.build_dir), + "server": str(server) if server else None, + "issues": issues, + } diff --git a/multi_turboquant/integration/godzilla_composition_cli.py b/multi_turboquant/integration/godzilla_composition_cli.py new file mode 100644 index 0000000..ef380c2 --- /dev/null +++ b/multi_turboquant/integration/godzilla_composition_cli.py @@ -0,0 +1,95 @@ +# SPDX-License-Identifier: MIT +"""CLI for the pinned Godzilla composition overlay.""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path + +from .godzilla_composition import ( + build_godzilla_composition, + plan_godzilla_composition, + prepare_godzilla_composition, + verify_godzilla_composition, +) + + +def _add_common(parser: argparse.ArgumentParser) -> None: + parser.add_argument("target", type=Path, help="New or prepared composition source directory") + parser.add_argument("--backend", choices=("cpu", "cuda"), default="cpu") + parser.add_argument("--max-jobs", type=int, default=2) + parser.add_argument("--generator", help="Optional CMake generator") + parser.add_argument("--cuda-toolkit", type=Path, help="CUDA toolkit root or nvcc path") + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + description="Prepare and qualify the pinned Godzilla PFlash/KVFlash composition overlay" + ) + children = parser.add_subparsers(dest="action", required=True) + for action, help_text in ( + ("plan", "Inspect the workflow without changing files"), + ("prepare", "Create a new pinned overlay source tree"), + ("build", "Build the prepared llama-server"), + ("all", "Prepare, build, and verify consecutively"), + ("verify", "Verify an existing prepared build"), + ): + child = children.add_parser(action, help=help_text) + _add_common(child) + child.add_argument("--json", action="store_true") + if action == "plan": + child.add_argument( + "--for-action", choices=("prepare", "build", "verify"), default="prepare" + ) + if action in {"prepare", "build", "all"}: + child.add_argument("--yes", action="store_true", help="Confirm downloads/builds") + return parser + + +def main(argv: list[str] | None = None) -> int: + args = build_parser().parse_args(argv) + action = args.for_action if args.action == "plan" else args.action + plan = plan_godzilla_composition( + args.target, + action=action, + backend=args.backend, + max_jobs=args.max_jobs, + generator=args.generator, + cuda_toolkit=args.cuda_toolkit, + ) + if args.action == "plan": + print(json.dumps(plan.to_dict(), indent=2)) + print("No files, downloads, or builds were changed.") + return 0 if plan.ready else 2 + if not plan.ready: + print(json.dumps(plan.to_dict(), indent=2)) + return 2 + try: + if args.action == "prepare": + result = prepare_godzilla_composition(plan, confirmed=args.yes) + elif args.action == "build": + result = build_godzilla_composition(plan, confirmed=args.yes) + elif args.action == "verify": + result = verify_godzilla_composition(plan) + else: + prepare_godzilla_composition(plan, confirmed=args.yes) + build_plan = plan_godzilla_composition( + args.target, + action="build", + backend=args.backend, + max_jobs=args.max_jobs, + generator=args.generator, + cuda_toolkit=args.cuda_toolkit, + ) + result = build_godzilla_composition(build_plan, confirmed=args.yes) + except (OSError, RuntimeError, ValueError) as exc: + print(f"ERROR: {exc}", file=sys.stderr) + return 2 + print(json.dumps(result, indent=2)) + return 0 if result.get("valid", False) else 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/multi_turboquant/integration/llamacpp_scan.py b/multi_turboquant/integration/llamacpp_scan.py index 79ccb5f..a16fc14 100644 --- a/multi_turboquant/integration/llamacpp_scan.py +++ b/multi_turboquant/integration/llamacpp_scan.py @@ -108,12 +108,20 @@ def supports_pflash(self) -> bool: def supports_kvflash(self) -> bool: return any( self.supports_flag(flag) - for flag in ("--kvflash", "--kvflash-policy", "--kvflash-drafter") + for flag in ( + "--kvflash", + "--kvflash-pages", + "--kvflash-policy", + "--kvflash-drafter", + ) ) @property def supports_ddtree(self) -> bool: - return self.supports_flag("--ddtree") or self.supports_flag("--ddtree-budget") + return any( + self.supports_flag(flag) + for flag in ("--ddtree", "--ddtree-budget", "--spec-branch-budget") + ) @property def supports_specla(self) -> bool: @@ -121,12 +129,15 @@ def supports_specla(self) -> bool: @property def runtime_family(self) -> str: + # KVarN and TriAttention are the strongest Godzilla markers. Check them + # before add-on flags so a reviewed Godzilla build does not change + # family merely because DDTree, PFlash, or KVFlash is also present. + if self.supports_kvarn or self.supports_triattention: + return "godzilla" if self.supports_flag("--pflash-mode") or self.supports_flag("--kvflash-policy"): return "pflash_llamacpp" if self.supports_ddtree or self.supports_flag("--prefill-compression"): return "lucebox" - if self.supports_kvarn or self.supports_triattention: - return "godzilla" return "llamacpp" @property diff --git a/multi_turboquant/optimizations/catalog.py b/multi_turboquant/optimizations/catalog.py index eae8c5e..ae82a98 100644 --- a/multi_turboquant/optimizations/catalog.py +++ b/multi_turboquant/optimizations/catalog.py @@ -538,6 +538,42 @@ "PFlash can omit prompt facts and must remain disabled for exact extraction and code tasks.", ), ), + OptimizationDescriptor( + id="godzilla_composition", + name="Pinned Godzilla PFlash/KVFlash composition", + source_url="https://github.com/aivrar/multi-turboquant", + kind=OptimizationKind.PREFILL, + maturity=OptimizationMaturity.EXPERIMENTAL, + integration_mode=IntegrationMode.NATIVE_BACKEND_REQUIRED, + license="MIT", + summary=( + "Exact-commit Godzilla overlay with request-gated PFlash and bounded " + "whole-idle-slot KVFlash residency." + ), + supported_engines=("godzilla",), + supported_compute=("cpu", "cuda"), + supported_os=("windows", "linux"), + supported_architectures=("x86_64", "amd64"), + composition_domains=("prefill_policy", "server_kv_residency"), + required_executables=("git", "cmake"), + required_artifacts=( + "A new source tree prepared from Godzilla commit 09214b160b402011359f0ef9d5fa8f8be1112e85", + "A workload-specific quality corpus before enabling PFlash", + ), + validation_gates=( + "Hash-bounded source inspection and exact-commit llama-server build", + "PFlash off/on quality, exact-task bypass, TTFT, and prompt-cache regression", + "KVFlash resident-budget, LRU eviction, concurrency, and long-context soak tests", + ), + quality_risk=QualityRisk.LOSSY, + reviewed_source_commit="09214b160b402011359f0ef9d5fa8f8be1112e85", + limitations=( + "PFlash is off by default, must be enabled at startup and per request, and skips chat, multimodal, embedding, rerank, and parallel-parent tasks.", + "KVFlash manages complete idle slots only; arbitrary KV-page restore, disk restore, and prefill skipping are not implemented or claimed.", + "TriAttention and KVarN remain mutually exclusive in the pinned upstream revision.", + "SpecLA is not included because it is a specialized linear-attention runtime rather than a stackable Godzilla add-on.", + ), + ), OptimizationDescriptor( id="resonance_jetlong", name="Resonance JetLongFreq research profile", diff --git a/multi_turboquant/ui/discovery.py b/multi_turboquant/ui/discovery.py index 8db0140..e75b8a1 100644 --- a/multi_turboquant/ui/discovery.py +++ b/multi_turboquant/ui/discovery.py @@ -664,6 +664,8 @@ def _classify_addon(path: Path, files: set[str], directories: set[str]) -> str | if "sageattention" in directories and "setup.py" in files and "csrc" in directories: return "sageattention" llama_markers = "CMakeLists.txt" in files and "ggml" in directories + if llama_markers and ".mtq-godzilla-composition.json" in files: + return "godzilla_composition" if ( llama_markers and (path / "tools" / "server" / "qwen36-smart-router.py").is_file() @@ -782,6 +784,11 @@ def scan_addon_roots( from ..integration import inspect_godzilla_checkout item["source"] = inspect_godzilla_checkout(directory) + if kind == "godzilla_composition": + from ..integration import inspect_godzilla_composition + + item["source"] = inspect_godzilla_composition(directory) + item["source_profile"] = kind if kind == "triattention": from ..calibration import inspect_official_triattention_checkout diff --git a/setup.py b/setup.py index b34add7..1929172 100644 --- a/setup.py +++ b/setup.py @@ -54,6 +54,7 @@ "mtq-triattention-stats=multi_turboquant.calibration.generate_stats:main", "mtq-godzilla-triattention=multi_turboquant.calibration.godzilla_triattention:main", "mtq-godzilla-gigatoken=multi_turboquant.integration.godzilla_gigatoken_cli:main", + "mtq-godzilla-compose=multi_turboquant.integration.godzilla_composition_cli:main", "mtq-weight-share=multi_turboquant.integration.weight_share_cli:main", "mtq-benchmark=multi_turboquant.benchmark.run_benchmark:main", "mtq-optimizations=multi_turboquant.optimizations.cli:main", diff --git a/tests/test_godzilla_composition.py b/tests/test_godzilla_composition.py new file mode 100644 index 0000000..e53372b --- /dev/null +++ b/tests/test_godzilla_composition.py @@ -0,0 +1,97 @@ +# SPDX-License-Identifier: MIT +from __future__ import annotations + +import json +import subprocess + +import pytest + +from multi_turboquant.integration import godzilla_composition as runtime +from multi_turboquant.integration.godzilla_composition_cli import build_parser + + +def _tools(monkeypatch, names=frozenset({"git", "cmake"})): + monkeypatch.setattr(runtime, "_supported_platform", lambda: True) + monkeypatch.setattr(runtime.shutil, "which", lambda name: f"/tools/{name}" if name in names else None) + + +def test_default_plan_is_exact_read_only_and_reports_bounded_kvflash(tmp_path, monkeypatch): + _tools(monkeypatch) + target = tmp_path / "composed" + plan = runtime.plan_godzilla_composition(target) + + assert plan.ready + assert not target.exists() + assert plan.to_dict()["godzilla_commit"] == runtime.GODZILLA_COMPAT_COMMIT + assert any(issue.code == "kvflash_slot_tier" for issue in plan.issues) + + +def test_composition_rejects_known_conflicts(): + issues = runtime.validate_godzilla_composition(runtime.GodzillaComposition( + triattention=True, kvarn=True, spec_la=True, + )) + assert {issue.code for issue in issues if issue.severity == "error"} == { + "triattention_kvarn_conflict", "specla_not_available", + } + + +def test_prepare_requires_new_target_and_confirmation(tmp_path, monkeypatch): + _tools(monkeypatch) + target = tmp_path / "composed" + plan = runtime.plan_godzilla_composition(target) + with pytest.raises(RuntimeError, match="not confirmed"): + runtime.prepare_godzilla_composition(plan) + target.mkdir() + assert not runtime.plan_godzilla_composition(target).ready + + +def test_inspection_is_hash_bounded(tmp_path, monkeypatch): + for name in runtime._RUNTIME_FILES: + path = tmp_path / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(name, encoding="utf-8") + manifest = { + "schema": runtime.COMPOSITION_SCHEMA, + "profile": runtime.COMPOSITION_PROFILE, + "runtime_files_sha256": runtime._file_hashes(tmp_path), + } + (tmp_path / runtime.MANIFEST_NAME).write_text(json.dumps(manifest), encoding="utf-8") + monkeypatch.setattr(runtime, "_git_revision", lambda _root: runtime.GODZILLA_COMPAT_COMMIT) + monkeypatch.setattr(runtime, "_EXPECTED_RUNTIME_SHA256", manifest["runtime_files_sha256"]) + monkeypatch.setattr( + runtime, + "_git_source_state", + lambda _root: (set(runtime._RUNTIME_FILES), {runtime.MANIFEST_NAME}), + ) + assert runtime.inspect_godzilla_composition(tmp_path)["valid"] + (tmp_path / "common/common.h").write_text("changed", encoding="utf-8") + assert not runtime.inspect_godzilla_composition(tmp_path)["valid"] + + +def test_verify_requires_composed_and_godzilla_flags(tmp_path, monkeypatch): + build = tmp_path / "build-mtq-composition-cpu" / "bin" + build.mkdir(parents=True) + (build / "llama-server.exe").write_bytes(b"") + monkeypatch.setattr(runtime, "inspect_godzilla_composition", lambda _root: { + "valid": True, "issues": [], + }) + monkeypatch.setattr(runtime, "_run_checked", lambda *_args, **_kwargs: subprocess.CompletedProcess( + [], 0, stdout=b"--pflash --kvflash-pages --spec-branch-budget --triattention-stats", stderr=b"" + )) + plan = runtime.GodzillaCompositionPlan( + "verify", tmp_path, tmp_path / "build-mtq-composition-cpu", "cpu", 2, None, None, (), (), + ) + assert runtime.verify_godzilla_composition(plan)["valid"] + + +def test_cli_plan_accepts_build_target(): + args = build_parser().parse_args(["plan", "combined", "--for-action", "build"]) + assert args.action == "plan" + assert args.for_action == "build" + + +def test_cuda_plan_requires_an_explicit_or_discovered_compiler(tmp_path, monkeypatch): + _tools(monkeypatch) + plan = runtime.plan_godzilla_composition(tmp_path / "composed", backend="cuda") + assert not plan.ready + assert any(issue.code == "cuda_compiler_missing" for issue in plan.issues) diff --git a/tests/test_llamacpp_scan.py b/tests/test_llamacpp_scan.py index dafdcd0..cc904a7 100644 --- a/tests/test_llamacpp_scan.py +++ b/tests/test_llamacpp_scan.py @@ -14,6 +14,7 @@ def test_parse_llamacpp_help_detects_context_and_extension_features(): --yarn-orig-ctx N --triattention-stats FILE --spec-type TYPE supported: dflash draft-mtp + --spec-branch-budget N DDTree branch nodes beyond the main draft path --spec-dflash-cross-ctx N """ @@ -31,13 +32,16 @@ def test_parse_llamacpp_help_detects_context_and_extension_features(): assert capabilities.supports_kvarn is True assert capabilities.supports_speculative is True assert capabilities.supports_dflash is True + assert capabilities.supports_ddtree is True assert capabilities.supports_cache_type("kvarn4") is True assert capabilities.supports_cache_type("turbo3_tcq") is True assert capabilities.supports_speculative_type("draft-mtp") is True + assert capabilities.runtime_family == "godzilla" data = capabilities.to_dict() assert data["binary"] == "llama-server-godzilla" assert data["supports_yarn"] is True + assert data["supports_ddtree"] is True assert "kvarn4" in data["cache_types"] @@ -54,6 +58,7 @@ def test_parse_llamacpp_help_handles_plain_upstream_binary(): assert capabilities.supports_yarn is False assert capabilities.supports_kvarn is False assert capabilities.supports_triattention is False + assert capabilities.supports_ddtree is False assert capabilities.supports_cache_type("f16") is True assert capabilities.supports_cache_type("kvarn4") is False @@ -115,6 +120,38 @@ def test_parse_llamacpp_help_detects_lucebox_runtime_capabilities(): assert capabilities.supports_specla is True +def test_godzilla_family_wins_when_reviewed_addon_flags_are_present(): + capabilities = parse_llamacpp_help( + """ +version: godzilla-composition-test + --cache-type-k TYPE supported: f16 kvarn2..kvarn8 + --triattention-stats FILE + --spec-type TYPE supported: dflash + --spec-branch-budget N + --pflash-mode MODE + --kvflash-policy POLICY +""" + ) + + assert capabilities.runtime_family == "godzilla" + assert capabilities.supports_ddtree is True + assert capabilities.supports_pflash is True + assert capabilities.supports_kvflash is True + + +def test_composition_overlay_kvflash_pages_is_detected(): + capabilities = parse_llamacpp_help(""" + --triattention-stats PATH + --pflash + --pflash-keep-ratio RATIO + --kvflash-pages N + """) + + assert capabilities.runtime_family == "godzilla" + assert capabilities.supports_pflash is True + assert capabilities.supports_kvflash is True + + def test_scan_llamacpp_binary_reports_missing_binary(): capabilities = scan_llamacpp_binary( "__definitely_missing_llama_server__", diff --git a/tests/test_optimizations.py b/tests/test_optimizations.py index ebba1ad..112b44f 100644 --- a/tests/test_optimizations.py +++ b/tests/test_optimizations.py @@ -50,6 +50,7 @@ def test_builtin_catalog_is_explicit_and_disabled_by_default(): "duoattention", "icecache", "pflash_llamacpp", + "godzilla_composition", "resonance_jetlong", } <= set(ids) assert all(not plugin.descriptor.default_enabled for plugin in plugins) diff --git a/tests/test_ui_workspace.py b/tests/test_ui_workspace.py index 400d069..057b2bd 100644 --- a/tests/test_ui_workspace.py +++ b/tests/test_ui_workspace.py @@ -250,6 +250,27 @@ def test_addon_scan_recognizes_blocked_source_as_informational(tmp_path: Path): assert "environment_profile" not in addon +def test_addon_scan_recognizes_prepared_godzilla_composition(tmp_path: Path, monkeypatch): + import multi_turboquant.integration as integration + + source = tmp_path / "renamed-runtime" + (source / "ggml").mkdir(parents=True) + (source / "CMakeLists.txt").write_text("project(godzilla)", encoding="utf-8") + (source / ".mtq-godzilla-composition.json").write_text("{}", encoding="utf-8") + monkeypatch.setattr( + integration, + "inspect_godzilla_composition", + lambda path: {"valid": True, "target": str(Path(path).resolve()), "issues": []}, + ) + + result = scan_addon_roots([tmp_path]) + addon = next(item for item in result["addons"] if item["path"] == str(source.resolve())) + + assert addon["kind"] == "godzilla_composition" + assert addon["source_profile"] == "godzilla_composition" + assert addon["source"]["valid"] is True + + def test_addon_scan_does_not_classify_an_empty_blocked_named_folder(tmp_path: Path): (tmp_path / "rocketkv").mkdir()