diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a9d992c..4a9e5c9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -23,6 +23,7 @@ jobs: - run: uv sync --extra dev - run: uv run ruff check src tests - run: uv run pytest -q + - run: uv run python scripts/build_package_manifest.py - run: uv run python scripts/validate_package.py - run: uv run python scripts/verify_results.py diff --git a/.github/workflows/package-manifest.yml b/.github/workflows/package-manifest.yml new file mode 100644 index 0000000..1b7c8f1 --- /dev/null +++ b/.github/workflows/package-manifest.yml @@ -0,0 +1,49 @@ +name: Refresh package manifest + +on: + push: + paths-ignore: + - PACKAGE_MANIFEST.json + +permissions: + contents: write + +concurrency: + group: package-manifest-${{ github.ref }} + cancel-in-progress: true + +jobs: + refresh: + if: github.actor != 'github-actions[bot]' + runs-on: ubuntu-latest + timeout-minutes: 10 + + steps: + - uses: actions/checkout@v7.0.1 + with: + ref: ${{ github.ref_name }} + fetch-depth: 0 + + - uses: actions/setup-python@v7 + with: + python-version: "3.12" + + - name: Regenerate manifest from the checked-out tree + run: python scripts/build_package_manifest.py + + - name: Validate regenerated package + run: python scripts/validate_package.py + + - name: Commit generated manifest when needed + shell: bash + run: | + set -euo pipefail + if git diff --quiet -- PACKAGE_MANIFEST.json; then + echo "PACKAGE_MANIFEST.json is current." + exit 0 + fi + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + git add PACKAGE_MANIFEST.json + git commit -m "chore: refresh package manifest" + git push origin "HEAD:${GITHUB_REF_NAME}" diff --git a/LIMITATIONS.md b/LIMITATIONS.md new file mode 100644 index 0000000..813477b --- /dev/null +++ b/LIMITATIONS.md @@ -0,0 +1,53 @@ +# Limitations and Claim Boundaries + +## Hardware scope + +Current retained GPU evidence is specific to a single NVIDIA GeForce RTX 4080 SUPER environment. It must not be extrapolated to datacenter accelerators, multi-GPU systems, AMD GPUs, Apple Silicon, or another driver/runtime fingerprint. + +## Incomplete cross-runtime evidence + +Capability and integration smoke runs show that the evidence path can operate. They do not establish which runtime is best for the primary 3B model. A deployment recommendation requires a complete matched matrix, quality evidence, confirmatory runs, and holdout replay. + +## Workload scope + +Exact-token synthetic workloads are useful for controlled measurement but do not automatically reproduce a real application's prompt distribution, session behavior, cache reuse, concurrency, or network overhead. A policy is valid only for the workload contract it records. + +## Quality scope + +A latency measurement without its required correctness and quality artifacts is not policy eligible. Small smoke-quality datasets cannot support broad language-model quality claims. + +## Timing scope + +Cold start, compilation, graph capture, warmup, steady state, and HTTP end-to-end timing are distinct quantities. Engine-only and HTTP measurements must not be ranked as if they were interchangeable. + +## Stability and thermal scope + +A run that does not meet repetition, drift, active-clock, and thermal requirements may remain useful for debugging, but it cannot support a formal performance headline. + +## Profiler and causal scope + +Profiler observations generate hypotheses. Kernel occupancy, launch count, bandwidth, or graph-break observations alone do not prove the cause of an end-to-end latency change. Promotion requires a matched intervention, mediator movement, negative control, and confirmatory timing. + +## Custom-kernel scope + +Reference parity for a Triton kernel establishes correctness only for the validated kernel version, GPU, dtype, and shape. It does not establish end-to-end model acceleration. An unvalidated key always falls back to the reference implementation. + +## Policy scope + +Policies are deterministic, fingerprinted decision lists. They may be incomplete when evidence coverage is sparse and become `STALE` after relevant environment or workload drift. EdgeFlow does not promise a universal fastest configuration. + +## Learned components + +Learned policy or copilot components remain deferred until the repository contains enough unique validated points and grounded labels. The core benchmark, validation, and policy path must remain functional without an agent or learned selector. + +## Control-plane scope + +The web application is a localhost-only control surface for one managed runtime and one GPU job at a time. It is not a remote multi-user orchestration platform, cloud control plane, or arbitrary shell interface. + +## External validation + +A second-machine clean reproduction and domain-user usability study remain outstanding. Local success on the development machine is not independent reproduction. + +## Public-claim rule + +Only measured runs with the required `PASS` verdict, matched scope, quality evidence, provenance, and confirmatory/holdout support may enter a public deployment conclusion. `demo`, `estimated`, profiled, `CONDITIONAL_PASS`, incomplete, or stale evidence must remain labelled as such. diff --git a/PACKAGE_MANIFEST.json b/PACKAGE_MANIFEST.json index 0e19cd9..0865b53 100644 --- a/PACKAGE_MANIFEST.json +++ b/PACKAGE_MANIFEST.json @@ -1,9 +1,9 @@ { "schema_version": "1.0", "package": "EdgeFlow", - "generated_at": "2026-08-04T21:18:18.269650+00:00", - "file_count": 186, - "total_bytes": 1418372, + "generated_at": "2026-08-11T01:05:38.387262+00:00", + "file_count": 190, + "total_bytes": 1432806, "files": [ { "path": ".github/ISSUE_TEMPLATE/bug_report.yml", @@ -22,8 +22,13 @@ }, { "path": ".github/workflows/ci.yml", - "bytes": 1171, - "sha256": "bc1f4694a170bbcc069757237e0e618fef33450adca77448c0cb039ec2cfdfef" + "bytes": 1232, + "sha256": "d573d01a4936a221958b2ccebaa2bcd9c94b5c3796471e2a0bd95d1fc6ddc675" + }, + { + "path": ".github/workflows/package-manifest.yml", + "bytes": 1297, + "sha256": "ed0135ff441a7c9ed14b0b50fd077d94965ff049beb209fc0c064c7aa44cc23b" }, { "path": ".gitignore", @@ -50,6 +55,11 @@ "bytes": 707, "sha256": "2312413594bfe65a55eb8a48ea378e413d8f973a9bb3a5ef247053292942fedc" }, + { + "path": "LIMITATIONS.md", + "bytes": 3354, + "sha256": "93fe9dfcae69699d2817170150084ff836bb9ec6db669ddb13d93e726679955d" + }, { "path": "MODEL_LICENSES.md", "bytes": 853, @@ -65,6 +75,11 @@ "bytes": 693, "sha256": "d0126b38ea76f4d38ff0a0dce377c9a0e4b0fbfa2ae07aa9f43881f3a79f5c56" }, + { + "path": "PORTFOLIO.md", + "bytes": 6456, + "sha256": "bd27517b9fc034ce9ad5292bb3aefeb025e9c2a005c221de8bbc34d90e53ef21" + }, { "path": "README.md", "bytes": 14894, @@ -75,6 +90,11 @@ "bytes": 13938, "sha256": "0e9fe4a07b99070bfae081e9ee5616e1a47b9bdc7e12e9ed329dfaab0d72c636" }, + { + "path": "REVIEWER_GUIDE.md", + "bytes": 3266, + "sha256": "d8a57bd456cfe6087488bda615ed55db87a3296f46124d5a76897c674aa07375" + }, { "path": "SECURITY.md", "bytes": 1431, diff --git a/PORTFOLIO.md b/PORTFOLIO.md new file mode 100644 index 0000000..e5798ba --- /dev/null +++ b/PORTFOLIO.md @@ -0,0 +1,136 @@ +# EdgeFlow Engineering Portfolio + +## Problem + +Local LLM deployment decisions are often made from summary numbers that mix incompatible workloads, warmup and steady-state latency, quality loss, thermal drift, profiler overhead, and changing software fingerprints. + +EdgeFlow treats a deployment recommendation as the output of a validated evidence pipeline rather than a benchmark leaderboard: + +```text +Observe → Diagnose → Intervene → Verify → Synthesize policy +``` + +## Engineering contribution + +EdgeFlow combines: + +- exact hardware, runtime, model, workload, and source fingerprints; +- PyTorch eager / `torch.compile`, llama.cpp, and vLLM runtime adapters; +- isolated per-run artifact directories with per-request raw JSONL; +- correctness, timing, stability, statistics, quality, and provenance gates; +- deterministic bottleneck hypotheses and controlled-intervention plans; +- correctness-cached Triton optimization with mandatory reference fallback; +- a localhost-only control plane, typed single-GPU worker, and evidence UI; +- policy synthesis that refuses to rank ineligible or stale evidence. + +The full architecture is documented in [`docs/01_SYSTEM_ARCHITECTURE.md`](docs/01_SYSTEM_ARCHITECTURE.md). + +## Why the design is technically interesting + +### Raw records are the unit of truth + +Summary tables are reconstructed from per-request rows. Failed and invalid runs remain available for audit instead of being silently deleted. + +### Performance cannot bypass correctness or quality + +A faster candidate is excluded if it fails reference parity or a workload-specific quality gate. Quantization and custom kernels therefore compete under explicit validity constraints. + +### Profiler output is not a conclusion + +A profiler observation produces a falsifiable `HYPOTHESIS`. EdgeFlow requires a matched intervention, mediator movement, and negative control before promoting a causal explanation. + +### Deployment policies are scoped and reversible + +A policy is conditioned on workload and environment fingerprints. When those inputs drift, the policy reports `STALE` and falls back rather than extrapolating an old result. + +### The browser is not a shell + +The local web app can select only allowlisted typed operations. It cannot provide arbitrary commands, paths, or environment variables to the worker. + +## Evidence currently established on main + +The main branch demonstrates the engineering evidence chain, including: + +- package/schema validation and multi-version Python CI; +- runtime capability probing and explicit unsupported-path handling; +- raw-row reconstruction of summaries; +- correctness/quality hard filtering; +- localhost Host, Origin, token, body-size, path, and process boundaries; +- managed llama.cpp/vLLM lifecycle smoke evidence; +- correctness-gated custom-kernel fallback. + +The retained RTX 4080 SUPER runs on main are capability and integration evidence. They are intentionally labelled `CONDITIONAL_PASS` where required and do **not** establish a cross-runtime winner or deployment recommendation. + +Read [`docs/IMPLEMENTATION_STATUS.md`](docs/IMPLEMENTATION_STATUS.md) before interpreting any result. + +## Evidence not yet claimed on main + +- a complete primary-3B cross-runtime matrix under one matched protocol; +- full preregistered workload-grid coverage; +- confirmatory and holdout deployment-policy evidence; +- causal mediator and negative-control evidence for every bottleneck claim; +- end-to-end model-level speedup from the Triton path; +- a learned policy trained from the required number of validated points; +- usability evidence from external domain participants; +- clean reproduction on a second machine. + +These are experiment gates, not tasks that unit tests can replace. + +## Five-minute interview demo + +1. Run `edgeflow inspect --json` and explain why the fingerprint precedes measurement. +2. Create an exact-token workload and screen capabilities without calling the output a recommendation. +3. Execute one isolated smoke run and inspect its raw `metrics.jsonl` plus manifest. +4. Run `edgeflow validate ` and show why a latency-only result remains conditional. +5. Open the Local Control Console and demonstrate typed, loopback-only job control. +6. Change a fingerprint or remove required quality evidence and show policy refusal/fallback. + +## Code-review path + +| Area | Start here | +| :--- | :--- | +| Contracts and canonical hashing | `src/edgeflow/core/` | +| Isolated run orchestration | `src/edgeflow/experiments/` | +| Runtime adapters | `src/edgeflow/runtimes/` | +| G0–G8 authority | `src/edgeflow/validation/` | +| Statistics and bootstrap | `src/edgeflow/metrics/` | +| Deterministic diagnosis | `src/edgeflow/profiler/` | +| Policy synthesis and staleness | `src/edgeflow/policy/` | +| Typed local jobs and services | `src/edgeflow/local/` | +| Local API and dashboard | `src/edgeflow/api/` and `dashboard/` | +| Custom-kernel fallback | `src/edgeflow/kernels/` | + +## Verification commands + +```bash +uv sync --extra dev +uv run ruff check src tests +uv run pytest -q +uv run python scripts/validate_package.py +uv run python scripts/verify_results.py +``` + +GPU experiments remain separate from public CPU CI. Their artifacts must pass the same validation authority before entering a policy or headline. + +## Recommended next public evidence package + +The highest-value next milestone is a narrow, fully matched study rather than additional framework breadth: + +- one primary 3B model and exact revision; +- three runtime/configuration candidates; +- three preregistered workload buckets; +- identical prompts and generated-token accounting; +- at least 30 measured repetitions per cell; +- correctness, quality, thermal, stability, and provenance gates; +- one confirmatory rerun and one held-out replay; +- raw artifacts plus one-command verification. + +This section describes the recommended promotion target, not a completed main-branch result. + +## Further reading + +- [`docs/00_EXECUTIVE_BLUEPRINT.md`](docs/00_EXECUTIVE_BLUEPRINT.md) — success criteria and project thesis +- [`docs/03_EXPERIMENT_CATALOG.md`](docs/03_EXPERIMENT_CATALOG.md) — E00–E30 hypotheses and exit criteria +- [`docs/07_VALIDATION_AND_STATISTICS.md`](docs/07_VALIDATION_AND_STATISTICS.md) — robust statistics and validation rules +- [`docs/11_REPRODUCIBILITY_RELEASE_SECURITY.md`](docs/11_REPRODUCIBILITY_RELEASE_SECURITY.md) — release, provenance, and data handling +- [`LIMITATIONS.md`](LIMITATIONS.md) — current claim boundaries diff --git a/REVIEWER_GUIDE.md b/REVIEWER_GUIDE.md new file mode 100644 index 0000000..c373804 --- /dev/null +++ b/REVIEWER_GUIDE.md @@ -0,0 +1,80 @@ +# Reviewer Guide + +This guide provides bounded review paths for recruiters, ML systems engineers, and research reviewers. + +## Five-minute review + +1. Read the project thesis and evidence rules in [`README.md`](README.md). +2. Read the engineering summary in [`PORTFOLIO.md`](PORTFOLIO.md). +3. Inspect the engineering-versus-experiment table in [`docs/IMPLEMENTATION_STATUS.md`](docs/IMPLEMENTATION_STATUS.md). +4. Read [`LIMITATIONS.md`](LIMITATIONS.md) before interpreting any retained run. + +## Fifteen-minute engineering review + +Run the public checks: + +```bash +uv sync --extra dev +uv run ruff check src tests +uv run pytest -q +uv run python scripts/validate_package.py +uv run python scripts/verify_results.py +``` + +Then inspect: + +- `src/edgeflow/core/` for typed contracts and canonical hashes; +- `src/edgeflow/validation/` for G0–G8 authority; +- `src/edgeflow/experiments/` for isolated run orchestration; +- `src/edgeflow/local/` for typed process and runtime control; +- `src/edgeflow/policy/` for eligibility, scope, and staleness behavior. + +## Thirty-minute ML systems review + +1. Create an exact-token workload. +2. Screen candidates and confirm that screening does not call itself a recommendation. +3. Run one bounded benchmark and inspect the artifact directory. +4. Recompute its verdict from raw rows. +5. Remove or invalidate a quality artifact and verify that ranking is refused. +6. Inspect a profiler summary and trace how it becomes a hypothesis rather than a conclusion. +7. Inspect custom-kernel cache keys and reference fallback behavior. + +## Research-method review + +Use these documents in order: + +1. [`docs/02_EXPERIMENT_MASTER_PLAN.md`](docs/02_EXPERIMENT_MASTER_PLAN.md) +2. [`docs/03_EXPERIMENT_CATALOG.md`](docs/03_EXPERIMENT_CATALOG.md) +3. [`docs/05_AUTOTUNING_AND_CAUSAL_METHOD.md`](docs/05_AUTOTUNING_AND_CAUSAL_METHOD.md) +4. [`docs/07_VALIDATION_AND_STATISTICS.md`](docs/07_VALIDATION_AND_STATISTICS.md) +5. [`docs/11_REPRODUCIBILITY_RELEASE_SECURITY.md`](docs/11_REPRODUCIBILITY_RELEASE_SECURITY.md) + +Check whether every claimed conclusion has: + +- a preregistered scope; +- controlled variables and a negative control; +- raw artifacts and exact provenance; +- correctness and quality evidence; +- sufficient repetitions and stability; +- confirmatory and holdout support; +- an explicit non-generalization boundary. + +## Questions the repository should answer + +- What exact workload and environment does this result describe? +- Can the summary be reconstructed from raw rows? +- Why is the run eligible, conditional, invalid, or stale? +- Which quality loss would disqualify the faster candidate? +- Is a bottleneck statement an observation, hypothesis, or verified intervention result? +- What happens after hardware, model, runtime, or workload drift? +- Can the local UI trigger anything outside the typed allowlist? + +## Questions the repository does not yet answer + +- Which runtime universally wins for local LLM inference? +- How do the retained results transfer to a different GPU? +- Does the custom kernel provide a confirmed end-to-end model speedup? +- Does a learned selector outperform the deterministic policy? +- Can a second machine reproduce the complete formal conclusion? + +Those questions require additional evidence rather than stronger wording.