The check.yml that `ai-eng init` writes can never go green: un-installable pin, a bare `doctor --ci`, and an `audit verify` with no chain

[soydachi]

What happened

ai-eng init --project writes .github/workflows/check.yml (template at src/ai_engineering/skeletons.py:218-233). On a real consumer repository (observed on arcasilesgroup/ai-shell, a public CLI repo, 2026-08-28) every step of that template that touches ai-engineering is broken in a different way. The template contradicts the framework's own doctrine: not evaluated is never green, and a step that can never pass is a bug, not a gate.

1. uv tool install ai-engineering==${PIN} — the pin is not installable

The template pins {version} = the local wheel's version, written into .ai/config.toml at init. If that version is not published to PyPI, CI dies at install. Observed:

Because there is no version of ai-engineering==1.0.0 ... we can conclude that your requirements are unsatisfiable.

The local install was 1.0.0 (editable from the framework repo); PyPI's latest was 0.13.0. The template assumes every pin exists upstream. Nothing at init checks that the version being written as the CI pin is actually resolvable.

2. ai-eng doctor --ci — a bare step that can never exit 0 on a runner

By the framework's own design, --ci counts every check a runner cannot answer (surface liveness, chain freshness, observability) as unanswered → INCOMPLETE → exit 1. So the step as written is red on every commit, on every repository. The framework's own check.yml solved exactly this (run ai-eng --json doctor --ci || true, then a small reader that fails only on decided faults and prints the unanswerable count) — but the shipped template still writes the naive command. The fix exists in the dogfood repo and is not propagated to consumers.

Same shape one line later: ai-eng audit verify on a runner has no sealed chain (the buffer lives under ~/.ai-engineering/state/, outside the clone), which is again INCOMPLETE → exit 1. The framework's own workflow seals the chain with a SessionEnd hook before verifying; the template neither seals nor tolerates the absence.

3. The template's actions are tag-pinned, not SHA-pinned

actions/checkout@v5, astral-sh/setup-uv@v5 — while the framework's own workflows pin full commit SHAs. A template that ships weaker supply-chain hygiene than the repo that generates it teaches consumers to ignore the doctrine.

4. The wired justfile recipe has the same problem

init also writes justfile with wired: ai-eng doctor (strict, no --ci), and check: wired build lint test security counts. Any consumer that follows the template's just check in CI inherits problem 2 through the back door.

Expected

The generated workflow should be one that can actually go green and that encodes the framework's own verdict rules:

• doctor: JSON report + fail only on decided faults, report the unanswerable (the pattern already proven in the framework's own check.yml), not a bare doctor --ci.
• audit verify: seal the run's own chain first (as the framework's own CI does), or make a missing chain an explicit, reported non-failure.
• pin: at init, verify the recorded version resolves on PyPI; if it does not, record an explicit install source (e.g. a git commit) instead of writing a pin CI cannot satisfy.
• actions: full-SHA pins in the template, consistent with the framework's own workflows.
• justfile wired: document or encode the CI-tolerant variant so just check is honest in both places.

Secondary: assertion 14 is blind to rulesets

doctor.py:1090-1104 queries only the legacy branch-protection API. A repository protected exclusively by a modern ruleset (which is what GitHub recommends for new repos) gets:

could not evaluate: the API call did not succeed — a fork, or a token without permissions

…while T0 protection is in fact active. The claim is true but the evidence path is incomplete: the reader should also query /repos/{owner}/{repo}/rulesets and match branch conditions, and the error text should stop misattributing a ruleset-only repository to "a fork or a token without permissions".

Reproduction

1. ai-eng init --project in any repository whose pin is not on PyPI.
2. Push, open the generated workflow: the install step fails (problem 1); fix the pin manually and the next steps fail red forever (problems 2-4).