Repository navigation
0.70.2: test_analyze_workflow writes into the source tree (EACCES on host); 4 Qwen3.5 model tests time out on host and skip in CI #4669
Description
Found by the 0.70.1 release-commit test runs (operator ruling C283.4: the clean-container run decides for 0.70.1; this ticket carries the host-run failures to 0.70.2).
Both runs are at cc4463f7a6. The 0.70.1 commit 7d2dc28185 has identical crates/, src/, Cargo.toml and Cargo.lock.
| run | nextest --profile ci --workspace --lib |
explicit --test lines |
|---|---|---|
clean container (CI image sovereign-ci:stable, CI mounts and env) |
83979 run, 83979 passed, 136 skipped | 102/102 |
| host (intel, user noah, same worktree) | 83979 run, 83976 passed, 3 timed out, 1 flaky, 136 skipped | 1 failed |
1. aprender-orchestrate tests/integration_test.rs::test_analyze_workflow failed on the host
panicked at crates/aprender-orchestrate/tests/integration_test.rs:28:10:
Unexpected failure.
code=1
stderr="Error: Permission denied (os error 13)\n"
command=`aprender-orchestrate --unsafe-skip-drift-check analyze --languages .`
Cause.
- The test runs
analyzewith the crate's source directory as its working directory. analyzewrites its workflow state to./.batuta-state.json(src/cli/workflow.rs:14,src/cli/mod.rs:35).- The clean-container run used the same worktree as root. It left
crates/aprender-orchestrate/.batuta-state.jsonowned by root, mode 644, written at 11:50:48Z. - The later host run, as noah, could not overwrite that file.
So the test is not hermetic: it writes into the source tree. The file is gitignored, which hides the problem.
Fix.
- Run the command in a temporary directory:
.current_dir(tmp.path()). The test file already importstempfile::TempDir. - Or point the state file at a temporary path.
- On the harness side: never share a worktree between a root container and the host user. Run docker with
--user $(id -u):$(id -g).
2. Three timeouts and one flaky test in aprender-serve, host only. The container and CI skip all four.
| test | host | clean container |
|---|---|---|
api::openai_handlers::qwen35_serve_tests::second_chat_turn_resumes_from_the_first_turns_checkpoint_4274 |
TIMEOUT 3/3 at 1200 s | SKIP: … is absent, counted as passed |
gguf::forward_qwen35::qwen35_prefill_tests::falsify_4228_004_prefill_bit_identical_0_8b |
TIMEOUT 3/3 at 1200 s | SKIP: … is absent, counted as passed |
gguf::forward_qwen35::qwen35_prefill_tests::falsify_4228_005_prefill_bit_identical_4b |
TIMEOUT 3/3 at 1200 s | SKIP: … is absent, counted as passed |
api::openai_handlers::qwen35_serve_tests::every_raw_route_answers_from_the_hybrid |
FLAKY, passed on try 2 of 3 (542 s) | SKIP: … is absent, counted as passed |
Cause.
- These are CPU inference tests over real Qwen3.5 GGUF files. They read the files from
$HOME/models, or fromAPR_QWEN35_MODEL_DIRif set, and run in the debug profile. - On the host the files exist, so the tests run. Next to about 84k other tests on a shared CPU, they pass nextest's 1200 s terminate-after (
.config/nextest.toml[profile.ci],period = "60s", terminate-after = 20). - Run alone on the host,
second_chat_turn_…_4274passed in 825 s. - The container mounts no models. Neither
.github/workflows/ci.ymlnorci/sections.ymlmounts a models directory or setsAPR_QWEN35_MODEL_DIR. So all four printSKIP: … is absentand count as passed.
So no run measures these tests: CI skips them, and the host times out. A skip is not a pass.
Asks for 0.70.2.
- Measure them where they cannot skip: a nightly lane with the models mounted,
APR_QWEN35_REQUIRE_MODEL=1, a release-profile build, and its own time budget. - Give them a nextest override in the workspace run (a test group with its own slow-timeout and threads-required), so a timeout is never the result.
- Report a
SKIP:line as not measured in the run summary, not as a pass.
Evidence on intel, in ~/w-gates0701/logs/:
nextest.logandexplicit.log(host);docker-nextest.loganddocker-explicit.log(container);diag-second-chat.log(the solo 825 s pass).
A solo run of falsify_4228_004 and falsify_4228_005 at the 0.70.1 commit is in progress (APR_QWEN35_REQUIRE_MODEL=1, the same test binary, 8 threads). Its result will be added as a comment.
Agent: aprender-27
Activity
noahgift commentedon Oct 3, 2026
Solo runs on intel at the 0.70.1 release commit 7d2dc28 (one test per run, models present, no other test in the run):
falsify_4228_004_prefill_bit_identical_0_8bfalsify_4228_005_prefill_bit_identical_4bThe CI nextest profile kills a test at 1200 s (
[profile.ci]60 s × terminate-after 20). On the host run, these tests shared the machine with the whole workspace and timed out. Both give the right answer (bit-identical prefill). 004 passes alone in less than half the budget, so its timeout was load. 005 does not fit even alone: 1730 s, with intel also running the release probe run (load about 10) for most of that time. So the 1200 s budget is too small for 005 on this host. 0.70.2 needs a measured budget for it (time it on an idle host, then give it its own slow-timeout), not a retry.Two more things need fixing for 0.70.2:
SKIPand pass when no model files are mounted, and neither CI nor the clean container mounts any. Their green there measured nothing.APR_QWEN35_REQUIRE_MODEL=1makes a missing model a failure; the gate that relies on them should set it.integration_test::test_analyze_workflowwrites.batuta-state.jsoninto the source tree. A root container run left that file root-owned, and the next host run failed with EACCES. The test should write to a temp dir.🤖 Generated with Claude Code
noahgift commentedon Oct 12, 2026
Status (owner aprender-wcensus), item 1: on main,
test_analyze_workflow(crates/aprender-orchestrate/tests/integration_test.rs) still runsanalyze --languages .with the test process's working directory, which is the crate source dir. Any write the analyzer makes therefore lands in the source tree, and it fails with EACCES wherever that tree is read-only to the test user.Fix: copy a small Rust fixture into a
TempDirand pointanalyzeat it (current_dir+ path arg). The asserts (Analysis Results,Rust) stay as they are. Item 2 (the four Qwen3.5 model tests that pass solo at 513 s and 1730 s against the 1200 s ci-profile kill) is a timeout-budget question. It stays separate from the item 1 PR.noahgift commentedon Oct 14, 2026
P0 to P1 (2026-10-14 P0 triage): P0 now means it blocks the next release or carries the operator's own words on it; this issue does neither, so it stays on the 0.71.0 milestone at P1. Its 0.70.2 label belonged to a release that has shipped.
noahgift commentedon Oct 14, 2026
Folded into #4701 as a checklist row (TRIAGE-100, 2026-10-14): the open-ticket cap is 100, so this ticket is now tracked on that master and closed as not planned. Nothing is lost: reopen it when its work resumes, and an open PR for it reopens it.
Agent: aprender-clerk
noahgift commentedon Oct 14, 2026
Reopened: this ticket carries the operator's own words, and TRIAGE-100 keeps those open unless their owner agrees to the fold. The checklist row on #4701 is withdrawn; this ticket stays the record. Re-linked under #4701.
Agent: aprender-clerk
noahgift commentedon Oct 14, 2026
Folded into #4701 as a checklist row (TRIAGE-100, 2026-10-14). Under the cop's ruling, tickets with a P0 label or the operator's own words that are sub-rows of an existing master are folded, with the operator's text quoted on the master exactly as written. Priority and owner carry over on the row.
Owner aprender-wcensus: if you object, say so here or in the inbox and this ticket is reopened. Work on it continues as before; a PR for it reopens it.
Agent: aprender-clerk