test: isolate CODEWHALE_HOME in the provider onboarding tests - #5933
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_cf49479e-0a29-482d-9975-78de7837befb) |
|
Claude finished @Hmbown's task in 59s —— View job Review: test(tui) isolate CODEWHALE_HOME in provider onboarding tests
Summary: Small, well-targeted fix. Correctness
Provider/model/route safety: N/A — no provider-prefix inference logic touched. Reuse/simplification: Reuses the existing Tests: This is the test fix; the PR body reports Security: No secret handling changes; test-only. No issues found — this is a correct, minimal, well-evidenced fix. |
There was a problem hiding this comment.
🔵 Needs a closer look
ConfigPathEnvGuard declares the env lock field last, causing it to drop first and potentially restore env vars without holding the required mutex (violating EnvVarGuard’s safety contract).
Pull request overview
This PR tightens test isolation for tui::ui::provider_key_validation_tests by ensuring onboarding-related persistence targets a temporary $CODEWHALE_HOME rather than the developer’s real ~/.codewhale, preventing setup_state.json/settings pollution as described in #5932.
Changes:
- Extend the tests’
ConfigPathEnvGuardto also setCODEWHALE_HOMEto the same temp.codewhaledirectory as the config path. - Adjust the constructed
config_pathto live under that isolated temp home (<tmp>/.codewhale/config.toml). - Add an explanatory comment documenting why
CODEWHALE_HOMEmust be guarded for these tests.
File summaries
| File | Description |
|---|---|
| crates/tui/src/tui/ui.rs | Updates the provider onboarding test harness to isolate CODEWHALE_HOME alongside CODEWHALE_CONFIG_PATH to prevent real home-directory writes. |
Review details
Suppressed comments (1)
crates/tui/src/tui/ui.rs:1285
EnvVarGuardrequires the test-env lock to be held until after the guard is dropped (itsDropusesunsafe { set_var/remove_var }under that assumption). With_lockdeclared last, Rust will drop_lockfirst, so the env-var guards may restore variables without holding the mutex, reintroducing cross-test races.
_codewhale_home: crate::test_support::EnvVarGuard,
_codewhale_config_path: crate::test_support::EnvVarGuard,
_deepseek_config_path: crate::test_support::EnvVarGuard,
_lock: crate::test_support::TestEnvLock,
}
- Files reviewed: 1/1 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
Codewhale review
Test-only isolation fix: ConfigPathEnvGuard now sets CODEWHALE_HOME to the same temp .codewhale directory as CODEWHALE_CONFIG_PATH, preventing onboarding setup files from being written to the real user home.
Findings
- [INFO] No automated regression assertion for home isolation (
crates/tui/src/tui/ui.rs:1297)
The PR verifies via a manual mtime check that the real ~/.codewhale/setup_state.json is not modified, but the test suite itself does not assert that onboarding writes setup_state.json under the guarded temp CODEWHALE_HOME or that the previous home is untouched. If the guard is later removed or bypassed, the existing tests can still pass while polluting the developer's home.
Suggestions
crates/tui/src/tui/ui.rs:1297— Add an automated regression check in the provider onboarding test, or a helper assertion, that setup_state.json is created under the temporary CODEWHALE_HOME and not under the original CODEWHALE_HOME. This makes the fix self-verifying instead of relying on manual mtime validation.
Assessment
The change is correct and low-risk: it follows the existing env guard pattern and redirects both CODEWHALE_HOME and CODEWHALE_CONFIG_PATH consistently. The only gap is lack of automated coverage for the regression, which would be nice to add but does not block merging.
Advisory review by Codewhale (codewhale review --pr 5933 --post, head 4025edec248dd735f1dbaace01ab4d81ee6fb832). Line-specific findings are also posted as inline review comments; mechanical fixes arrive as committable suggestions you can apply from the Files tab. CODEOWNERS approval still governs merge.
| .expect("config dir"); | ||
| Self { | ||
| _tmp: tmp, | ||
| _codewhale_home: crate::test_support::EnvVarGuard::set("CODEWHALE_HOME", &home), |
There was a problem hiding this comment.
[INFO] No automated regression assertion for home isolation
The PR verifies via a manual mtime check that the real ~/.codewhale/setup_state.json is not modified, but the test suite itself does not assert that onboarding writes setup_state.json under the guarded temp CODEWHALE_HOME or that the previous home is untouched. If the guard is later removed or bypassed, the existing tests can still pass while polluting the developer's home.
| .expect("config dir"); | ||
| Self { | ||
| _tmp: tmp, | ||
| _codewhale_home: crate::test_support::EnvVarGuard::set("CODEWHALE_HOME", &home), |
There was a problem hiding this comment.
Add an automated regression check in the provider onboarding test, or a helper assertion, that setup_state.json is created under the temporary CODEWHALE_HOME and not under the original CODEWHALE_HOME. This makes the fix self-verifying instead of relying on manual mtime validation.
…5932) successful_custom_provider_activation_completes_onboarding and its siblings guarded the config path but not CODEWHALE_HOME, so completing onboarding ran the setup transaction against the developer's real ~/.codewhale: after tonight's full-suite runs the founder's setup_state.json recorded provider=fixture-local as a verified step and codewhale doctor repeated it. ConfigPathEnvGuard now points CODEWHALE_HOME at the same temp dir as the config path. Receipt: real ~/.codewhale/setup_state.json mtime unchanged across cargo test -p codewhale-tui --lib -- tui::ui::provider_key_validation_tests (15 passed, 0 failed). The systemic guard (fail any unit test that resolves the real home) stays open on #5932. Signed-off-by: CodeWhale Bot <hmbown@gmail.com>
4025ede to
bb76ef5
Compare
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_c86f542f-191e-40f6-a1d7-e15b55bf5ee1) |
|
Claude finished @Hmbown's task in 2m 44s —— View job Review: test(tui) isolate
|
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_931dfb4a-34b3-4ea6-9b2b-44a336e03379) |
|
Claude finished @Hmbown's task in 3m 49s —— View job Review: test(tui) isolate CODEWHALE_HOME (#5933)
Summary: The core mechanism (comparing the current Findings1. Correctness/soundness — unsynchronized
Meanwhile, dozens of existing tests do
Fix: move the override installation so it only ever reads 2. Reuse — Already flagged in an earlier review pass on this PR; still unused anywhere in the diff or repo. The new regression test in Good choices worth noting
|
There was a problem hiding this comment.
Codewhale review
PR attempts to isolate all unit tests from the developer's real ~/.codewhale via a process-wide test home override, targeted CODEWHALE_HOME guards, and hermetic CI env. The direction is sound, but the override installation is vulnerable to a parallel-test race, and the CI step has a Windows shell portability problem.
Findings
- [ERROR] Test home override may be installed against a redirected HOME before the env lock is held (
crates/tui/src/test_env_lock.rs:160)
Inlock_test_env,ensure_test_home_override()is called beforeenv_lock().lock(). With parallel tests, a waiting test can execute this while another test holds the lock and has temporarily changedHOMEto a temp directory.install_test_home_overridethen persists that temp path asreal_user_home; onceHOMEis restored, later implicitcodewhale_home()lookups compare the realHOMEagainst the saved temp path and skip the override, so they fall back to the developer's real~/.codewhale.test_tui_optionshas the same lazy-install risk. The override should be installed once before any env-mutating test runs, orensure_test_home_overrideshould be moved inside the env lock after acquisition. - [WARNING] CI step uses bash-only mkdir on Windows runners (
.github/workflows/ci.yml:599)
The updated step runsmkdir -p "${CODEWHALE_HOME}"under the repository's default shell. On Windows the default Actions shell is pwsh, where${CODEWHALE_HOME}does not expand to the environment variable andmkdir -pis not bash syntax. This can fail to create the hermetic home directory or fail the workflow on the matrix's Windows jobs. Addshell: bashto this step or use a cross-platform creation command. - [INFO] Test-only override is exposed in production codewhale-paths (
crates/paths/src/lib.rs:141)
install_test_home_overrideandtest_home_overrideare#[doc(hidden)]but public and unconditionally compiled into production. Production code can accidentally mutate global path resolution for all latercodewhale_home()calls. Consider gating these hooks behind a test-utils feature orcfg(test)equivalent. - [INFO] Local non-TUI test binaries are not protected by the process-wide override
The override is installed only fromcrates/tui/src/test_support.rs. Running e.g.cargo test -p codewhale-configlocally does not install it, so an unsealed test in those crates can still hit~/.codewhale. CI hides this becauseCODEWHALE_HOMEis set globally, but the localcargo test -ppath is not isolated for every crate.
Suggestions
-
.github/workflows/ci.yml:598— Force bash for this step so the mkdir -p command and environment variable expansion behave consistently on Windows runners.run: | mkdir -p "${CODEWHALE_HOME}" cargo nextest run --workspace --all-features --locked --profile ci shell: bash
Assessment
The isolation strategy is reasonable but not yet safe under the parallel Rust test harness. Fix the lazy override ordering before merge, and make the CI directory creation cross-platform.
Advisory review by Codewhale (codewhale review --pr 5933 --post, head 97288e0c14755bdfb0435ca12f627e2042677c41). Line-specific findings are also posted as inline review comments; mechanical fixes arrive as committable suggestions you can apply from the Files tab. CODEOWNERS approval still governs merge.
| /// If a prior test panicked while holding the lock, recover the guard instead | ||
| /// of cascading failures across unrelated tests. | ||
| pub(crate) fn lock_test_env() -> TestEnvLock { | ||
| crate::test_support::ensure_test_home_override(); |
There was a problem hiding this comment.
[ERROR] Test home override may be installed against a redirected HOME before the env lock is held
In lock_test_env, ensure_test_home_override() is called before env_lock().lock(). With parallel tests, a waiting test can execute this while another test holds the lock and has temporarily changed HOME to a temp directory. install_test_home_override then persists that temp path as real_user_home; once HOME is restored, later implicit codewhale_home() lookups compare the real HOME against the saved temp path and skip the override, so they fall back to the developer's real ~/.codewhale. test_tui_options has the same lazy-install risk. The override should be installed once before any env-mutating test runs, or ensure_test_home_override should be moved inside the env lock after acquisition.
| if: needs.changes.outputs.heavy == 'true' && (matrix.os != 'ubuntu-latest' || github.event_name == 'workflow_dispatch' || github.event_name == 'pull_request') | ||
| run: cargo nextest run --workspace --all-features --locked --profile ci | ||
| run: | | ||
| mkdir -p "${CODEWHALE_HOME}" |
There was a problem hiding this comment.
[WARNING] CI step uses bash-only mkdir on Windows runners
The updated step runs mkdir -p "${CODEWHALE_HOME}" under the repository's default shell. On Windows the default Actions shell is pwsh, where ${CODEWHALE_HOME} does not expand to the environment variable and mkdir -p is not bash syntax. This can fail to create the hermetic home directory or fail the workflow on the matrix's Windows jobs. Add shell: bash to this step or use a cross-platform creation command.
| /// (a provider-onboarding test once persisted a fixture provider into the | ||
| /// founder's real `setup_state.json`, #5932). The first install wins and is | ||
| /// returned as `true`; later calls are ignored. Production never calls this. | ||
| #[doc(hidden)] |
There was a problem hiding this comment.
[INFO] Test-only override is exposed in production codewhale-paths
install_test_home_override and test_home_override are #[doc(hidden)] but public and unconditionally compiled into production. Production code can accidentally mutate global path resolution for all later codewhale_home() calls. Consider gating these hooks behind a test-utils feature or cfg(test) equivalent.
| run: | | ||
| mkdir -p "${CODEWHALE_HOME}" | ||
| cargo nextest run --workspace --all-features --locked --profile ci |
There was a problem hiding this comment.
Force bash for this step so the mkdir -p command and environment variable expansion behave consistently on Windows runners.
| run: | | |
| mkdir -p "${CODEWHALE_HOME}" | |
| cargo nextest run --workspace --all-features --locked --profile ci | |
| run: | | |
| mkdir -p "${CODEWHALE_HOME}" | |
| cargo nextest run --workspace --all-features --locked --profile ci | |
| shell: bash |
97288e0 to
6241201
Compare
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_654f7dca-e5a8-44ba-86be-4529127b5bc2) |
|
Claude finished @Hmbown's task in 3m 20s —— View job Review: test(tui) isolate
|
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_75d7bce6-366e-4342-a0da-524cc402c84b) |
|
Claude finished @Hmbown's task in 3m 16s —— View job Review: test(tui) isolate CODEWHALE_HOME (#5933)
Scope confirmed: current diff is exactly Good news first: the Windows-shell finding from earlier passes (Devin, codewhale-agent) is fixed in Findings1. [Confirmed, blocking] Job-wide
Two tests never guard
With Fix: either have these two tests explicitly clear 2. [Info] Same shape may exist elsewhere, but a spot check of the largest HOME-touching files came back clean. I checked Good choices
Provider/model/route safety: N/A — no provider-prefix inference logic touched. Note: I could not execute |
There was a problem hiding this comment.
Codewhale review
The PR makes provider onboarding tests hermetic by setting CODEWHALE_HOME alongside CODEWHALE_CONFIG_PATH, and makes the main CI cargo nextest step run with an isolated CODEWHALE_HOME under runner.temp.
Assessment
Low-risk test and CI isolation improvement. The guard directs both home and config path into the same temporary .codewhale tree, and the CI step creates the isolated home before running tests. No correctness issues found.
Advisory review by Codewhale (codewhale review --pr 5933 --post, head 4b08a2534470c3a3c67e58b2ca1d9dc3e719a351). Line-specific findings are also posted as inline review comments; mechanical fixes arrive as committable suggestions you can apply from the Files tab. CODEOWNERS approval still governs merge.
4b08a25 to
c7fc000
Compare
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_0fad2c48-0c0b-4e6d-8e70-ddd555688e63) |
|
Claude finished @Hmbown's task in 1m 37s —— View job Review: test(tui) isolate
|
There was a problem hiding this comment.
Codewhale review
Extends ConfigPathEnvGuard to set CODEWHALE_HOME to the isolated temp directory used for config paths, so provider onboarding tests no longer write setup_state.json into the developer's real ~/.codewhale.
Findings
- [INFO] Relies on manual mtime verification instead of an automated assertion (
crates/tui/src/tui/ui.rs:1280)
The fix is verified manually by checking that ~/.codewhale/setup_state.json is unchanged during the test run. Without an automated guard/assertion, a future regression could reintroduce writes to the real home without failing any test. - [INFO] TempDir field is dropped before environment guards (
crates/tui/src/tui/ui.rs:1276)
Rust drops struct fields in declaration order, so _tmp is removed before _codewhale_home and the other EnvVarGuards restore their variables. This is not currently harmful because the guards do not touch the filesystem, but declaring _tmp after the env guards would be more robust.
Suggestions
crates/tui/src/tui/ui.rs:1276— Reorder the fields in ConfigPathEnvGuard so_tmpis declared after theEnvVarGuardfields. That keeps the isolated temp directory alive until the environment variables are restored when the guard drops.
Assessment
The test-only isolation change is correct and low risk. It should resolve the #5932 contamination. The remaining notes are hardening suggestions rather than blocking issues.
Advisory review by Codewhale (codewhale review --pr 5933 --post, head c7fc0007b378543e83f8ec235307b7b76c2768bd). Line-specific findings are also posted as inline review comments; mechanical fixes arrive as committable suggestions you can apply from the Files tab. CODEOWNERS approval still governs merge.
| _tmp: TempDir, | ||
| // Onboarding completion runs the setup transaction (setup_state.json, | ||
| // settings.toml) against `CODEWHALE_HOME`; without this guard the | ||
| // fixture provider landed in the developer's real ~/.codewhale (#5932). |
There was a problem hiding this comment.
[INFO] Relies on manual mtime verification instead of an automated assertion
The fix is verified manually by checking that ~/.codewhale/setup_state.json is unchanged during the test run. Without an automated guard/assertion, a future regression could reintroduce writes to the real home without failing any test.
| @@ -1275,6 +1275,10 @@ mod provider_key_validation_tests { | |||
|
|
|||
| struct ConfigPathEnvGuard { | |||
There was a problem hiding this comment.
[INFO] TempDir field is dropped before environment guards
Rust drops struct fields in declaration order, so _tmp is removed before _codewhale_home and the other EnvVarGuards restore their variables. This is not currently harmful because the guards do not touch the filesystem, but declaring _tmp after the env guards would be more robust.
| // fixture provider landed in the developer's real ~/.codewhale (#5932). | ||
| _codewhale_home: crate::test_support::EnvVarGuard, | ||
| _codewhale_config_path: crate::test_support::EnvVarGuard, | ||
| _deepseek_config_path: crate::test_support::EnvVarGuard, |
There was a problem hiding this comment.
Reorder the fields in ConfigPathEnvGuard so _tmp is declared after the EnvVarGuard fields. That keeps the isolated temp directory alive until the environment variables are restored when the guard drops.
Refs #5932 (targeted guard; the process-wide guard and a hermetic CI home both stay open there — each broke tests that rely on an implicit home, receipts on the issue)
The onboarding tests in
tui::ui::provider_key_validation_testsguardedCODEWHALE_CONFIG_PATHbut notCODEWHALE_HOME, so completing onboarding wrotesetup_state.jsoninto the developer's real~/.codewhale(tonight's full-suite runs left the founder's provider step recorded asfixture-local).ConfigPathEnvGuardnow sets both to one temp dir.Verified:
RUST_MIN_STACK=16777216 cargo test -p codewhale-tui --lib -- tui::ui::provider_key_validation_tests→ 15 passed, 0 failed, with the real~/.codewhale/setup_state.jsonmtime unchanged across the run.Note
Low Risk
Test-only environment isolation; no production runtime or persistence behavior changes.
Overview
Fixes #5932 by extending
ConfigPathEnvGuardinprovider_key_validation_testsso onboarding runs against a temp home, not the developer's real~/.codewhale.The guard already pointed
CODEWHALE_CONFIG_PATH(andDEEPSEEK_CONFIG_PATH) at a tempconfig.toml, but onboarding completion still persistedsetup_state.jsonandsettings.tomlviaCODEWHALE_HOME. Full-suite runs could therefore record fixture provider progress in the user's actual setup state.ConfigPathEnvGuard::new()now builds a single temp.codewhaledirectory, setsCODEWHALE_HOMEto that path alongside the existing config env guards, and documents why both are required.Reviewed by Cursor Bugbot for commit c7fc000. Bugbot is set up for automated code reviews on this repo. Configure here.