Skip to content

CI fat_driver: leftover root-owned ws makes setup_workspace raise FileExistsError (runner-poisoning) #4507

Closed

Description

@noahgift

Source: aprender-57, 2026-09-26. It blocked #4479 (the head of the ONT-10 chain) twice on framework16-2.

Symptom: FileExistsError on _work/temp/fat/workspace-test-shard_1_3/ws. fat_driver setup_workspace calls rmtree(ignore_errors=True), which silently leaves behind files that a docker section wrote as root, and then mkdir raises.

Fix: move the leftover aside (rename) before mkdir, and never ignore_errors silently.

Done when:

  • A test plants a root-owned or undeletable leftover ws, and setup_workspace succeeds with a fresh ws.
  • The rename-aside path is logged, and the aside dirs are cleaned or bounded.
  • A mutant that restores rmtree+mkdir turns the test RED.

Activity

  1. added this to the 0.70.0 milestone on Sep 26, 2026
  2. modified the milestones: 0.70.0, 0.71.0, backlog on Oct 4, 2026
  3. noahgift commented on Oct 12, 2026

    @noahgift
    ContributorAuthor

    evidence for close: 989cb01 (#4655), ancestor of main 6bd89b0. Checked at main: clear_dir in scripts/ci/fat_driver.py moves a leftover unremovable workspace aside and logs "moved aside to" instead of raising FileExistsError; its self-test asserts that log line, and ci.yml runs that self-test.

  4. noahgift commented on Oct 12, 2026

    @noahgift
    ContributorAuthor

    Cop batch close (FLOW-004 drain, ≤43/d): fix verified on main — merged #4655 ; ancestor of origin/main: 989cb01 . Evidence: #4507 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions