fix(ci): enforce the ordinary eval output workspace

[mrizzi]

Summary

Implements TC-6773, fixing TC-6772.

• Explicitly grant Claude access to the requested ordinary eval workspace and require that exact absolute path throughout execution, grading and aggregation.
• Reject exit-zero runs unless benchmark.json, feedback.json and summary.md are nonempty regular files in that workspace.
• Preserve attempts for subsequent skills after failures.
• Implements TC-6774: reject symlinked required result paths, with regression coverage for all three artifacts and continued execution after a linked result is rejected.

This addresses PR #299 run 37617450108, which posted “No results produced” despite a successful ordinary job. The misplaced-output and missing-output cases are reproduced by tests executing the real workflow shell with a non-inference Claude stub.

Validation

• Full script suite: 129 passed.
• Skillsaw: 0 errors, 7 existing warnings.
• Plugin manifest validation and git diff --check: passed.
• Regression coverage: correct, absent, relocated, empty and directory artifacts; explicit workspace grant; sandbox flags; CLI failures; subsequent-skill continuation.

Rollout and limits

Hosted PR evals execute the workflow from trusted main, so this small fix must merge before rerunning #299. It changes only the ordinary invocation and required output-file checks. WIF, credential isolation, approval gates, native source pins, native assertions, publication and timeouts are unchanged.

The earlier verify-pr background-case stall remains a separate issue. Required-file presence is enforced here; complete assertion validation is outside this task. Adding the workspace directory grant addresses a possible permission obstacle; the historical logs do not prove a permission denial caused the relocation.

Summary by Sourcery

Enforce the ordinary eval workspace and validate required result artifacts before reporting CI success.

Bug Fixes:

• Enforce that ordinary evals write nonempty, regular benchmark, feedback, and summary artifacts directly in the requested workspace.
• Reject relocated, missing, empty, directory, and symlinked eval result artifacts while allowing subsequent skills to continue after failures.

Enhancements:

• Grant eval agents access to the exact workspace and require its unchanged use throughout evaluation, grading, and aggregation.

CI:

• Add regression coverage for workspace authorization, output validation, sandbox settings, CLI failures, and continuation across multiple skills.

Tests:

• Exercise the real ordinary-eval CI shell against missing, misplaced, empty, directory, and symlinked result scenarios.

[sourcery-ai]

Reviewer's Guide

The ordinary PR eval workflow now explicitly authorizes and consistently uses the requested absolute output workspace, rejects successful CLI runs that do not produce all three nonempty result files there, and continues processing subsequent skills after failures. End-to-end tests exercise the real shell workflow and cover valid output, misplaced or invalid artifacts, CLI failures, sandbox settings, workspace grants, and continuation behavior.

Sequence diagram for enforced ordinary eval workspace

sequenceDiagram
    participant Workflow
    participant Claude
    participant Workspace
    participant Validator
    participant NextSkill

    Workflow->>Claude: invoke with workspace and --add-dir workspace
    Claude->>Workspace: write benchmark.json
    Claude->>Workspace: write feedback.json
    Claude->>Workspace: write summary.md
    Workflow->>Validator: check required files in workspace
    alt all files are nonempty regular files
        Validator-->>Workflow: validation passes
    else missing, empty, directory, or relocated artifact
        Validator-->>Workflow: mark skill failed
    end
    Workflow->>NextSkill: continue processing subsequent skill

Loading

Flow diagram for ordinary eval result validation

flowchart TD
    A[Start skill eval] --> B[Invoke Claude with exact workspace grant]
    B --> C{CLI exits successfully?}
    C -- No --> D[Mark skill failed]
    C -- Yes --> E{benchmark.json feedback.json summary.md are nonempty regular files in workspace?}
    E -- Yes --> F[Accept skill results]
    E -- No --> D
    D --> G[Continue to next skill]
    F --> G
    G --> H[Finish workflow with failure status if any skill failed]

Loading

File-Level Changes

Change	Details	Files
Enforce a single authorized workspace for ordinary eval execution and output generation.	Instruct Claude and all subagents to use the requested absolute workspace unchanged for eval, grading, aggregation, and final artifacts. Grant Claude explicit access to that workspace while preserving existing permission and sandbox settings. Require benchmark.json, feedback.json, and summary.md to be nonempty regular files directly in the requested workspace. Mark failures without aborting the loop so later skills are still attempted.	.github/workflows/eval-pr-run.yml
Add end-to-end regression coverage for workspace authorization, artifact validation, and failure continuation.	Execute the real workflow shell with a synthetic non-inference Claude stub across valid, relocated, absent, empty, directory, and CLI-failure scenarios. Verify exact workspace authorization, preserved sandbox flags, required artifacts, and continuation to subsequent skills.	plugins/sdlc-workflow/scripts/test_native_fullsend_eval_ci.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've found 1 issue

Prompt for AI Agents

Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path=".github/workflows/eval-pr-run.yml" line_range="398" />
<code_context>
             find "${workspace}" -type f 2>/dev/null | head -80 || true
+
+            for result in benchmark.json feedback.json summary.md; do
+              if [ ! -f "${workspace}/${result}" ] || [ ! -s "${workspace}/${result}" ]; then
+                echo "::error::Missing or empty required eval result: ${workspace}/${result}"
+                failed=1
</code_context>
<issue_to_address>
**Outside files pass result validation**

When a required result path is a symlink to a nonempty file outside the workspace, `[ -f ]` and `[ -s ]` follow the link, so validation accepts the outside file; the publisher also follows a linked `summary.md` and publishes its contents as eval results, even though no regular result file was written in the workspace.

Reject symlinked result paths and require each required artifact to be a nonempty regular file inside the workspace.

Also at `.github/workflows/eval-pr-run.yml:399`.
</issue_to_address>

Sourcery assessment

Approval pending. 1 finding to address first.

Blocking findings: .github/workflows/eval-pr-run.yml:398

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

[mrizzi]

[sdlc-workflow/verify-pr] Re: @sourcery-ai review — Classified as code change request — sub-task TC-6774 created to address this feedback (the symlink-following result validation at .github/workflows/eval-pr-run.yml:398: reject symlinked result paths and require each required artifact to be a nonempty regular file inside the workspace).

[mrizzi]

Verification Report for TC-6773 (commit 409f1e0)

Check	Result	Details
Review Feedback	WARN	1 code change request (Sourcery symlink-validation) → sub-task TC-6774 created
Root-Cause Investigation	DONE	implement-task skill gap → root-cause task TC-6775 created
Scope Containment	PASS	Exactly the 2 task-specified files; none out-of-scope, none missing
Diff Size	PASS	2 files, +86 lines — proportionate
Commit Traceability	PASS	Commit 409f1e0 references TC-6773 (Implements TC-6773)
Sensitive Patterns	PASS	No secrets/credentials in added lines
CI Status	PASS	All 5 checks green (Skill Lint, Plugin Validation, Eval PR Run, Trigger Eval Dispatch, Sourcery)
Acceptance Criteria	PASS	5 of 5 criteria met
Test Quality	PASS	Repetitive tests PASS; test docs PASS; Eval Quality: N/A
Test Change Classification	ADDITIVE	Append-only: +1 helper, +3 tests, +14 parametrized cases, 0 removals
Verification Commands	PASS	pytest 125 passed; claude plugin validate passed; git diff --check clean (skillsaw blocked by local sandbox cache perms, green in CI Skill Lint)

Overall: WARN

The one actionable review finding — the Sourcery symlink-following validation gap at .github/workflows/eval-pr-run.yml:398 ([ -f ]/[ -s ] follow symlinks, so a required artifact symlinked to a nonempty file outside the workspace would pass) — is tracked in sub-task TC-6774, with systemic root-cause improvement TC-6775 (implement-task boundary-containment verification method). All deterministic guardrails pass and the core fix meets every acceptance criterion. This skill does not merge the PR — a human reviewer decides.

---

This comment was AI-generated by sdlc-workflow/verify-pr v0.13.9.

[sourcery-ai]

Sourcery assessment

Approved.

[mrizzi]

Verification Report for TC-6773 (commit b512bc0)

Check	Result	Details
Review Feedback	WARN	1 code change request (Sourcery symlink-validation) tracked in sub-task TC-6774; already fixed in this PR by commit 9cdcfe25. No new feedback since last run.
Root-Cause Investigation	N/A	No new sub-tasks created this run (root-cause TC-6775 was created in the prior run)
Scope Containment	PASS	Exactly the 2 task-specified files (eval-pr-run.yml, test_native_fullsend_eval_ci.py); none out-of-scope, none missing
Diff Size	PASS	2 files, ~+100 lines — proportionate to the fix + regression coverage
Commit Traceability	WARN	2 of 3 commits reference TC-6773; commit 9cdcfe25 references child sub-task TC-6774 (the symlink fix), traceable within the TC-6773 lineage
Sensitive Patterns	PASS	No secrets/credentials/keys in added lines
CI Status	PASS	All 4 checks green (Skill Lint, Plugin Validation, Trigger Eval Dispatch, Sourcery); eval-execution jobs correctly skipped (PR changes workflow/tests, not tested skill inputs)
Acceptance Criteria	PASS	5 of 5 criteria met
Test Quality	PASS	Repetitive tests PASS; test docs PASS; Eval Quality: N/A
Test Change Classification	ADDITIVE	Append-only: +1 helper, +3 tests, +17 parametrized cases, no reductive signals
Verification Commands	PASS	pytest 129 passed; git diff --check clean (skillsaw/claude plugin validate blocked by local sandbox cache perms, green in CI)

Overall: WARN

The core fix is sound and complete: the ordinary eval CI now grants Claude the exact requested workspace (--add-dir "${workspace}"), instructs it to write results there unchanged, and a post-run loop rejects missing, empty, misplaced, or symlinked benchmark.json/feedback.json/summary.md — so an exit-zero run can no longer silently pass with absent or outside results. All 5 acceptance criteria are met, all CI checks are green, no secrets, and the +17 new parametrized test cases are append-only (ADDITIVE).

WARN is driven by two non-blocking items, both already handled:

• Review Feedback — the sole Sourcery finding (symlink-following [ -f ]/[ -s ] validation) is tracked in sub-task TC-6774 and is already fixed in this PR by commit 9cdcfe25 (the -L guard + symlink-* test cases).
• Commit Traceability — commit 9cdcfe25 references the child sub-task TC-6774 rather than the parent TC-6773; still fully traceable within the task lineage.

This skill does not merge the PR and does not transition the issue — a human reviewer decides.

---

This comment was AI-generated by sdlc-workflow/verify-pr v0.13.10.