Skip to content

Evals: the two implement-spec cases carry identical copies of the calc-subtract spec #35

Open

Description

@sabind

Before you file

  • I searched the existing issues.
  • I read evals/README.md.

Component

implement-spec (skill)

Behaviour

evals/skills/implement-spec/works-one-wave/files/.claude/specs/calc-subtract/ and evals/skills/implement-spec/works-all-waves/files/.claude/specs/calc-subtract/ are byte-identical copies of the same three spec files (diff -r on 2026-10-16 reports no difference). A change to the calc-subtract spec has to be made in both or the two cases drift apart. Noted as a known gap in the description of PR #25.

Proposed: keep one copy, under evals/fixtures/ beside the calc tree, and have both cases' scaffold.sh copy it in, the way they already call evals/fixtures/setup.sh. Each case's files/ then holds only what is specific to that case, or goes away.

Grading

No grader changes. Both cases keep passing: make eval-skill-implement-spec EVAL_ARGS="--runs 1", with the cost ceiling raised as evals/README.md says. make validate still passes.

Prompt

Unchanged.

Activity

  1. added
    qualityThe behaviour of the skills, agents and rules, and the evals and checks that guard it
    triageNot yet classified by a maintainer
    on Oct 16, 2026
  2. added theissue type on Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    qualityThe behaviour of the skills, agents and rules, and the evals and checks that guard ittriageNot yet classified by a maintainer

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions