Skip to content

apr eval --task: 8 of 10 dispatch arms are reachable but undocumented in --help #2584

Closed as not planned

Description

@noahgift

apr eval --task dispatches 10 named arms plus the default perplexity path. --help
documents two.

Verified at HEAD (4bbfeb0)

Arms at crates/apr-cli/src/dispatch_analysis.rs:1383-1419:

classify  code  compare  contamination  correlation  human  humaneval  mbpp  plan  verify

Help at crates/apr-cli/src/extended_commands.rs:138:

/// Task type: omit for perplexity, "classify" for classification eval

Eight reachable and undocumented: humaneval, mbpp, code, contamination,
compare, verify, correlation, human, plan.

Why it matters

These are not dead arms — they dispatch. A user cannot discover apr eval --task humaneval from the tool, only from the source. For a benchmark command that is worse
than a missing feature: the capability exists, is unreachable by documentation, and its
absence from help implies it is unsupported.

Falsifier

A test that enumerates the dispatch arms at runtime and asserts every one appears in
the help text. The obvious weaker version — a hand-written list of expected arms — would
drift out of date exactly the way this did, and would have passed while eight arms were
undocumented.

Refs #2582

Activity

  1. added this to the 0.69.0 milestone on Sep 10, 2026
  2. added
    bugSomething isn't working
    P1High priority
    on Sep 11, 2026
  3. modified the milestones: 0.69.0, 0.71.0, 0.70.0, 0.73.0 on Sep 17, 2026
  4. noahgift commented on Sep 23, 2026

    @noahgift
    MemberAuthor

    milestone → 0.73.0 per docs/roadmaps/milestone-triage-2026-09-23.md (#4024), operator approval 2026-09-23 (relayed by aprender-cf). Reason: undocumented --task arms: debt ratchet pillar D (#3997), slice 4

  5. noahgift commented on Sep 25, 2026

    @noahgift
    MemberAuthor

    Parked by APR-EPIC-001: over S3 budget (rank 12/142). Reopen = pull. Applied by aprender-d1 under a 3-lane quorum receipt (handoff/balance-quorum-rev4.json).

  6. modified the milestones: 0.73.0, backlog on Sep 25, 2026
  7. added
    parkedClosed over epic budget (APR-EPIC-001 rule 5). Reopen = pull.
    on Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High prioritybugSomething isn't workingparkedClosed over epic budget (APR-EPIC-001 rule 5). Reopen = pull.

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions