Skip to content

feat(task): add structured validation runner - #250

Merged
BobbyZhouZijian merged 2 commits into
Human-Agent-Society:devfrom
steven-ji:feat/structured-validation-runner
Aug 23, 2026
Merged

feat(task): add structured validation runner#250
BobbyZhouZijian merged 2 commits into
Human-Agent-Society:devfrom
steven-ji:feat/structured-validation-runner

Conversation

@steven-ji

Copy link
Copy Markdown
Contributor

Motivation

Issue #242 proposes frontend-independent task validation that can be shared by
the CLI, a future TUI, and future web clients. PR #249 introduced structured
static diagnostics, but the full validation workflow—workspace preparation,
grader environment setup, grader loading, and baseline grading—still lived in
cmd_validate() and reported runtime failures only through CLI control flow.

This PR completes a focused part of roadmap phase 1 by adding a reusable
validation runner with structured progress and failure data.

Refs #242.

Changes

  • Add serializable ValidationProgressEvent, ValidationFailure, and
    ValidationRunResult types to coral.task.validation and export them from
    coral.task.
  • Add run_validation() to run structural checks, prepare an isolated
    workspace and private data, set up/load the grader, and grade the baseline.
  • Emit ordered started / completed / failed progress events through both
    a callback and the final result.
  • Return structured failures for static checks, task reloads, temporary
    workspace preparation, grader environment setup, grader loading, and
    baseline crashes.
  • Make coral validate a presentation adapter over the shared runner while
    preserving its successful output and static-error wording.
  • Add regression coverage for success, event order, serialization, every
    failure stage, public exports, and CLI compatibility.
  • Update CLAUDE.md to describe the expanded coral/task/ responsibility.

This PR does not add coral validate --json, a TUI, a TUI framework
dependency, or persistent TaskDesign metadata.

Test plan

  • uv run pytest tests/ -v — 711 passed, 1 skipped.
  • uv run ruff check . — passed.
  • uv run ruff format --check . — 139 files already formatted.
  • uv run mypy --follow-imports=skip coral/task/validation.py coral/cli/author.py
    — passed.
  • uv run coral validate examples/dna_designValidation: OK, 100/100
    sequences valid, and a finalized numeric baseline score.
  • git diff --check upstream/dev...HEAD — passed.

Affected areas

  • coral/agent/ (runtime, manager, heartbeat, warmstart)
  • coral/grader/ (daemon, TaskGrader, subprocess grader, loader)
  • coral/hub/ (attempts, notes, skills, checkpoint)
  • coral/workspace/ (project setup, worktrees, grader env)
  • coral/cli/ (commands, helpers)
  • coral/hooks/ (post_commit / submit_eval)
  • coral/template/ (CORAL.md, bundled skills/agents)
  • coral/gateway/ (LiteLLM gateway)
  • coral/web/ (dashboard)
  • examples/ (new or modified task)
  • docs/ (docs site)
  • CI / tooling / packaging
  • Other: frontend-independent task validation API and developer guidance

Checklist

  • PR targets the dev branch (not main).
  • Title follows Conventional Commits (feat:, fix:, docs:, refactor:, ...).
  • uv run pytest tests/ -v passes locally.
  • uv run ruff check . and uv run ruff format --check . pass.
  • Added or updated tests under tests/ for any behavior change.
  • Updated docs (docs/content/, README, or relevant skill under .claude/skills/) for any user-visible or contract change.
  • If this changes a config field, CLI flag, hook, or runtime contract — noted the migration path in the PR description. (Not applicable; no such contract changes.)
  • New examples/<task>/: coral validate <task> succeeds and a smoke run produces at least one finalized score. No hidden answer keys committed under seed/. (Not applicable; no new example.)
  • AI-assisted PR: a human author has read every changed line and can defend the design. See AGENTS.md.

Authored with assistance from Codex. The human author reviewed every changed
line and confirmed the change is ready for maintainer review; the checks listed
above were run locally with Codex assistance.

@vercel

vercel Bot commented Aug 22, 2026

Copy link
Copy Markdown

@steven-ji is attempting to deploy a commit to the Compounding Intelligence Team on Vercel.

A member of the Team first needs to authorize it.

@BobbyZhouZijian

Copy link
Copy Markdown
Collaborator

@steven-ji thanks for the PR! Looks good overall. Could you add an sync run_validation_async() before merging? The current implementation uses asyncio.run(), which fails when called from an existing event loop, e.g. the expected environment for the future TUI/web clients.

@steven-ji

Copy link
Copy Markdown
Contributor Author

Implemented in 50fe9fe.

  • Added public async run_validation_async() for callers already inside an event loop.
  • Kept run_validation() as the synchronous wrapper, so the existing CLI behavior is unchanged.
  • Exported both entry points from coral.task.
  • Added regression coverage for running the async API in an existing event loop and for the public export.

Validation:

  • uv run pytest tests/ -q — 712 passed, 1 skipped
  • uv run ruff check .
  • uv run ruff format --check .
  • uv run mypy --follow-imports=skip coral/task/validation.py coral/cli/author.py
  • uv run coral validate examples/dna_design — Validation: OK, 100/100 sequences valid

@BobbyZhouZijian BobbyZhouZijian left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm.

@BobbyZhouZijian
BobbyZhouZijian merged commit 6523a8c into Human-Agent-Society:dev Aug 23, 2026
5 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants