feat(task): add structured validation runner - #250
Merged
BobbyZhouZijian merged 2 commits intoAug 23, 2026
Merged
Conversation
|
@steven-ji is attempting to deploy a commit to the Compounding Intelligence Team on Vercel. A member of the Team first needs to authorize it. |
Collaborator
|
@steven-ji thanks for the PR! Looks good overall. Could you add an sync |
Contributor
Author
|
Implemented in 50fe9fe.
Validation:
|
BobbyZhouZijian
merged commit Aug 23, 2026
6523a8c
into
Human-Agent-Society:dev
5 of 7 checks passed
22 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Issue #242 proposes frontend-independent task validation that can be shared by
the CLI, a future TUI, and future web clients. PR #249 introduced structured
static diagnostics, but the full validation workflow—workspace preparation,
grader environment setup, grader loading, and baseline grading—still lived in
cmd_validate()and reported runtime failures only through CLI control flow.This PR completes a focused part of roadmap phase 1 by adding a reusable
validation runner with structured progress and failure data.
Refs #242.
Changes
ValidationProgressEvent,ValidationFailure, andValidationRunResulttypes tocoral.task.validationand export them fromcoral.task.run_validation()to run structural checks, prepare an isolatedworkspace and private data, set up/load the grader, and grade the baseline.
started/completed/failedprogress events through botha callback and the final result.
workspace preparation, grader environment setup, grader loading, and
baseline crashes.
coral validatea presentation adapter over the shared runner whilepreserving its successful output and static-error wording.
failure stage, public exports, and CLI compatibility.
CLAUDE.mdto describe the expandedcoral/task/responsibility.This PR does not add
coral validate --json, a TUI, a TUI frameworkdependency, or persistent
TaskDesignmetadata.Test plan
uv run pytest tests/ -v— 711 passed, 1 skipped.uv run ruff check .— passed.uv run ruff format --check .— 139 files already formatted.uv run mypy --follow-imports=skip coral/task/validation.py coral/cli/author.py— passed.
uv run coral validate examples/dna_design—Validation: OK, 100/100sequences valid, and a finalized numeric baseline score.
git diff --check upstream/dev...HEAD— passed.Affected areas
coral/agent/(runtime, manager, heartbeat, warmstart)coral/grader/(daemon, TaskGrader, subprocess grader, loader)coral/hub/(attempts, notes, skills, checkpoint)coral/workspace/(project setup, worktrees, grader env)coral/cli/(commands, helpers)coral/hooks/(post_commit / submit_eval)coral/template/(CORAL.md, bundled skills/agents)coral/gateway/(LiteLLM gateway)coral/web/(dashboard)examples/(new or modified task)docs/(docs site)Checklist
devbranch (notmain).feat:,fix:,docs:,refactor:, ...).uv run pytest tests/ -vpasses locally.uv run ruff check .anduv run ruff format --check .pass.tests/for any behavior change.docs/content/, README, or relevant skill under.claude/skills/) for any user-visible or contract change.examples/<task>/:coral validate <task>succeeds and a smoke run produces at least one finalized score. No hidden answer keys committed underseed/. (Not applicable; no new example.)Authored with assistance from Codex. The human author reviewed every changed
line and confirmed the change is ready for maintainer review; the checks listed
above were run locally with Codex assistance.