Calibrate a small outcome and human-effort score on completed runs #49

Current triage — 16 October

Not planned now. Keep reporting verified product outcomes, necessary human help and rework in ordinary handoffs. A separate scoring exercise does not unblock the current repairs. Revisit after a repeated product-delivery failure yields a specific improvement question and a small existing sample; do not add a grader to every run.

A merged PR or a completed agent turn does not show whether the owner had to catch routine omissions or repair the workflow. Factory's vision already names useful outcomes, human effort, rework and usage. Turn that into one repeatable assessment using existing evidence, without adding a grader to every run.

Acceptance

• Select a small sample of completed work containing success, assisted recovery and an unsuccessful or inconclusive path. Reuse the application evidence in Prove a complete delivery on an owner-selected application #4 and the lead/recovery evidence in Prove Agent Ops across idea research, assignments and accepted backlog #3 and Follow blocked work through to verified progress #42 where sufficient. #48 supplies the available data and its limits; do not recollect unchanged evidence.
• Define a compact rubric before grading: whether the accepted outcome was verified, whether avoidable human intervention was needed, and what rework occurred. Record elapsed time and usage only where supported. Separate legitimate product/authority decisions, avoidable coordination and native capability failures.
• Bind every verdict to concrete evidence. Missing observations are inconclusive, not zero cost or full autonomy. Check the rubric with an independent reader on the same sample and retain disagreements or cases it cannot classify.
• Produce one small results table and identify at most one useful improvement hypothesis, including its alternative explanation. The rubric must distinguish a resolved status/card from actual useful worker progress. Prefer sampling; record evaluation effort so measurement itself does not become disproportionate.

Owner: AgentOps. #8 already established the observed-improvement process; use its existing review/adoption route for any later candidate. A score neither changes task acceptance nor grants delivery authority.

Sources: The software factory stack — Improvement, Zach's demonstration around 13:29–15:23.