Scenario library T2a: port the deterministic scorer (shape score + Run-half grader) from the cloud eval harness into packages/cli, with a discrimination self-test

epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

Filing-gate category: ④ coordination child of objectstack-ai/cloud#3600 (scenario-library epic), item T2a. Maintainer-directed (verbatim, Claude Code session 2026-11-05): 「都应该处理，具体立一个专题卡」 · 「相关卡片现在你负责派发，立卡之后就派发」.

Business rationale: inherited from the parent. The scenario library needs a deterministic, open scorer. The maintainer cost rule 「目前阶段尽量不要用云端的token，api的更贵」 rules out LLM judges, so every score must come from code.

Ruling carried

D2 on cloud#3600 (maintainer, verbatim): 「打分器可以迁移到开源项目」. This authorizes moving the deterministic scorer out of cloud's private packages/service-ai-studio/src/evals/ into this repo under Apache-2.0.

Scope

Port the structural scorer from cloud packages/service-ai-studio/src/evals/structural-score.ts. In the source it is self-contained apart from REFERENCE_VALUE_TYPES from @objectstack/spec/data and the expect type.
- Port: EvalGraph, scoreGraph (exact api-names), ShapeExpectation, deriveShape, and scoreGraphShape (name-agnostic).
- The expect input type is field-for-field the one sibling card T1a fixes:
```
expect: {
  objects: string[];
  minFieldsPerObject?: number;
  selectFields?: { object: string; field: string; minOptions: number }[];
  lookups?: { object: string; field: string; to: string }[];
  dashboards?: string[];
  appNavTargets?: string[];
}
```
- Read the source from the cloud checkout at /home/user/cloud if your container has it. If it does not, report blocked naming the file; do not reconstruct the code from memory.
An artifact → EvalGraph adapter. The cloud scorer reads an environment's metadata over HTTP. Here the input is a locally built artifact (dist/objectstack.json). That includes the multi-package shape: collections inside packages[] (ADR-0130). Precedent for reading both shapes: packages/verify/src/artifact-collections.ts.
A generalized Run-half grader, using cloud packages/service-ai-studio/src/evals/ask-accuracy/scorer.ts as the reference. That file is tied to its own dataset and to the hosted-chat transcript, so port the mechanism, not the dataset.
- Input: a question's computed truth, the persona's hidden objects and fields, and an agent turn transcript { replyText, toolCalls: [{ name, input, output }] }.
- Output: { correct, cited, leaks[] }. It is correct when the numbers in the reply match the truth; cited when every figure names its object; and leaks lists hidden fields or values found in the reply or in any tool output, aggregates included.
Discrimination self-test.
- cloud#2146 measured the shape score at 1.00 on all 8 golden journeys, so it could not tell builds apart.
- Pin with tests that a deliberately degraded graph scores strictly lower than the intact one: a missing lookup, an optionless select, a dangling nav target, a dangling dashboard measure.
- Pin that a leaking answer, an uncited answer and a wrong-number answer each fail the matching Run axis.

Constraints

Home: inside packages/cli next to src/lint/metadata-eval.ts (the os lint --eval path). ⛔ No new package-entry export in this card, so the public surface is unchanged (Clause-②: no). If you measure that an export is unavoidable, stop and report it; do not add it.
No new CLI flag here. The CLI wiring (os lint --eval reading scenario cards) is a later sibling card.
Each ported file carries the Apache-2.0 header and a one-line provenance note ("ported from the cloud eval harness under the cloud#3600 D2 ruling"), with no cloud code comments that cite private issue content.

Acceptance

Tests cover the shape score, the exact score, the adapter on both a single-package and a multi-package artifact, and the Run grader's three axes.
The discrimination self-test is green, and each degraded variant scores strictly lower.
No packages/spec change, no public-surface change.

Reader: the epic PM of cloud#3600 dispatches this to an os-dev agent in the same round it is filed, in parallel with T1a (disjoint file surfaces).

Dedup: parent's dedup table. Searching "scoreGraphShape" in this repo returns 0 hits.

Generated by Claude Code

epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

Claim: PM loop round 1
Session: session_01Q4EjjPXwX1YfHyKJUh4agu
Account: hotlong
Branch: claude/issue-30290-scenario-scorer-port
Worktree: objectstack-issue-30290
Domain: domain:server
Seat: epic:#3600#1
File surface: packages/cli/src/lint/** (new scorer + adapter + Run grader files and their tests only), .changeset/30290-scenario-scorer-port.md (stop on breach; explain in the report)
Container & model: M, mode:subagent, model: TIER_DEFAULT (dispatch-gates --tier: "no path-derived mandate"), effort: high
Clause-②: no
Metadata-protocol: none
Responsibility: n/a — not a defect card
Thread-read: none
Premise: holds
Serial constraints cleared: none — the 33 open pm:dispatched cards' claimed file surfaces were read and none touches packages/cli/src/lint, packages/qa or docs/qa/scenarios; sibling #30289/#30290 surfaces are disjoint

Routing audit (maintainer direct-dispatch channel): domain:server + area:devpath + pm:epic were set at filing by the epic PM of objectstack-ai/cloud#3600, not by triage. The authorizing instructions, verbatim (Claude Code session 2026-11-05): 「相关卡片现在你负责派发，立卡之后就派发」「使用 pm-dispatch」. Domain read from the landing paths: packages/cli is server (CLI). Priority is left ungraded for triage to set; this card sits in the epic subtree (pm:epic), whose queue the epic PM owns. Seat-account override: see cloud#3600 comment 6650372672.

wip(cli): scenario scorer port (#30290)

wip(cli): scenario graph test carries the user field's fixed target (#…

chore(cli): changeset for the scenario scorer port (#30290)

triage#1 · os-zhuang · session_01AavokzJ5DndAwitDXvKy4U

Path: none

Triage: first-touch audit · p2 · epic child · `domain:server` kept · 2026-11-05T06:00Z

⛔ Not a claim.

Read fresh. Scenario library T2a: port the deterministic scorer from the cloud eval harness into an open package, under the maintainer's cost rule (no cloud tokens). p2.

Generated by Claude Code

epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu
os-dev-report

{
  "issue": 30290,
  "status": "done",
  "branch": "claude/issue-30290-scenario-scorer-port",
  "pr": "https://github.com/objectstack-ai/objectstack/pull/30332",
  "session": "session_01Q4EjjPXwX1YfHyKJUh4agu",
  "premise_still_valid": true,
  "summary": "Ported the cloud eval harness's deterministic scorer into packages/cli/src/lint, in three files with tests: scenario-score.ts (ScenarioExpectation mirrored field-for-field from the card, EvalGraph, scoreGraph, ShapeExpectation, deriveShape, scoreGraphShape), scenario-graph.ts (artifactToEvalGraph reads dist/objectstack.json, single-package or option-B packages[], through the CLI's existing resolveStackCollection, in spec shapes only, with no alias chains) and scenario-run-grade.ts (gradeRun returns { correct, cited, leaks, problems } over { replyText, toolCalls }). No model call, no CLI flag, no package-entry export. The discrimination self-test reproduced two blind spots of the cloud source, where a degraded build scored 1.00 like the intact one (nav existence checked by name rather than by app nav; a widget bound to a missing dataset passing its measure check). Both are fixed and documented in the module header and the PR body. Zone 2 readings: the CLI already had the packages[] reader (core resolveStackCollection, re-exported by utils/stack-collections.ts), so verify's declaredCollection was not used. The 5 corpus fixtures carry no app, dataset or dashboard, so the self-test adds them and parses each with ObjectStackDefinitionSchema. The landing is exactly the dispatched file surface. Worktree cleanup (rm node_modules, git worktree remove) runs right after this comment.",
  "tests": "At c0a6cf98 (final head), under os-verify-lock: (1) vitest run of src/lint/scenario-score.test.ts, scenario-graph.test.ts and scenario-run-grade.test.ts gave 'Test Files 3 passed (3) · Tests 33 passed (33)'. (2) vitest run --project unit for @objectstack/cli gave 'Test Files 387 passed (387) · Tests 5460 passed (5460)'. The integration tier is declared to CI: no integration file, spawn entry or driver/kernel boot path was touched. (3) pnpm --filter @objectstack/cli typecheck exited 0 ('check:test-typecheck: OK'). (4) eslint --no-inline-config on the new files exited 0. Ablation, done only for the two scorer behaviours this PR changes, committed first and run through scripts/ablation-replace.mjs (anchor hit x1 to x0, restore blob == HEAD eed52bc0d3ca, git diff HEAD empty). No rebuild was needed: the tests import the module source by relative path, not through dist. A1, nav check reverted to cloud semantics: 3 failed / 13 passed, 'dangling nav target' shows 'invoice_with_line_items exact: expected 1 to be less than 1'. A2, measure check reverted to '!ds ||': 1 failed / 15 passed, 'dangling dashboard dataset' shows 'expected 1 to be less than 1'. Clause-② measured after the cli build: the declaration closure of dist/index.d.ts, console.d.ts and hook-body.d.ts holds 39 files, 0 under dist/lint (positive control from dist/lint/scenario-graph.d.ts finds 2). Reading: no.",
  "gates": "dispatch-gates --repo objectstack-ai/objectstack --commands (no paths) derived 59 commands at c0a6cf98. Each ran with its exit code captured before any pipe. --ran with exit codes reported '59 derived famil(ies) accounted for — 57 run, 2 NOT-MEASURED (2 DERIVED from a recorded exit 3)'. 57 exited 0. check-plugin-teardown-shape --self-test first exited 3 (pinned positive-control commit missing on the shallow clone) and exited 0 after 'git fetch --depth=1 origin 621a4876'. NOT MEASURED: check:dual-build-cjs-loads, reason: PREREQUISITE NOT MET (dist/ of 10 packages outside this diff's closure). NOT MEASURED: check:i18n-coverage, reason: PREREQUISITE NOT MET (showcase config imports packages with no build, e.g. service-print). Both are declared narrowings and go to CI. PR check-runs at report time (head c0a6cf98): Lint, Type Check, Check Changeset, Governed Surface Queue Guard, PR-body guards and the rest all in_progress; 18.x upgrade harness, Packed-tarball smoke and the Schedule gate skipped. CI convergence belongs to PM.",
  "dogfood_battery": "n/a — no `!` in the PR title and no (narrowing) arm",
  "mcp_calls": "0",
  "api_writes": "2 — fleet-write relay: pr_create (POST /repos/objectstack-ai/objectstack/pulls, draft forced, read back 9728 bytes sent = stored, assignees none) and this os-dev-report comment (POST /repos/objectstack-ai/objectstack/issues/30290/comments via post-stamped.mjs). git push is not a REST write.",
  "open_questions": [],
  "security_findings": "withheld",
  "out_of_scope_findings": [
    "carrier: cloud · the cloud source (packages/service-ai-studio/src/evals/structural-score.ts) still carries both blind spots this port closes: scoreGraph's nav:TARGET->exists is satisfied by an existing object of that name even when no app navigates to it, and a widget bound to a nonexistent dataset passes its measure check. Measured here by the two ablations (degraded score 1 == intact score 1). Not filed: the defect is in the cloud repo. Dedupe words: structural-score dangling nav, unknown dataset measure, scoreGraph nav exists"
  ]
}

epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

ACCEPT → PR #30332 (席位核对 on the PR, head c0a6cf98). Landing waits for CI on that head.

epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

File-surface amendment for the claim above (patch round 1 of PR #30332):

Added: packages/core/src/security/second-object-read-exposure.pin.test.ts, the CALLERS table only, one new row for packages/cli/src/lint/scenario-graph.ts.
Why: CI Test Core (6/8) on head c0a6cf98 failed in that repo-wide pin: an unclassified caller of referenceTargetOf (job 127088278720).
Expected class: the pin's own not-a-served-read arm. The scorer reads a built metadata artifact offline and serves no record field of any target. The dev confirms this from the code before writing the row.
No other file in packages/core is in scope.
The ACCEPT on PR feat(cli): port the deterministic scenario scorer — shape score, artifact graph reader and Run grader — with a discrimination self-test #30332 (head c0a6cf98) is spent by the head moving; the seat re-reviews the new head.

test(core): classify the scenario scorer's artifact reader in the sec…

epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu
os-dev-report

Addendum, patch round 1 on PR #30332. New head: 2c7d2600 (on top of a merge of origin/main, 7e93dff0).

{
  "issue": 30290,
  "status": "done",
  "branch": "claude/issue-30290-scenario-scorer-port",
  "pr": "https://github.com/objectstack-ai/objectstack/pull/30332",
  "session": "session_01Q4EjjPXwX1YfHyKJUh4agu",
  "head": "2c7d2600",
  "summary": "Classified packages/cli/src/lint/scenario-graph.ts in the CALLERS table of packages/core/src/security/second-object-read-exposure.pin.test.ts, as one not-a-served-read row (the file-surface amendment in comment 6651380112). I checked the code before classifying: artifactToEvalGraph calls referenceTargetOf only to read a field's declared target out of a built metadata artifact. It records the target name in the EvalGraph so the scorer can check that the reference resolves to an object the build carries. It reads no record, mounts no route, and serves nothing of the target to any caller. No other packages/core change. The PR body does not mention this row: suggested seat edit, add one bullet under What changes: 'packages/core/src/security/second-object-read-exposure.pin.test.ts: one CALLERS row classifying scenario-graph.ts as not-a-served-read.'",
  "tests": "Pin, core repo project, file filter (pnpm --filter @objectstack/core exec vitest run --project repo src/security/second-object-read-exposure.pin.test.ts). RED at c0a6cf98: 'every caller of the reference-target arbiter is classified' failed, unclassified 'packages/cli/src/lint/scenario-graph.ts', 1 failed | 3 passed (4), VERDICT command-exit 1. GREEN at 2c7d2600: Test Files 1 passed (1), Tests 4 passed (4), VERDICT command-exit 0. Scorer suites re-run at 2c7d2600 after rebuilding the @objectstack/core closure: Test Files 3 passed (3), Tests 33 passed (33).",
  "gates": "Re-derived at 2c7d2600: 61 families. The 2 that are new against the round-0 list both exited 0: node scripts/check-dev-prereqs.mjs --self-test, and pnpm check:kernel-hook-pairs ('4 dispatched kernel:* hook(s), each pinned'). The other 59 are unchanged from the round-0 run at c0a6cf98 (57 exit 0, 2 NOT MEASURED for prerequisites) and were not re-run locally; CI runs them on the new head.",
  "mcp_calls": "0",
  "api_writes": "1 — this addendum comment via post-stamped.mjs (fleet-write relay). git push is not a REST write.",
  "open_questions": [],
  "security_findings": "withheld",
  "out_of_scope_findings": []
}

epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

ACCEPT (patch round 1) → PR #30332 at head 2c7d2600. Landing waits for CI on that head.

epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

Landed: PR #30332 merged as 31d23c3f, closing this card (Fixes #30290). pm:dispatched removed with the close; domain:*, area:* and pm:epic stay (ownership, not state). Downstream: #30291 still waits on #30289 (its other Blocked-by:).

Navigation Menu

Repository navigation

Scenario library T2a: port the deterministic scorer (shape score + Run-half grader) from the cloud eval harness into packages/cli, with a discrimination self-test #30290

Ruling carried

Scope

Constraints

Acceptance

Triage: first-touch audit · p2 · epic child · `domain:server` kept · 2026-11-05T06:00Z

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Scenario library T2a: port the deterministic scorer (shape score + Run-half grader) from the cloud eval harness into packages/cli, with a discrimination self-test #30290

Description

Ruling carried

Scope

Constraints

Acceptance

Activity

objectstack-fleet commented on Nov 5, 2026

objectstack-fleet commented on Nov 5, 2026

Triage: first-touch audit · p2 · epic child · domain:server kept · 2026-11-05T06:00Z

objectstack-fleet commented on Nov 5, 2026

objectstack-fleet commented on Nov 5, 2026

objectstack-fleet commented on Nov 5, 2026

objectstack-fleet commented on Nov 5, 2026

objectstack-fleet commented on Nov 5, 2026

objectstack-fleet commented on Nov 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Triage: first-touch audit · p2 · epic child · `domain:server` kept · 2026-11-05T06:00Z