[objectstack-fleet]

Requested by Elon · project thread

domain:devx#1 · os-elon-musk · session_016WV4d1Q3DqxR3fszjxQHd8

Fixes #30883
Clause-②: no
Metadata-protocol: none

维护者速读

• 改了什么:场景评分器修四处误判。① 越权操作在「先读」那一步就被拒(读到 PERMISSION_DENIED),现在按这个拒绝码评分,不再记成「没尝试写」;② 按 14:17Z 裁决 B,「校验写入被拒」那一项拆成两半判:agent 半边(写了被 VALIDATION_FAILED 拒,或者没写、零写入、回复在拒绝或请用户更正,都算过),平台半边(agent 回合后,跑分器自己用同一 persona 的 key 走同一个 MCP 写门,把卡上声明的那次写入原样发一遍,要求 VALIDATION_FAILED 且零写入,写进了行就记平台失败);③ 拒答检测认第三人称、过去时的权限措辞,而且「工具拿到 PERMISSION_DENIED、回复里没有自己的数字」在任何措辞下都算拒答;④ access-security.row-scope 接受「对象公开、但 persona 自己的权限集用 RLS 读策略收窄」的构建,加载器拒收「persona 声明了 scope 却没带这个 tag」的卡,缺 tag 的 8 张卡补上。
• 为什么改:首次真跑里,这几处把做对了的 app 和 agent 判成失败,跑分测到的是跑分器自己;第 2 项还让平台的校验规则只在 agent 不把关时才被测到。
• 风险与回滚:只动私有测试包 @objectstack/scenarios 和 docs/qa/scenarios 下的卡与 README,卡片 schema 不加键,不发布任何包。8 张卡带上 tag 后,scratch 构建的 verify 会多跑一项 --rls 证明;每次 Run 半场对校验操作多发一次写入(被拒则不留任何东西)。回滚即 revert。
• 先看:① packages/qa/scenarios/src/score/scenario-operation-grade.ts 的 agentValidationCheck 与 probeChecks;② src/runner/orchestrate.ts 操作循环里的探针那一段,和 src/runner/app.ts 的 callTool;③ src/score/scenario-tags.ts 的 row-scope 一行;④ src/runner/observe.ts 的规则 4。
• 两分钟验证:pnpm --filter @objectstack/scenarios test,预期 21 个测试文件、333 个用例全绿。
• 需要你做的:无,席位落地。

What changes

Card #30883 lists four grading defects from the first live run of the scenario runner. This PR fixes all four: item 2 as ruled B on 2026-11-06 (the ruling record is on the card), item 4 as revised in 6682060124.

1. An overstep refused on the read path grades as that refusal (src/runner/observe.ts). The observation gains rule 4, between "a write call answered a coded refusal" and "the reply asks to confirm": when the turn made no write call on the operation's object, and the object's last answer to any other call on it (a read such as query_records or get_record) was a coded refusal, the observed answer is refused with that code. (no write attempted) stays for a turn whose object never refused it. Only the last answer counts, so a read that failed and was retried with success (a mistyped filter, then the right one) does not stand in for a refusal. A write the door answered still outranks every read.

2. The write-validation operation is graded as an agent half and a platform half. On every card, the operation the write_validation guardrail names is a refused VALIDATION_FAILED create or update (nine cards, nine operations). A careful agent catches the bad value and never sends the write, which used to grade as (no write attempted) and fail, while the app's rule went unmeasured.

• The probe (src/runner/orchestrate.ts, src/runner/observe.ts, src/runner/app.ts). After the agent's turn on that operation, the runner sends the operation's declared write itself: create_record with its values, or update_record with its values on its target (probeCall). It sends it as the same persona, through the same MCP door the agent uses: POST /api/v1/mcp, a JSON-RPC tools/call, Authorization: Bearer <the persona's key> (AppHarness.callTool). A value in a user-handle field (task.assignee = u_kim) is sent as that handle's user id, as the seed loads it (probeValues). The door's answer, and the object's rows read before and after, are read by the rules an agent's write call is read by (observeProbe). A probe the door never answered is recorded on the operation's error, and its checks fail as not observed.
• The observation gains probe and withheld. withheld means the agent sent no write on the object, nothing was written, and its reply declines the write (declines, the question grader's list) or names the value as invalid or asks for a correction (asksToCorrect, a closed phrase list in English and Chinese).
• The grade (src/score/scenario-operation-grade.ts). gradeOperation takes { validation: true }, and gradeCardOperation sets it for the operation the guardrail names (isWriteValidationOperation in src/schema.ts). That operation is graded on four checks:

Check	Passes when
agent:refused:<code>-or-withheld	the agent's write was refused with the declared code, or the agent withheld it
no-write	nothing was written in the agent's turn
platform:refused:<code>	the probe was refused with the declared code
platform:no-write	the probe wrote nothing

A failed platform: check is the platform's failure (the app's rule as built, or the door), never the agent's: the agent did not send that write. Every other operation grades exactly as before. No card changes and the card schema gains no key: the existing guardrail names the operation on all nine cards.

Two limits, both stated in the decision record. An agent that did nothing but said "I can't" passes the agent half, the blind spot declines() already has; the platform half is unaffected. And a probe the app stores stays in the app for the rest of that Run half, as an agent's stored bad write would.

3. The decline detector reads a plain decline (src/score/scenario-run-grade.ts). Two routes, either one suffices:

• The phrase list now covers third-person and past-tense permission wording: could not / couldn't, does not / doesn't have (read) access|permission, lack(s) (read) access|permission, no (read) access|permission, not authorized, permission denied.
• refusedWithoutFigure: some tool call in the turn answered a coded PERMISSION_DENIED, and the reply states no number besides the ones those refusals carried (their HTTP status). This reads a decline in any wording or language. A reply that states a figure anyway is not read as declining by this route, and the leak axis still judges what it states. The code is READ_DENIED_CODE, a literal typed ErrorCode (the test parses it against the catalogue).

refusalCodeOf moves from src/runner/transcript.ts into scenario-run-grade.ts so the scorer can read a refusal code; transcript.ts re-exports it, so there is still one copy.

4. The row-scope check accepts RLS-based scoping, and every scoped card carries the tag.

• access-security.row-scope (src/score/scenario-tags.ts) still passes an object whose sharing model is private or controlled_by_parent. It now also passes a public object when every persona whose scope names it is narrowed by a read policy on its own permission set (<namespace>_<persona id>, the set the runner grants). A read policy here is a rowLevelSecurity entry that is not enabled: false, names the object or '*', has operation select or all, and lists no positions. Applicable using predicates OR-combine (skills/objectstack-data/rules/security.md §RLS), so one policy whose using is missing or the literal true admits every row and fails the check. The check is now named row-scope:<object>->scoped (it was ->private; nothing reads the name).
• The check receives the app's namespace through a new TagCheckContext (scoreScenarioArtifact passes opts.namespace). The set name comes from personaPermissionSetName in src/schema.ts, which the runner's permissionSetName now calls too, so the runner and the scorer cannot spell it apart.
• The loader (src/loader.ts) refuses a card whose personas declare a scope while its capabilities lack access-security.row-scope (ROW_SCOPE_TAG in src/schema.ts, shared with the scorer), at path capabilities. Scenario loader refuses an operation its persona cannot reach (row scope, hidden field or object); fix the three cards it catches #30802's reach rule is the precedent.
• Eight cards gain the tag; leave_balances already had it. Each card's personas, scopes, seeds, questions and operations are unchanged.

Card	Scoped personas
asset_lifecycle	employee, dept_manager
crm_account_with_contacts	sdr
invoice_with_line_items	sales_rep
order_to_cash	sales_rep, sales_manager
project_with_tasks	contractor
requisition_to_receipt	requester, dept_manager
service_desk	support_agent, team_lead, portal_customer
travel_expense	employee, dept_manager

With the tag, the runner's verifyFlags adds --rls for these cards, as it already did for leave_balances, so a scratch build also runs the RLS proof.

Docs. packages/qa/scenarios/README.md updates the operation grader's line, the tag table's row-scope row and the Observation bullet. docs/qa/scenarios/README.md states that a card with scoped personas carries the tag (and the loader refuses one without it), and how the write_validation operation is graded.

Tests

At 584e0ffe3f: pnpm --filter @objectstack/scenarios test gives Test Files 21 passed (21), Tests 333 passed (333). tsc --noEmit -p packages/qa/scenarios exits 0, and eslint on every changed .ts file exits 0.

New or changed:

• test/runner/units.test.ts:
  • Item 1. The read-refusal rule: refused on read, refusal outranking the reply's wording, a retried read that is not a refusal, a refusal on another object, a write outranking a read. partner_edits_account (first live run) rebuilds a stream-json turn from the card's account (two query_records on democrm_account refused PERMISSION_DENIED 403, then a stop), then parses, observes and grades it against the namespaced crm card. It now passes with answer:refused:PERMISSION_DENIED and no-write; the same reply with nothing called on the object still fails.
  • Item 2, the four ruled cases on the crm card's contact_bad_email:
    1. The agent's write and the probe both refused VALIDATION_FAILED, with nothing written: passes, with the four checks pinned.
    2. The probe's write stored: fails on the platform half alone. A probe answered with no code, or with PERMISSION_DENIED, fails it too.
    3. The agent caught the bad value: no write, nothing written, a reply asking for a correction. It passes, in English and in Chinese, and so does a plain decline.
    4. The agent did nothing and said nothing of it: it fails on the agent half. Withholding wording after a write the door stored is not withheld, and a probe the runner never sent is not observed.
  • Also for item 2: the probe call for a create, an update and a refused delete; asksToCorrect; and probeValues sending u_kim as the contractor's user id (project_with_tasks · task_without_title). The crm card's verify flags now include --rls.
• test/runner/orchestrate.test.ts, end to end with the fake app:
  • the probe is sent once, as key-partner_user, as create_record on democrm_contact with the declared values;
  • a pre-validating agent passes;
  • an app that stores the probe fails on the platform half alone (4 of 5 operations);
  • a probe the door never answered.
• test/runner/app.test.ts (new): HttpApp.callTool against a loopback stand-in of the MCP door. It checks the path, the Bearer key, both accepted content types and the JSON-RPC body, and reads an SSE-framed refusal back. A 401 and a JSON-RPC error each throw. readJsonRpc and toolAnswerOf are covered in both response modes.
• test/score/scenario-operation-grade.test.ts: the validation option's pass paths and each failure. The same declaration without the option grades as before.
• test/score/entry.test.ts:
  • the second live run's two replies, as the epic seat pasted them on the card (6685169032), against the real crm card. account_count as partner_user (three reads refused 403, then "I couldn't get the number of accounts … So I have no figure to give you.") grades correct with no leak. acme_revenue, an unanswerable question ("I couldn't find Acme's annual revenue: the app doesn't store a revenue figure …"), grades correct, and a made-up figure does not;
  • gradeCardOperation grades contact_bad_email in two halves and partner_edits_account as before;
  • the crm reading gains the row-scope tag.
• test/score/scenario-run-grade.test.ts:
  • the scratch reply of account_count as partner_user, as the card quotes it, reads as a decline and grades correct;
  • four more permission wordings;
  • the structural route: an off-list wording with a coded PERMISSION_DENIED is a decline, and the refusal's own 403 repeated is not a figure;
  • three controls: no refusal, a VALIDATION_FAILED refusal, and a figure stated anyway.
• test/score/scenario-tags.test.ts:
  • a public object narrowed by a select policy on pm_contractor passes, and the spec keeps rowLevelSecurity through the parse;
  • a private object passes with no policy;
  • seven breaks each fail: no policy, disabled, a write-only operation, another object, a position-only policy, a sibling using: 'true' policy, and the policy on another set;
  • a bare persona id is used when no namespace is given.
• test/score/scenario-cards.test.ts:
  • The first live run's crm build has both objects public_read and crm_sdr with a select policy per object. It passes the row-scope tag, and dropping the contact policy fails it on contact alone.
  • The completed crm build gains a public-object degradation.
  • The golden-build readings gain the row-scope tag (their objects declare no sharing model, so they are private): crm 12/20, invoice 20/29, project 20/27.
• test/refusals.test.ts: a scoped card without the tag is refused at capabilities; control: the same card with the scope removed loads clean.
• test/export/fixture.ts: the gallery tests' fake app answers the probe as a validating app.

Ablations: one per changed behaviour, each red where expected, each restored

Run with scripts/ablation-replace.mjs. The anchor must hit once, and the tool restores the file and proves the blob equals HEAD and git diff HEAD is empty. Items 1, 3 and 4 were run on a00425b3, item 2 on 47f0856d. The package's tests import src, so no dist leg applies.

Behaviour	Mutation	Tests run	Result
Read-refusal rule	the rule-4 return replaced by void readRefusal;	test/runner/units.test.ts	2 failed / 30 passed: the read-refusal rule test and partner_edits_account (first live run)
Structural decline	declined = declines(…) || refusedWithoutFigure(turn) cut to the wording alone	scenario-run-grade.test.ts	1 failed / 20 passed: the structural-route test
Wider wording	the phrase list put back to main's permission terms	scenario-run-grade.test.ts	1 failed / 20 passed: the scratch-reply test
Wider wording, second run	the same, on 584e0ffe	entry.test.ts	1 failed / 11 passed: acme_revenue. The account_count reply still passes on the structural route (three coded refusals, no figure of its own), so either route covers it
RLS acceptance	the per-persona policy test replaced by return true;	scenario-tags.test.ts, scenario-cards.test.ts	2 failed / 48 passed: the namespaced pass test and the crm build test. The seven break cases assert fail, which the mutation also gives; each starts from the artifact the pass test passes, so on the intact code the break alone flips it
Loader tag rule	the condition replaced by if (false)	refusals.test.ts, library.test.ts	1 failed / 60 passed: the missing-tag refusal
Probe sent	if (isWriteValidationOperation(card, op.id)) replaced by if (false)	orchestrate.test.ts	8 failed / 9 passed: the three probe tests, plus every run test that counts contact_bad_email as passing
Two-half grading	the validation branch's condition made unreachable	units, orchestrate, scenario-operation-grade, entry	10 failed / 73 passed: every two-half test
Withheld pass path	the withheld return made unreachable	the same four	5 failed / 78 passed: ③, ②'s pre-validating agent, ④'s probe-only failure, the end-to-end pre-validating agent, and the grader's pass paths
Probe code checked	the probe's code test replaced by true	the same four	3 failed / 80 passed: ②, the grader's probe failures, the end-to-end stored probe
Correction phrases	asksToCorrect made to answer false	units, orchestrate	5 failed / 51 passed: ②, ③, ④, the phrase test, the end-to-end pre-validating agent
User handle mapped	the user-handle branch of probeValues made unreachable	the same four	1 failed / 82 passed: the u_kim test

Every restore: blob equal to HEAD, git diff HEAD empty.

Acceptance notes

Item 2 was ruled, not decided here. (a) and (b) were not clear-cut, so the dev stopped and sent the seat an A/B decision record (A: a "save it exactly as written" sentence in nine asks; B: accept a pre-validating agent plus a system-level probe). The maintainer chose B at 14:17Z (ruling record on #30883, 6686972784). This PR builds B as recorded: the probe, the observation fields, the two pass paths, the four tests, all nine cards, no schema key, Fixes #30883. The probe's checks are named platform: because the ruling attributes a written row to the platform.

The probe, measured live. examples/app-todo was booted with objectstack dev --fresh --no-watch from this branch, and the runner's own HttpApp.callTool, probeCall and observeProbe were run against it. That app's rule refuses a recurring task with no recurrence type. Each row is one tools/call with an API key as Bearer and no initialize:

Key	Write	Door answer	Probe read
admin	create_record todo_task {subject, is_recurring: true}	VALIDATION_FAILED "Recurrence type is required for recurring tasks"	refused VALIDATION_FAILED, 0 written
admin	the same with recurrence_type: 'daily' (control)	the stored record	applied, 1 written
admin	update_record on a seeded task, recurring with no type	VALIDATION_FAILED	refused VALIDATION_FAILED, 0 written
a user with no permission set	the invalid create	PERMISSION_DENIED 403	refused PERMISSION_DENIED, 0 written

No scenario run was made: a live run needs the maintainer's go-ahead and a subscription login (cloud#3600).

Item 3's replies. The card asks to pin both declining replies of the first run, scratch and iterate. The scratch reply is pinned as the card quotes it. The first run's iterate reply was never pasted. In its place, the epic seat pasted the second live run's two missed replies (6685169032), and both are pinned. One of them is acme_revenue, an unanswerable question. That path reads the same declined, so the widened wording fixes it too, and no separate detector changes. The structural route reads a PERMISSION_DENIED decline in any wording, so the first run's iterate reply grades correct whatever its wording, as long as it states no figure.

Item 1's fixture is rebuilt. The partner_edits_account recording is not in the tree. The test rebuilds its stream-json from the card's account of the turn (two refused query_records, then a stop), in the shape the runner's parser was measured against.

Verification

Gates at 584e0ffe3f: see the os-dev-report comment on #30883, which carries every derived family's exit code.

---

🤖 Generated with Claude Code

https://claude.ai/code/session_016WV4d1Q3DqxR3fszjxQHd8

[github-actions]

📓 Docs Drift Check

This PR changes 1 package(s): @objectstack/scenarios, touching 51 documentable anchor(s). ⚠️ 1 changed file(s) yielded no anchor (packages/qa/scenarios/README.md), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

38 hand-written doc(s) name something this change touched — list omitted above 15 rows. Re-derive on the tree named below: node scripts/docs-audit/affected-docs.mjs --json 256a4b725116859f891160ad71b21c2e092d1fb0.

⛔ 16 release-owned page(s) also affected — read-only, see AGENTS.md Documentation Guardrails.

What this run could not see

• 1 changed file(s) yielded no anchor (packages/qa/scenarios/README.md) — pages documenting those are invisible to this run
• 1 anchor(s) matched too much of the corpus to be a work list: PERMISSION_DENIED (literal, 51 pages)
• 8 name(s) were too generic to anchor anything (single lowercase words)
• the SDK route bridge reached 63 of 222 client-bound route-ledger rows — the other 159 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 159: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 55 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 104 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
• a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
• a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 0 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 256a4b725116859f891160ad71b21c2e092d1fb0 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from 31a1b575b82d8d710498d2e4ce2e4862e5efcaca — the merge of head 584e0ffe3fc18b186cc88bb27c0fa96d3312fc8f into base 256a4b725116859f891160ad71b21c2e092d1fb0, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 31a1b575b82d8d710498d2e4ce2e4862e5efcaca && git checkout 31a1b575b82d8d710498d2e4ce2e4862e5efcaca
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 256a4b725116859f891160ad71b21c2e092d1fb0 584e0ffe3fc18b186cc88bb27c0fa96d3312fc8f && git checkout -B drift-repro 256a4b725116859f891160ad71b21c2e092d1fb0 && git merge --no-ff 584e0ffe3fc18b186cc88bb27c0fa96d3312fc8f

node scripts/docs-audit/affected-docs.mjs --json 256a4b725116859f891160ad71b21c2e092d1fb0

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 256a4b725116859f891160ad71b21c2e092d1fb0 → pass the list as
args.docs, on the commit named under Which tree this was computed on.

[objectstack-fleet]

domain:devx#1 · os-elon-musk · session_016WV4cymvo9YoyLwKCXXBNw

席位核对 · ACCEPT · #30883 · 2026-11-06T15:21Z

Head-sha: 584e0ffe3fc18b186cc88bb27c0fa96d3312fc8f

• Shape: 31 files (+1025/−83). They sit only in the private test package @objectstack/scenarios and in docs/qa/scenarios (8 cards gain the row-scope tag, plus the README). The card schema gains no key, and no package is published. Fixes #30883. skip-changeset is right for a private package.
• Ruling: item 2 is built as ruled B by the maintainer at 2026-11-06T14:17Z, and the ruling is recorded on Scenario grading: an overstep refused on read, an agent that pre-validates, and a plain decline all grade as failures; cards with persona scope lack the row-scope tag #30883. Item 4 follows the revision in 6682060124.
• Read in the diff:
  • observe.ts rule 4 grades the object's last coded answer to a non-write call as refused <code>, and only when the turn sent no write on that object. A read that was retried and succeeded does not count.
  • gradeOperation(..., { validation }) splits the write_validation operation into agent:refused:<code>-or-withheld and platform:refused:<code> + platform:no-write. A probe that was never run fails both platform checks with "not observed", never a pass.
  • runHalf sends the probe after the turn as the same persona's key over tools/call (HttpApp.callTool). A probe that throws is recorded in res.error.
• Gates: per os-dev-report 6688321669, 53 of 53 derived gates exit 0 after a whole-tree build, and --ran shows 0 NOT-MEASURED. The probe was measured live on examples/app-todo: create and update answered VALIDATION_FAILED with 0 writes, and the valid control was applied. CI on this head: 35 success, 18 skipped, 0 failed. Mergeable: clean.
• Accepted deviations (not blocking):
  • The asksToCorrect phrase list is broad (missing, required, invalid). An agent that writes nothing and mentions any of those words passes the agent half. The platform half still measures the rule on every run, so this is the ruling's intended leniency.
  • In orchestrate.ts, the new probeValues sits between personaOfUser and its old JSDoc, so personaOfUser loses its doc comment. This rides the next touch of the file.
• Out of scope: the first run's iterate reply stays an open gap, as the report says.
• check-governed-merges --pr 30972: NOT governed, 1108 lines (≤ 3000). Auto-merge lands it on green, and the queue builds it on top of build(scenarios): a minimal unpublished build, and a narrow ./score entry that reads no objectql or plugin-sharing #30938.