[lanyue-llk]

Summary

Add an opt-in --try-layout flag to the runner and Moli cohort command. Remove auto layout selection, task classification, and preliminary qualification.

• Keep layout off by default and finish all three normal attempts for every case.
• With --try-layout, rerun each failed Moli case with layout on for another complete three attempts (or the same configured k).
• Replace a case's original results only when every rerun attempt passes. Otherwise retain the entire original failed case. Never combine passing attempts across the two batches.
• Preserve the original matrix, rerun matrix, separate artifacts, seeds, and hashes. The final matrix keeps the original denominator and repetition count. Reports disclose recovered cases and additional execution cost.
• Keep --moli-layout on available for fixed-layout runs, without an additional rerun.

Validation

Official Moli 1.1.7 on macOS ARM64, six selected regression cases, three attempts per case. These are targeted implementation checks, not a full benchmark score.

Case	Original layout off	Layout-on rerun	Final
v2_diag_dispatchmouse_geometry	0/3	3/3	Pass, replaced
v2_diag_gbcr_fixed	0/3	3/3	Pass, replaced
ab_main_find_text	0/3	3/3	Pass, replaced
ab_main_get_title	3/3	Not rerun	Pass, retained
pw_raw_browser_getversion	3/3	Not rerun	Pass, retained
v4_cdp_emulation_setemulatedostextscale	0/3	0/3	Fail, original retained

• Final result: 5/6 cases; 18 authoritative result rows. The 12 additional physical executions do not expand the denominator.
• Verified sequential and parallel initial runs, default off without recovery, and explicit on without an extra rerun.
• Linux full regression: 612 passed, 10 skipped. Final focused checks: 24 passed, including an additional interrupted-run reporting guard.
• Verified report generation and rejection of a tampered final matrix.

Dependency

Based on #25. This changes the benchmark harness only; no Moli source or binary is modified. Existing historical reports are not rewritten.

[lanyue-llk]

Review: request changes to the automatic layout classifier

[P1] The off branch includes existing tasks whose contract requires real layout. choose_layout() in runner/moli_layout_policy.py only matches a small feature-token and operation-name regex, then treats every unmatched raw CDP/Node task as safe for mock layout. I ran it against the checked-in task definitions: v2_t2_dom_getnodeforlocation is assigned off, even though its DOM.getNodeForLocation step performs a coordinate hit test and the task explicitly grades the resulting backendNodeId. v2_diag_domsnapshot_computed_styles is also off despite grading DOMSnapshot.captureSnapshot layout styles. The Node probe v2_leg_probe_a3_checked_reveal is off despite declaring web.css.rendered_visibility and reading innerText. pw_raw_emulation_devicemetrics likewise ends up off while setting and reading device metrics. These are different forms of the same root problem: the absence of a regex match is not evidence that an inspectable task is nonvisual. The task ID/hash receipt faithfully freezes the wrong assignment, so provenance does not prevent the regression.

Please make off an explicit allowlist of known layout-independent contracts/operations (or encode an authoritative layout requirement in each task contract), and default every unclassified feature, command, or probe script to on. Add regression coverage by classifying the whole frozen corpus and asserting the known geometry/visual tasks stay on, then run the prospective auto versus all-layout comparison before using this as a benchmark policy. A local direct classifier check reproduced the assignments above; 74 relevant unit tests passed, but those tests cover only a handful of selected task IDs. The PR is already marked draft, so this is a pre-merge finding rather than a claim that published results have regressed.

[lanyue-llk]

Addressed in c9cb898. The reported assignments were reproducible: unmatched raw CDP/Node contracts incorrectly fell through to off. Policy v2 now uses a complete contract allowlist for the two metadata reads and defaults all unknown features, operations, scripts, scenes and grading requirements to on.

The full 1,928-task classifier regression now permits only the two metadata probes to run without layout. All four reported visual tasks select on; mutation tests cover future unrecognized contract components. 44 targeted tests passed locally. A prospective official 1.1.7 comparison of the four reported tasks plus the two allowed metadata probes (three attempts per task per mode, 36 calls) had zero verdict differences between auto and all-layout, with actual launch flags verified.

The draft remains a candidate: this targeted regression check does not substitute for the prospective full auto/all-layout/Chromium benchmark required before adopting the policy.