Skip to content

Close verification gaps between reviewed source, delivered artifact and native acceptance #76

Open

Description

@gustavonline

Reviewed source and passing CI have still left the owner discovering functional failures in delivered plugins. Investigate where current Factory ADLC checks lose the connection between the requested outcome, reviewed source, shipped artifact, active host version and actual user workflow. AgentOps must not report product completion while required acceptance evidence is missing.

Observed evidence

These are verification and delivery escapes, not evidence of a security/data leak or proof that every review failed. Plugin lifecycle and native UI limitations explain some incomplete observations, but do not excuse missing production-package checks. Private receipts remain with the existing leads; do not publish host paths, logs, credentials or owner screenshots.

Bounded investigation and correction

  1. Trace the actual handoffs through specification, implementation, tests/evals, independent review, packaging, installation and native acceptance. Identify which check could have caught each observed failure, its responsible actor and what evidence was missing. Distinguish product defects, test gaps, host limitations and inaccurate completion reporting.
  2. Define the smallest sufficient acceptance evidence for the owner's promised workflow and the exact delivered artifact. Source/CI, package completeness, active host binding and real end-to-end behaviour are separate claims. NOT RUN, a blocked native step or an acknowledged handoff must not become a green result.
  3. Make AgentOps reconcile actual child outcomes and outstanding acceptance conditions before reporting done. It should own supported verification work and arrange one concrete owner-only checkpoint only where native access genuinely requires the owner. A source-ready or installed-but-unaccepted result must be labelled accordingly. “All green” means all applicable agreed acceptance conditions have evidence for the delivered candidate, not just a green CI aggregate or universal absence of defects.
  4. Add the smallest useful regression/eval or review/delivery correction at the point of failure. For OPD, demonstrate that a production artifact missing the required editor payload would fail the check. Use realistic failure cases; do not add a generic checklist or duplicate existing tests without evidence.
  5. Independently review the correction and observe its effect on the ongoing delivery. Report what is now prevented, what remains unproven, and the next responsible action. No universal reliability claim from one successful run.

Ownership and scope

Existing Factory AgentOps owns the bounded investigation with the existing OPD, AIOS and Bridge leads. Keep current product delivery moving; no competing writers, new platform, custom runtime/approval service or broad backlog implementation is required. Prefer existing native tools, project checks and a small canonical method change only if the evidence warrants it.

This is a new current-method functional acceptance issue. It builds on the completed learning cycle in #8, complements the secret-safe release scope of #44, and must not wait for the future platform verification work in #68.

Activity

  1. gustavonline commented on Oct 17, 2026

    @gustavonline
    ContributorAuthor

    The method correction is delivered in PR77 and Factory 0.1.18.

    Two concrete escapes were confirmed:

    • OPD: production smoke was optional before adoption. Six passing library tests used a substitute resource list, so they did not prove the installed editor payload.
    • AIOS: the contrast-only writer/review brief dropped the accepted centred-geometry amendment. Source/proxy colours did not prove native appearance.

    The existing skills now require applicable proof from the exact production artifact before adoption; missing proof holds installation. Accepted amendments must reach both writer and reviewer, and AgentOps must reconcile actual child results before reporting completion. The retained six-asset output fails for missing editor.js and panel.js; the complete replacement passes membership and receipt checks. The cache cause remains unproven. Independent review caught and closed the adoption-timing gap; CI and the bounded decision exercises passed. Both existing skill copies were verified, with Codex explicit loading checked; automatic invocation and fresh Claude loading are not claimed.

    Latest product readback:

    • AIOS delivered: owner icon PASS, native MCP opening and Settings checks passed; #56 is closed and PR57 merged with the reviewed/installed tree unchanged. Separate global-sidebar navigation and Setup were not claimed.
    • OPD remains in progress: the owner chose one localhost web editor, usable in Codex’s browser or an ordinary browser. The MCP panel is optional. Its existing lead and same healthy writer own direct create/open/edit/Saved/close/edited-reopen plus agent edits to the same canvas and conflict-safe persistence. Preserve the earlier package proof at its original revision; qualify the next candidate rather than requesting another owner drawing test now.
    • Bridge: queue-history owner PASS stands. The bounded metadata readback is complete, but expected toast/owner-action binding remains unknown; the earlier failed Undo stays separate. No repeat probe or speculative/custom Undo.

    Method delivery is complete. #76 stays open to reconcile OPD’s actual delivery result through its existing lead; it is not a new platform or parallel product task.

  2. gustavonline commented on Oct 17, 2026

    @gustavonline
    ContributorAuthor

    Bridge's bounded archive readback is complete. The actual app/build and bundled CLI match the retained binding. Two archive operations and a separate-thread unarchive have explicit desktop-to-native trace correlation, successful finishes and semantic lifecycle records. This is protocol completion evidence, not visible-feedback acceptance.

    The new owner test's exact thread/action was not supplied, and a same-build, same-action native feedback reference remains unavailable. Missing toast is still an observation, not an established Bridge defect. The earlier failed Undo remains separate and unresolved; no custom or forced Undo is authorized. RC10 queued-user history at turn start remains owner-reported PASS. Pending-time visibility and timely browser feedback remain unqualified.

    The method boundary is precise: matching the installed version and completing correlated requests does not prove the owner's visible workflow or create a new UX requirement. The existing readback is fulfilled; no generic repeat is needed. Only an already available exact owner-action binding/native reference, or an evidenced contract divergence, can advance the remaining archive diagnosis. Bridge AgentOps retains delivery and any product correction with its existing writer/reviewer; Factory retains the canonical-method work.

    Private evidence stays with Bridge. No product change, build/review, repeated GUI test or host mutation was performed for this reconciliation.

  3. added a commit that references this issue on Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions