Complete the AgentOps Value Based Delivery materials #483

Activity

1. github-actions

   added the

   Task

   issue type on Sep 2, 2026
2. placerda commented on Sep 6, 2026

   

   Paulo Lacerda (placerda)

   on Sep 6, 2026

   ContributorAuthor

   More actions

   Unblock note for the Evaluate lab: #484 is done and shipped in v0.15.0.

   agentops eval run now wraps the current azd ai eval surface, so a lab using evals/azure.eval.yaml gets normalized results.json / report.md, threshold and rubric gating, baseline comparison, and the standard 0/2/1 exit codes.

   One thing to plan the lab around: azure.ai.evaluations is not published yet. It is still an unmerged PR against Azure/azure-dev and is absent from the azd extension registry, so azd extension install azure.ai.evaluations does not resolve on a clean machine.

   For a lab this means either:

   • pre-build the extension from the PR branch and publish it into a local azd extension source on the lab image (steps are in docs/tutorial-hosted-agent.md), or
   • run the lab on the legacy eval.yaml surface, which is fully supported and unchanged, and switch once the extension ships.

   Nothing else in the accelerator is affected: the new surface is strictly opt-in and only activates when evals/azure.eval.yaml exists.

   Leaving this issue open since the VBD materials are broader than the evaluation gap.
3. placerda commented on Sep 6, 2026

   

   Paulo Lacerda (placerda)

   on Sep 6, 2026

   ContributorAuthor

   More actions

   Evaluate lab dependency update: #484 is closed and the adapter shipped in AgentOps Accelerator v0.15.0. This completes the Accelerator implementation dependency, not the hands-on lab acceptance criteria. Azure/azure-dev#9500 is still open and unmerged as of 2026-09-06; the closing report for #484 confirms that the current evaluation extension is not yet available through the public registry and live Azure end-to-end validation was not performed. The VBD Evaluate lab currently contains its definition, topics, and outline only. Keep "Write and validate the complete Evaluate hands-on lab" unchecked. Remaining work: prepare the runnable lab assets and detailed instructions, establish an installable supported evaluation path for participants, then run the exercise against Azure and validate the 60-minute duration. Building the unpublished extension from source is an engineering validation option, not an assumed workshop prerequisite.
4. placerda commented on Sep 6, 2026

   

   Paulo Lacerda (placerda)

   on Sep 6, 2026

   ContributorAuthor

   More actions

   Evaluate lab implementation update: public tooling path

   This supersedes the extension dependency decision in #483 (comment). #484 shipped through #485 in v0.15.0. azure-dev#9500 remains open/unmerged and azure.ai.evaluations is not available from the checked registry, but that extension no longer gates this lab.

   Implemented locally on placerda-agentops-vbd in placerda/AzureAIGovernance (not committed or pushed):

   • Complete 60-minute learner lab and 20-minute instructor subset, plus instructor pre-work.
   • Shared Python Responses-protocol hosted help desk source with synthetic local knowledge and controlled tool/source defects. Uses existing assigned Foundry projects/models; no Search or Foundry IQ provisioning requirement.
   • Eight versioned cases, explicit item/sample mappings, native dataset/rubric registration and cloud agent evaluation, local SDK smoke, calibration/conversation teaching runs, and existing red-team run export/review guidance.
   • A clearly identified lab-owned bridge using released Accelerator v0.15.0 internals for normalization, full-denominator pass-rate gates, baseline compatibility and reports. This is not an invented Accelerator CLI backend. Original native payloads are retained; missing/error/ambiguous evidence fails closed.
   • README/implementation-guide alignment and regenerated one-pager.

   Validation: 26 offline tests passed, including actual agentops report generate, baseline/gate/error cases, request-shape mocks, polling and ambiguous-submission recovery. Dataset, Python syntax, relative links, 21 external links, deck text/notes alignment, one-pager render and git diff --check passed. Tests used exact v0.15.0 release source with release-version build metadata; remaining evaluation requirements resolved/imported successfully. No test fixtures are presented as observed cloud evidence.

   Still outstanding: the local resolver could not obtain the published hosted framework packages, and direct wheel downloads failed TLS negotiation. Host startup/Python 3.13 validation is therefore unverified. No assigned deployment handoff was available for authorized cloud execution. Real candidate/baseline, judge calibration, conversation and red-team evidence, participant access/concurrency, and timed 60/20-minute rehearsal remain pre-work gates. No billed cloud operations were performed.

   The Write and validate the complete Evaluate hands-on lab checkbox remains unchecked, and this broader issue remains open. Adoption of the new evaluation extension is follow-up work, not the current blocker.
5. placerda commented on Sep 6, 2026

   

   Paulo Lacerda (placerda)

   on Sep 6, 2026

   ContributorAuthor

   More actions

   Evaluate redesign: supported public CLI (supersedes the bridge approach)

   The local VBD working material now uses the released AgentOps Accelerator v0.15.0 hosted-agent cloud workflow, not a lab-owned integration with internal Python modules.

   • Main learner path: agentops init / init show, dataset + agentops.yaml, agentops eval run --config ... --output ... --baseline ..., native report review, and agentops report generate --in ... --out ... for saved results. Configuration uses documented execution: cloud, protocol: responses, a real versioned hosted endpoint and inline JSONL.
   • Explicit supported CLI evaluators: CoherenceEvaluator -> builtin.coherence and SimilarityEvaluator -> builtin.similarity; teaching thresholds are mean scores >=3 on a pilot-verified 1-5 scale. The eight-row help desk dataset remains live-agent input, not prerecorded responses. All columns are strings to match this release's cloud item schema.
   • Removed obsolete lab.py, evidence.py, bridge tests and separate gate JSON. No new wrapper, importer, internal-module dependency or pseudo execution mode. The real tool-using help desk host remains isolated instructor pre-work; learners need no hosting SDK knowledge.
   • Native Foundry domain rubric/calibration, safety, conversation and red-team evidence is supplementary and separate from the two-metric CLI gate. Nothing claims skipped custom/safety evaluators ran through Accelerator.
   • Boundaries are explicit: baseline deltas are reporting-only; available-score averages are not a completeness/critical-case gate; raw rows and actual tools require review; builtin service versions and deployed source need provenance checks; no public CLI resume/collect command. Legacy azd evaluation remains aggregate-only, and the unpublished azure.ai.evaluations extension is not required.
   • Updated lab, TOOLING, pre-work, shared agent README, config/data/decision/evidence assets, root README, Evaluate implementation guidance, and synchronized one-pager Markdown/PDF. Preserved 60-minute hands-on and 20-minute demo modes, with setup outside the clock.

   Offline verification completed: 10 unittest checks using public CLI subprocesses (release/help/init/config analysis, early config/missing-baseline errors, report pass/fail/error rendering and corrupt input), dataset/supplement checks and deterministic tool tests; 54 relative links resolve; diff whitespace check passes; regenerated one-page PDF opened/rendered/visually reviewed. Report fixtures are explicitly marked OFFLINE, not cloud evidence.

   Still pending: supported package installation rehearsal (authoring used unchanged exact-tag source build), isolated host dependency/startup verification, assigned project/identity/permissions/quota, real agent/cloud evaluations and supplementary evidence, concurrency and 60/20-minute rehearsal. No assigned cloud project was available, no cloud/billable operations were run, and no live validation is claimed. Changes remain local, uncommitted and unpushed. Keep the complete-and-validate Evaluate checklist item and this broader issue open until those delivery checks pass.
6. placerda commented on Sep 11, 2026

   

   Paulo Lacerda (placerda)

   on Sep 11, 2026

   ContributorAuthor

   More actions

   One-pager finalized

   The customer-facing AgentOps VBD one-pager is reviewed and approved. The Markdown source and one-page PDF are published on main in placerda/AzureAIGovernance at f3a9c8e.

   Final one-pager (PDF)

   The one-pager checklist item remains complete. This closes the one-pager work only; the remaining hands-on labs, implementation guides, and cross-material review remain open.