Skip to content

feat(programs): programs own their orchestrator flows and prompts - #1498

Draft
edwinyjlim wants to merge 2 commits into
dag/d1-framework-traitsfrom
dag/w2-program-flows
Draft

edwinyjlim wants to merge 2 commits into
dag/d1-framework-traitsfrom
dag/w2-program-flows

Conversation

@edwinyjlim

@edwinyjlim edwinyjlim commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

Summary

The five orchestrator programs declare their task graph in code and carry their own task prompts. With wizard-agent-dag on, the orchestrator runs on them. Off, it fetches prompts from context-mill and follows the planner, as today. Either way, each run logs where the typed graph and the planner disagree.

Stack

The four PRs, in merge order:

PR What it does How it fits in
C1 · PostHog/context-mill#447 Adds a title, description and tags to every skill-menu entry. Wizards in the field ignore the new fields. Ships first, in a context-mill release. W1 picks skills by these tags, and W1 and W2 record their fixtures and prompts from this release.
W1 · #1497 Adds project traits and a tag-based skill pick that runs beside today's pick, plus the wizard-agent-dag flag that decides which pick installs. Base of the wizard stack, on main. Reads C1's tags but works without them. D1 builds on its traits service, and W2 on its traits and flag.
D1 · #1499 Records every framework write as a project trait with its source. No flag; the values a run uses don't change. On W1. Extends W1's traits service, so the traits carry the framework as it was written. W1's skill pick and W2's flow conditions read it.
W2 · #1498 (this PR) Moves the task graphs and prompts of the five orchestrator programs into the wizard. With wizard-agent-dag on, runs use them instead of the fetched prompts and the planner's graph. Top of the stack, on D1. Its flow conditions read W1's traits, W1's flag switches it on, and its prompts come from C1's release.

W1, D1 and W2 merge in that order and ship together in one wizard release, with wizard-agent-dag off. A later PR, W3, removes the flag and the paths it replaced after a week at 100%.

How a run uses them

flowchart TD
  flow["flow.ts<br/>nodes, dependsOn, when"] --> resolve["resolveFlow<br/>with the run's traits and flags"]
  resolve -->|RunConfig.taskGraph| orch
  prompts["agent/*.md<br/>imported as text"] -->|RunConfig.agentPrompts| orch
  orch["Orchestrator run"] --> flag{"wizard-agent-dag"}
  flag -- off --> off["Prompts fetched from context-mill<br/>Queue as the planner builds it"]
  flag -- on --> on["Bundled prompts, fetched if loading fails<br/>Queue rewired to the typed graph"]
  off & on -.-> events(["orchestrator graph diff<br/>orchestrator exclusions differ"])
Loading
wizard-agent-dag off wizard-agent-dag on
Task prompts fetched from context-mill bundled; fetched from context-mill if the bundle fails to load or has no planner prompt
Excluded task types the program's excludedTaskTypes the nodes resolveFlow dropped
Task dependencies as the planner queues them rewired to the typed graph when the two differ
Graph diff and exclusions events sent sent

Example: posthog-integration

flowchart LR
  install --> identify
  init --> identify
  install --> et["error-tracking"]
  init --> et
  identify --> capture
  capture --> aio["ai-observability"]
  aio --> logs
  logs --> review
  et --> review
  capture --> dashboard
  review -.-> warehouse
  dashboard -.-> warehouse
  review --> report
  dashboard --> report
  warehouse -.-> report
Loading
  • ai-observability runs unless wizard-default-aio-logs is false or the callsLlmApi trait is no. logs runs unless that flag is false.
  • warehouse (dotted) is queued only when the wizard finds data sources.
  • Edges that others imply are left out. A dropped node's dependents wait on its own dependencies.

The five flows

Program Tasks Runs last Prompts
posthog-integration install, init, identify, error-tracking, capture, ai-observability, logs, review, dashboard, warehouse, report report 12
error-tracking install, init, capture-exceptions, credentials, configure, wire-ci, link-releases, verify, report, test-setup test-setup 11
feature-flags install, init, create-flag, evaluate, report report 6
metrics verify-sdk, instrument-metrics, report report 4
replay-vision install, init, enable-replay, scanner-broken-experiences, scanner-user-frustration, scanner-session-summaries, report report 8

Each program has one prompt per task plus its planner's, 41 in all, in src/programs/<id>/agent/<type>.md. Its agent-prompts.ts imports them as text, and ProgramConfig.agentPrompts loads it on first use. The build inlines the files with tsdown's .md loader, tests load them through a Vitest plugin, and pnpm try through a Node load hook (scripts/markdown-loader.no-jest.mjs). Context-mill's integration-v2 flow lands in posthog-integration/agent/. In error tracking, verify runs after every task that edits code except wire-ci, and the report waits for it.

Telemetry

Event Sent
orchestrator graph diff after the planner queues its tasks: whether its graph agrees with the typed one, which one decides, and the types and edges that differ
orchestrator exclusions differ at the start, when the dropped nodes differ from the program's excluded task types
orchestrator agent prompts loaded with the flag on: source is bundle, or fallback when the prompts came from context-mill

Test plan

  • pnpm typecheck, pnpm lint (no errors, no new warnings in new files)
  • pnpm test: 263 files, 3,878 tests
  • Bundled prompts are byte-identical to the recorded context-mill build, and assemble the same seed and task prompts as the fetched ones, for all five flows
  • Flow tests: one node per task prompt, the same sink, acyclic, every node reaches the sink, runner-seeded tasks only feed the sink, the same exclusions as excludedTaskTypes for every AIO/Logs flag value, and the integration graph matches its planner prompt
  • The built package's prompt chunks are byte-identical to the .md files, for all five programs
  • Before merge: re-record fixtures from the context-mill release that carries feat(menu): describe skills with title, description and tags context-mill#447, and copy its prompts into src/programs/*/agent/
  • Not in this PR: a warlock scan of src/programs/*/agent/*.md in CI. Context-mill's scan triages matches with an LLM through the gateway, which needs CI secrets in this repo. It is needed before prompts are edited here.
  • After release: graph-diff rate with the flag off, recorded as the baseline

🤖 Generated with Claude Code

@github-actions

Copy link
Copy Markdown

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

  • /wizard-ci all

Test all apps in a directory:

  • /wizard-ci ai-observability
  • /wizard-ci basic-integration
  • /wizard-ci feature-flags
  • /wizard-ci mcp-analytics
  • /wizard-ci replay-vision
  • /wizard-ci revenue
  • /wizard-ci self-driving
  • /wizard-ci warehouse
  • /wizard-ci warehouse-seeded

Test an individual app:

  • /wizard-ci ai-observability/anthropic
  • /wizard-ci ai-observability/google-adk
  • /wizard-ci ai-observability/groq
Show more apps
  • /wizard-ci ai-observability/manual-capture
  • /wizard-ci ai-observability/openai
  • /wizard-ci ai-observability/openai-agents
  • /wizard-ci ai-observability/opentelemetry
  • /wizard-ci ai-observability/vercel-ai
  • /wizard-ci basic-integration/android
  • /wizard-ci basic-integration/angular
  • /wizard-ci basic-integration/astro
  • /wizard-ci basic-integration/django
  • /wizard-ci basic-integration/fastapi
  • /wizard-ci basic-integration/flask
  • /wizard-ci basic-integration/flutter
  • /wizard-ci basic-integration/javascript-node
  • /wizard-ci basic-integration/javascript-web
  • /wizard-ci basic-integration/laravel
  • /wizard-ci basic-integration/next-js
  • /wizard-ci basic-integration/nuxt
  • /wizard-ci basic-integration/python
  • /wizard-ci basic-integration/rails
  • /wizard-ci basic-integration/react-native
  • /wizard-ci basic-integration/react-router
  • /wizard-ci basic-integration/sveltekit
  • /wizard-ci basic-integration/swift
  • /wizard-ci basic-integration/tanstack-router
  • /wizard-ci basic-integration/tanstack-start
  • /wizard-ci basic-integration/vue
  • /wizard-ci feature-flags/django
  • /wizard-ci feature-flags/next-js
  • /wizard-ci mcp-analytics/custom-dispatcher
  • /wizard-ci mcp-analytics/typescript-sdk
  • /wizard-ci replay-vision/javascript-node
  • /wizard-ci replay-vision/next-js
  • /wizard-ci replay-vision/react-vite
  • /wizard-ci revenue/stripe
  • /wizard-ci self-driving/astro
  • /wizard-ci self-driving/fastapi
  • /wizard-ci self-driving/nuxt
  • /wizard-ci self-driving/react-router
  • /wizard-ci self-driving/sveltekit
  • /wizard-ci warehouse/monorepo-env
  • /wizard-ci warehouse/multi-source-next
  • /wizard-ci warehouse/stripe-node
  • /wizard-ci warehouse/zero-source
  • /wizard-ci warehouse-seeded/next-stripe
  • /wizard-ci warehouse-seeded/next-stripe-declined

Test against a Context Mill branch:

  • /wizard-ci all context-mill:my-branch

Add context-mill:<branch> to any command above to pin the Context Mill branch. It defaults to main.

Results will be posted here when complete.

The five orchestrator programs (posthog-integration, error-tracking,
feature-flags, metrics, replay-vision) declare their task graph in
`flow.ts` with `defineFlow` from the program contract, and keep their
task prompts in `agent/<type>.md`. `pnpm bundle:agent-prompts` generates
each program's `agent-prompts.ts`, which `ProgramConfig.agentPrompts`
loads on first use.

`resolveFlow` applies the run's project traits and flags, dropping a node
whose `when` fails and splicing its edges into its dependents. The
program runner hands the graph to the orchestrator as
`RunConfig.taskGraph`, and the prompts as `RunConfig.agentPrompts`.

With `wizard-agent-dag` on, the orchestrator loads the bundled prompts,
falling back to context-mill's on any failure, and rewires the planner's
queue to the typed graph through `QueueStore.replaceDependencies`. Off,
it fetches the prompts as before. Either way it logs
`orchestrator graph diff` and `orchestrator exclusions differ` where the
typed graph and the planner disagree.

The prompts match the recorded context-mill release byte for byte, and
assemble the same seed and task prompts as the fetched ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@edwinyjlim
edwinyjlim removed this pull request from stack #1503 October 29, 2026 22:52
@edwinyjlim
edwinyjlim changed the base branch from dag/w1-skills-by-traits to dag/d1-framework-traits October 29, 2026 22:52
@edwinyjlim
edwinyjlim force-pushed the dag/w2-program-flows branch from 8d5e5b6 to e3c6869 Compare October 29, 2026 22:52
@edwinyjlim
edwinyjlim added this pull request to stack #1504 October 29, 2026 22:52
Each program's agent-prompts.ts imports its agent/<type>.md files as text
instead of carrying a generated copy. tsdown's .md text loader inlines them
in the build, a Vitest plugin loads them in tests, and pnpm try registers a
Node load hook that does the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant