Add /cpa:product-review command

[kristenoneill]

Summary

Adds /cpa:product-review — a new command under canvas-plugin-assistant/commands/ that evaluates a plugin from a product perspective. Complementary to /cpa:security-review (engineering concerns); this one surfaces product judgments a human needs to make.

The command checks nine tenets:

1. Clinical data model / USCDI V3 — flags CustomModels that restate data Canvas already has in its USCDI-backed native models.
2. Placement — manifest scope must match intent (provider_menu_item / global / patient_specific / portal_menu_item).
3. Coded data capture — flags string-based matching on clinical concepts; recommends SNOMED/LOINC/RxNorm/ICD-10/CVX/NUCC.
4. Native Canvas primitives — CustomModels that should be Questionnaires, Tasks, Notes, Goals, Commands, CareTeamMembership, Organization/Location, etc.
5. Alert fatigue — trigger scope, dismissibility, and blocking-vs-passive assessment for each interruption.
6. Configurability — flags hardcoded clinical thresholds and code lists that should live in settings or content.
7. Audit trail / attribution — flags writes that don't capture the actor and shadow-override patterns.
8. Terminology consistency — Canvas vocabulary (Patient/Staff/Note/Command) and clinical code-system field naming.
9. ONC certified-criteria overlap — cross-references plugin behavior against every §170.315 criterion Canvas is certified for; classifies overlaps as augment vs. replace, and flags replace-class overlaps HIGH for compliance risk.

The command writes a timestamped report to .cpa-workflow-artifacts/product-review-*.md and uses AskUserQuestion to route next steps. It explicitly does not auto-fix USCDI or ONC findings — always confirms first, because they touch the designated record set and compliance posture.

Also updates the workflow reference tables in wrap-up.md and security-review.md so the new command appears in the standard chain.

Test plan

• Run /cpa:product-review against a plugin with a clean bill of health → verify ✅ PASS verdict
• Run against a plugin with CustomModel fields that duplicate USCDI concepts → verify Tenet 1 flags with file:line references
• Run against a plugin that sends referrals / transmits PHI externally → verify Tenet 9 flags ONC overlap as HIGH
• Verify report is written to .cpa-workflow-artifacts/product-review-<timestamp>.md
• Verify AskUserQuestion next-steps prompt appears after findings
• Confirm the command does not silently mutate code for USCDI or ONC findings without confirmation

🤖 Generated with Claude Code

[kristenoneill]

@andhines - would love your eyes on this from a product perspective - anything I missed? I ran it against a handful of plugins to confirm it aligns with my general thinking. It might be overly sensitive in areas - but figure its better to flag/have a conversation.

[andhines]

Yes I'll take a deeper look today. I scanned briefly and noticed two things. First thing is high level, shouldn't this be skill applied along the way, not just a review command? Would be great to guide correct decisions along the way not just invoke on review. Second, tactical thing, it seems our data model is characterized as USCDI v3 but it's so so so so so so much more than that and includes operational and financial stuff, need to characterize that too.

…

On Mon, Apr 20, 2026 at 3:04 PM Kristen ONeill ***@***.***> wrote: *kristenoneill* left a comment (canvas-medical/coding-agents#78) <#78?email_source=notifications&email_token=AATYZNLLBGRAW65RSFLFO5T4W2NGPA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMRYGQ2TOMZSGQZKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJNLQOJPWG33NNVSW45C7N5YGK3S7MNWGSY3L#issuecomment-4284573242> @andhines <https://github.com/andhines> - would love your eyes on this from a product perspective - anything I missed? I ran it against a handful of plugins to confirm it aligns with my general thinking. It might be overly sensitive in areas - but figure its better to flag/have a conversation. — Reply to this email directly, view it on GitHub <#78?email_source=notifications&email_token=AATYZNLLBGRAW65RSFLFO5T4W2NGPA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMRYGQ2TOMZSGQZKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJNLQOJPWG33NNVSW45C7N5YGK3S7MNWGSY3L#issuecomment-4284573242>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AATYZNKMXPQ62S7WZ4NZLB34W2NGPAVCNFSM6AAAAACX7746J6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DEOBUGU3TGMRUGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Andrew Hines Founder, Canvas Medical (415) 610-5175 220 Montgomery Street, Suite 991 San Francisco, CA 94104

-- Disclaimer: This e-mail and any attachments may contain confidential information. If you are not the intended recipient, any disclosure, copying, distribution or use of any information contained herein is strictly prohibited. If you have received this transmission in error, please immediately notify the Security Officer ***@***.***>, and destroy the original transmission and any attachments without reading or saving.

[kristenoneill]

@andhines broadened to evaluate based on what is typically considered a part of the designated record set.

When it asked me skill vs command - the examples of the skills it gave as a reference/to differentiate didn't seem like parallels - but makes sense.

[aduane]

Evaluation: ran the new tenets against 10 plugins in msf-canvas

Walked the proposed /cpa:product-review tenets through 10 plugins in msf-canvas by hand, simulating what the command would surface. Sample covered three provider-companion plugins (provider_register_patient_companion, provider_note_vitals_companion, provider_clinical_summary_companion) plus seven varied non-companion plugins (high-risk-medications, hcc_capture, custom-observation-management, chronic-care-management-activity-tracker, bmi_coding_automation, abstractive_health, claim_coding_agent).

Overall: the tenet structure is sound and produces the right calls on the strongest real findings (substring matching on med names, 1000-code inline ICD lists, API-key writes without staff session). Six suggestions below — most are framing/sharpening rather than structural changes.

---

1. Provider companion scopes are missing from Tenet 2's mapping table

The scope→surface table at commands/product-review.md:140-147 enumerates provider_menu_item / global / patient_specific / portal_menu_item, but the new 3-tier provider companion scopes (provider_companion_global, provider_companion_patient_specific, provider_companion_note_specific) aren't there. All three companion plugins in the sample use one of those scopes, so Tenet 2 produces an inconclusive or false-positive result on every modern companion plugin.

Suggested addition to the table:

Plugin purpose	Scope	UX surface
Provider mobile launcher (no patient/note context)	provider_companion_global	Provider companion home
Provider mobile patient view	provider_companion_patient_specific	Companion patient page
Provider mobile note view	provider_companion_note_specific	Companion note view

2. Tenet 7 detection has some false-positive risk worth softening

Tenet 7 (commands/product-review.md:299-345) actually has good nuance already — it distinguishes actor from routing, has a "Not a Tenet 7 flag" carve-out for routing-only API-key endpoints, etc. The risk is in the Detect heuristics at :309-317: greps over .save()/.create()/Effect/etc. will surface every native-effect emission, and a reviewer may read the surfaced grep hits as findings even though the rule's logic would clear them.

Rather than tightening the rule, the better posture is to encourage attribution where an actor is reachable:

• For handlers extending StaffSessionAuthMixin, the staff is right there in the session — capture and pass it through to whatever the handler emits, even if the platform also stamps it.
• For CronTask and other actor-less contexts, default to a canvas bot fallback until a plugin-specific fallback actor is available.

A line under the Detect block — "if the handler is StaffSessionAuthMixin and emits native Effects, the platform attributes the dispatching staff; do not flag merely because actor capture isn't visible in the source" — would help the reviewer interpret the grep hits correctly.

3. Add a "confirm-and-document" surfacer for cascade / auto-finalize patterns

Three plugins in the sample finalize clinical or billing writes from system-driven event handlers without a provider-driven trigger:

• bmi_coding_automation/protocols/bmi_diagnosis.py:180,189 — originate() + commit() of Diagnose/UpdateDiagnosis on VITALS_COMMAND__POST_COMMIT
• claim_coding_agent/protocols/agent.py:30-40 — AddBillingLineItem.apply() on PERFORM_COMMAND__POST_ORIGINATE
• chronic_care_management_activity_tracker/handlers/ccmat_monthly_cron.py:236,272,316 — Note + Diagnose + billing emitted from a monthly cron

These aren't necessarily anti-patterns — sometimes "when action A happens, action B should also happen" is the desired design. But they're worth surfacing so the developer confirms intent and documents the cascade in the README. Suggested behavior: when the command sees a .commit() (or equivalent finalization) emitted from a system-driven event handler, surface as a "confirm-and-document" item rather than a HIGH finding.

4. Tenet 6 needs a clinical-content vs. UI-mechanics rubric

Tenet 6 (commands/product-review.md:277-296) currently says "hardcoded threshold → flag, hardcoded code list → flag." That works for clear-cut cases but produces noise on UI knobs that happen to be numeric.

From the sample:

• Should flag: hcc_capture/protocols/condition_annotations.py:10-1090 (1000+ ICD-10 codes inline), bmi_coding_automation/protocols/bmi_diagnosis.py:80-131 (22 BMI bands), claim_coding_agent/protocols/agent.py:32-38 (hardcoded CPT 99213 + units + modifiers), chronic_care_management_activity_tracker/handlers/ccmat_monthly_cron.py:36,310-321 (CCM minute thresholds + CPT 99490/99439)
• Borderline: 1-year DOB fuzzy-match window in register_patient_companion, patterns-via-secret in high-risk-medications
• Should not flag: _VITALS_MAX_PANELS = 12 in clinical_summary_companion, 4-digit phone narrowing in register_patient_companion

Suggested rubric to add near the top of Tenet 6:

Flag a hardcoded value when it encodes a clinical, regulatory, or billing rule that a non-engineer would expect to be content/settings (CMS thresholds, code lists, medical-society cutoffs, billing rates). Don't flag UI mechanics or implementation knobs (page sizes, retry counts, polling intervals, fuzzy-match windows that are heuristics rather than clinical thresholds). When in doubt: would a clinician or biller want to change this without filing an engineering ticket?

5. Consider a manifest-accuracy tenet (or a Tenet 8 extension)

claim_coding_agent's manifest description reads "Example plugin using LLM inference to manage claim coding in real time" and declares OPENAI_SECRET_KEY as a required secret, but protocols/agent.py:23-24 instantiates LlmOpenai and never calls it — the plugin actually emits a hardcoded CPT 99213 with no LLM in the path. That's a real product issue (misleading description for installers, declared secret that isn't used) but no current tenet covers it.

Suggested addition (either under Tenet 8 or as a new small tenet):

Manifest accuracy. The plugin's manifest description should match its actual behavior. Flag when:

• declared secrets are imported but never read
• the description references a capability (LLM, external API, scheduled job) that the code doesn't exercise
• declared components reference dead classes

6. Reference data duplicated between command and skill

The PR adds both commands/product-review.md (573 lines) and skills/product-review/product_review_context.txt (323 lines). Most of the bulk in the command file is reference data — USCDI/DRS tables (product-review.md:51-133), ONC criteria list (product-review.md:378-440) — and the skill exists specifically to hold the rule set as inline-applicable context. Two copies of the same reference data is a maintenance hazard (two places to update on the next ICD release / criterion change).

Suggested: trim the command file to the workflow steps (validate env, orient, walk tenets, generate report, route next steps) and reference the skill for the rule data — e.g., "See product-review skill context for the USCDI/DRS table." Keeps one source of truth for the rule definitions.

---

Happy to discuss any of these or open separate issues if it's easier to track. Items 1, 4, and 6 are the ones I'd flag as most likely to bite reviewers immediately; the others are sharpening.

[andhines]

@kristenoneill I recommend updating https://github.com/canvas-medical/coding-agents/blob/main/canvas-plugin-assistant/CLAUDE.md to get the tenets broadly applied

[andhines]

Further guidance, mostly from claude-code-guide, on the topic of integrating the skill into the plugin more broadly / cohesively:

Higher-leverage integration points

  In rough order of impact:                                                                                            
   
  1. The plugin-brainstorm agent's own .md file (highest leverage)                                                     
  Since agents are isolated, the agent definition itself needs to reference the product-review skill. Either embed a
  "before proposing data model or scope, apply Tenets 1, 2, 4, 9" instruction, or have the agent invoke the skill      
  explicitly during the planning dialogue.
                                                                                                                       
  2. The skill's own description: frontmatter                                                                          
  Skills auto-trigger based on description matching. The current description is good but trigger keywords could be
  strengthened — e.g. explicit phrases like "creating CustomModel", "declaring Application scope", "matching clinical  
  concepts" help Claude self-invoke without being told. This is the most "passive" integration: well-written
  descriptions mean the skill activates without anyone reminding Claude.                                               
                           
  3. Cross-references from related skills                                                                              
  plugin-patterns, custom-data-patterns, and canvas-sdk are loaded when their domains come up. A one-line pointer in
  each ("before finalizing custom data model, also apply product-review skill — Tenet 1") is more reliable than        
  CLAUDE.md because it fires in-context, when the relevant work is happening. The product-review SKILL.md already does
  the reverse (lines 44–49 reference these skills); the back-references are missing.                                   
                           
  4. Workflow commands (/new-plugin, /cpa:wrap-up, etc.)                                                               
  Commands have explicit step lists. /cpa:wrap-up is already named as the gating step in the skill. Verify /new-plugin
  and any scaffolding commands include a product-review checkpoint after the data model / manifest is drafted but      
  before code generation begins.
                                                                                                                       
  5. CLAUDE.md (your suggestion — still useful, just narrower)                                                         
  Best used for a short, specific reminder: "When planning new plugins or modifying CustomModels/AttributeHubs/manifest
   scope, invoke the product-review skill." Don't restate the tenets — that's the skill's job. CLAUDE.md should route  
  to the skill, not duplicate it.
                                                                                                                       
  6. Hooks (heavier-handed, optional)
  A PostToolUse hook on Write/Edit to CANVAS_MANIFEST.json or files matching *custom_model* could inject a system
  reminder to apply Tenets 1/2. Useful as a backstop but adds noise — only worth it if drift is a recurring problem.   
   
  The mental model to give the developer                                                                               
                           
  ▎ CLAUDE.md is the passive reminder layer — fine for routing. The real integration happens by (a) writing skill      
  ▎ descriptions that auto-trigger, (b) cross-referencing from neighboring skills so it surfaces in-context, and (c) 
  ▎ embedding the trigger into agents and commands that don't inherit project CLAUDE.md.                               
                           
  Best ROI: fix the plugin-brainstorm agent first (the planning blind spot), then add cross-references in              
  plugin-patterns and custom-data-patterns, then the targeted CLAUDE.md line.