docs: cover payments, policies/guardrails, insights, harness, datasets, web-search by notgitika · Pull Request #1577 · aws/agentcore-cli

notgitika · 2026-06-18T15:46:40Z

Why

Two customer-facing docs had drifted from the actual feature surface:

README.md — only mentioned a small subset of resource types in the add/remove descriptions, and several features that already had docs/*.md files (payments, knowledge bases, container builds, transaction search, telemetry, tui-harness) weren't surfaced anywhere in the README.
src/assets/agents/AGENTS.md — the AI-assistant context file we vend into every agentcore create project. Its AgentCoreProjectSpec resource list was missing 9+ top-level arrays that customers can edit (onlineInsightsConfigs, knowledgeBases, harnesses, policyEngines, policies, payments, configBundles, datasets, runtimeEndpoints), so AI assistants pointed at the file would happily fabricate the wrong shapes when asked to add those resources.

What changed

`README.md`

add/remove descriptions broaden from "agents, memory, credentials, evaluators, targets" to the full 15+ resource types
New command-table sections linking to existing docs/ files:
- Knowledge Bases
- Insights [preview] (run insights, add online-insights, pause/resume online-insights, view insights, archive insights)
- Harness (add harness + export harness, with the EXPORT_NOTES.md reminder)
- Policies & Guardrails (add policy-engine, add policy — including the canonical Bedrock content-filter enum: VIOLENCE | HATE | SEXUAL | MISCONDUCT | INSULTS)
- Payments (add payment-manager, add payment-connector)
- Datasets (add dataset, dataset download/publish-version/remove-version)
- Web Search Gateway Targets
Capabilities section now reflects the actual feature set
Documentation section is grouped (Reference / Resources & features / Evaluation & quality / Operations) and adds the missing entries: payments, container-builds, transaction_search, telemetry, tui-harness, PERMISSIONS, TESTING

`src/assets/agents/AGENTS.md` (vended to every customer project)

AgentCoreProjectSpec resource list now reflects every top-level array in the schema
Key Types covers the new resource shapes (PaymentManager, PaymentConnector, PolicyEngine, Policy, Harness, KnowledgeBase, ConfigBundle, OnlineInsightsConfig, Dataset, RuntimeEndpoint)
Common Enum Values adds PaymentProvider, PolicyEnforcementMode, GuardrailContentFilter — the spelled-out INSULTS enum so AI assistants emit the canonical name (issue TUI/CLI accepts INSULT as a contentFilter value, fails CFN deploy with ValidationException #1571 was a manifestation of the wrong-spelling problem)
CLI Commands replaced with grouped tables (lifecycle / resources / jobs / config bundles & datasets / observability / utilities) so the customer-vended file matches what the CLI actually exposes

Snapshot

src/assets/__tests__/__snapshots__/assets.snapshot.test.ts.snap updated via npm run test:update-snapshots per the convention in AGENTS.md.

Out of scope (filed as follow-ups)

The features without their own docs/*.md file get a brief table-row treatment in the README but no dedicated guide. Worth filing as separate issues:

docs/insights.md
docs/harness.md
docs/policies.md
docs/datasets.md

The internal contributor AGENTS.md (/AGENTS.md at repo root) is also slightly stale on the primitives list and run/view/archive/stop/promote commands — left for a separate PR since it's not customer-facing.

Verification

npm run typecheck ✅
npm run test:unit ✅ (5396/5396, snapshot updated)
npm run test:update-snapshots ✅
npx prettier --check ✅

…s, web-search The README and the vended AGENTS.md (copied into every agentcore create project as the AI-assistant context file) had grown stale relative to the feature surface. Several capabilities had docs in docs/ but no mention in the README, and the AGENTS.md customers see still listed only the original handful of resource types. README.md - Broaden 'add'/'remove' descriptions to enumerate the actual resource types (15+ now, was 'agents, memory, credentials, evaluators, targets') - Add command-table sections: Knowledge Bases, Insights ([preview]), Harness, Policies & Guardrails, Payments, Datasets, Web Search Gateway Targets — each linking to docs/ where one already exists - Refresh Capabilities to list the actual feature set - Group Documentation links into Reference / Resources & features / Evaluation & quality / Operations and add the missing entries (payments, container-builds, transaction_search, telemetry, tui-harness, PERMISSIONS, TESTING) src/assets/agents/AGENTS.md - AgentCoreProjectSpec resource list now reflects every top-level array in the schema (was missing onlineInsightsConfigs, knowledgeBases, harnesses, policyEngines, policies, payments, configBundles, datasets, runtimeEndpoints) - Key Types covers the new resource shapes - Common Enum Values adds PaymentProvider, PolicyEnforcementMode, GuardrailContentFilter (the spelled-out 'INSULTS' enum so AI assistants emit the canonical name) - CLI Commands replaced with grouped tables (lifecycle / resources / jobs / config bundles & datasets / observability / utilities) so the customer-vended file matches what the CLI actually exposes

agentcore-devx-automation · 2026-06-18T15:56:58Z

Claude Security Review: no high-confidence findings. (run)

github-actions · 2026-06-18T15:57:02Z

Package Tarball

aws-agentcore-0.20.1.tgz

How to install

gh release download pr-1577-tarball --repo aws/agentcore-cli --pattern "*.tgz" --dir /tmp/pr-tarball
npm install -g /tmp/pr-tarball/aws-agentcore-0.20.1.tgz

avi-alpert · 2026-06-18T17:31:36Z

+| `resume online-insights` | Resume a paused online insights config                      |
+| `archive insights`       | Delete an insights job on the service + clear local history |
+
+### Harness


there's some risk adding this to the main README while the feature is still only available in preview version. Perhaps we should add a temporary note that its only available in preview version?

that makes sense, I will add a note specifying that.

I think adding it in README will make it more visible so I would prefer to have it here

Done — added a [preview] tag to the section header and an npm install -g @aws/agentcore@preview install hint right under it. Confirmed the gating: src/cli/primitives/registry.ts:26 instantiates HarnessPrimitive only when isPreviewEnabled(), so the commands legitimately do not exist on the GA tarball.

…@preview Per Avi's review on PR aws#1577 — harness commands are only registered in preview builds (registry.ts:26 gates HarnessPrimitive on isPreviewEnabled()), so a GA-tarball user reading the main README would hit 'unknown command' if they tried any of the harness commands. Add a [preview] tag to the section header and a one-line install hint.

agentcore-devx-automation · 2026-06-18T17:59:37Z

Claude Security Review: no high-confidence findings. (run)

avi-alpert · 2026-06-18T18:24:53Z

+| --------------------- | ------------------------------------------------------------------------- |
+| `add harness`         | Add a harness resource (runtime + tools + skills + memory)                |
+| `add harness --tool`  | Add a tool to an existing harness                                         |
+| `add harness --skill` | Add a skill to an existing harness                                        |


aren't the commands agentcore add skill and agentcore add tool?

oh wait yes for the tool and i was not sure about the skill one. good catch!
let me confirm and make the change. I will also take a look at al the other flags

Good catch — fixed in e104e721. You were right on both counts: add tool and add skill are top-level subcommands under add, not flags on add harness (verified at src/cli/cli.ts:137-139 registering them via registerAddTool/registerAddSkill, and src/cli/commands/add/{tool,skill}-command.ts defining them with --harness <name> as a required option).

While I was in there I audited every other command row in the README against the source (config-bundle aliases, run/view/pause/resume/stop/promote/archive subcommands, evals history, online-eval/insights pairs, payment commands, policy-engine/policy, knowledge-base, dataset subcommands, web-search via gateway-target connector). The rest matches what the CLI actually exposes — only the harness rows were wrong.

Avi spotted that 'add harness --tool' and 'add harness --skill' are wrong: tool and skill are top-level subcommands under add, not flags on add harness. Verified against src/cli/cli.ts:137-139 (registerAddTool/registerAddSkill) and src/cli/commands/add/{tool,skill}-command.ts. Real shape: 'agentcore add tool --harness <name> --type <type>' and 'agentcore add skill --harness <name> --path|--s3|--git'. Also clarified 'add harness' is just runtime + model + memory — tools and skills get added separately afterward. Audited every other command row in the README against the source while in here. Everything else (config-bundle aliases, run/view/pause/resume/ stop/promote/archive subcommands, evals history, online-eval/insights pairs, payment commands, policy-engine/policy, knowledge-base, dataset subcommands, web-search via gateway-target connector) matches what the CLI actually exposes.

avi-alpert · 2026-06-18T18:31:25Z

+| \`agentcore run recommendation\` | Optimize prompts or tool descriptions from real traces |
+| \`agentcore run insights\` _[preview]_ | Run failure-pattern analysis across sessions |
+| \`agentcore run ab-test\` | Start an A/B test (config-bundle or target-based) |
+| \`agentcore run ingest\` | Ingest dataset rows for evaluation |


according to help screen this command has to do with knowledge base ingestion, not datasets

agentcore-devx-automation · 2026-06-18T18:31:39Z

Claude Security Review: no high-confidence findings. (run)

notgitika requested a review from a team June 18, 2026 15:46

This was referenced Jun 18, 2026

docs: write docs/insights.md #1578

Open

docs: write docs/datasets.md #1579

Open

docs: write docs/harness.md #1580

Open

docs: write docs/policies.md (Cedar policy engine + Bedrock guardrails) #1581

Open

notgitika temporarily deployed to e2e-testing June 18, 2026 15:52 — with GitHub Actions Inactive

github-actions Bot added the size/m PR size: M label Jun 18, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Jun 18, 2026

github-actions Bot added the agentcore-harness-reviewing AgentCore Harness review in progress label Jun 18, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Jun 18, 2026

github-actions Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Jun 18, 2026

avi-alpert reviewed Jun 18, 2026

View reviewed changes

github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Jun 18, 2026

notgitika temporarily deployed to e2e-testing June 18, 2026 17:58 — with GitHub Actions Inactive

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Jun 18, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Jun 18, 2026

avi-alpert reviewed Jun 18, 2026

View reviewed changes

avi-alpert requested changes Jun 18, 2026

View reviewed changes

github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Jun 18, 2026

notgitika requested a deployment to e2e-testing June 18, 2026 18:30 — with GitHub Actions In progress

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Jun 18, 2026

avi-alpert reviewed Jun 18, 2026

View reviewed changes

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Jun 18, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: cover payments, policies/guardrails, insights, harness, datasets, web-search#1577

docs: cover payments, policies/guardrails, insights, harness, datasets, web-search#1577
notgitika wants to merge 3 commits into
aws:mainfrom
notgitika:docs/feature-coverage-update

notgitika commented Jun 18, 2026

Uh oh!

agentcore-devx-automation Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

avi-alpert Jun 18, 2026

Uh oh!

notgitika Jun 18, 2026

Uh oh!

notgitika Jun 18, 2026

Uh oh!

notgitika Jun 18, 2026

Uh oh!

agentcore-devx-automation Bot commented Jun 18, 2026

Uh oh!

avi-alpert Jun 18, 2026

Uh oh!

notgitika Jun 18, 2026

Uh oh!

notgitika Jun 18, 2026

Uh oh!

avi-alpert Jun 18, 2026

Uh oh!

agentcore-devx-automation Bot commented Jun 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

notgitika commented Jun 18, 2026

Why

What changed

README.md

src/assets/agents/AGENTS.md (vended to every customer project)

Snapshot

Out of scope (filed as follow-ups)

Verification

Uh oh!

agentcore-devx-automation Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Package Tarball

How to install

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agentcore-devx-automation Bot commented Jun 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agentcore-devx-automation Bot commented Jun 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

`README.md`

`src/assets/agents/AGENTS.md` (vended to every customer project)