Skip to content

Feat: Add prompt caching to bedrock models #270

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sQu4rks wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Feat: Add prompt caching to bedrock models #270

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d808c2c
feat(bb-agent): opt-in Bedrock prompt caching via ModelConfig.cacheCo…
Jul 27, 2026
972da1b
Merge branch 'aws-devtools-labs:main' into feat/bb-agent-prompt-caching
sQu4rks Jul 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .changeset/bb-agent-prompt-caching.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
---
"@aws-blocks/bb-agent": minor
---

feat(bb-agent): opt-in Bedrock prompt caching via `ModelConfig.cacheConfig`

Agents can now enable Strands prompt caching for the `bedrock` provider by
setting `cacheConfig: { strategy: 'auto' | 'anthropic' }` on a model config.
Caching reuses the cached request prefix (tools + system prompt + prior turns)
across requests, cutting input-token cost and latency for agents with long
system prompts, many tools, or multi-turn conversations.

Off by default and additive — existing configs are unaffected. Use
`'auto'` for the `BedrockModels` presets (Strands places cache points for
known model IDs) and `'anthropic'` when `modelId` is an ARN application
inference profile that `'auto'` can't detect. Ignored by the `openai-api`
and `canned` providers.
2 changes: 2 additions & 0 deletions packages/bb-agent/API.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,8 @@ export interface Message {
// @public (undocumented)
export interface ModelConfig {
apiKey?: string | (() => Promise<string>);
// Warning: (ae-forgotten-export) The symbol "CacheConfig" needs to be exported by the entry point index.aws.d.ts
cacheConfig?: CacheConfig;
// (undocumented)
endpoint?: string;
// Warning: (ae-forgotten-export) The symbol "GuardrailsConfig" needs to be exported by the entry point index.aws.d.ts
Expand Down
16 changes: 16 additions & 0 deletions packages/bb-agent/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,7 @@ Model configuration is optional. When omitted, the agent defaults to `BedrockMod
| `endpoint` | `string` | API endpoint. For openai-api (defaults to api.openai.com). |
| `apiKey` | `string \| () => Promise<string>` | API key for openai-api. Accepts a string or async resolver. Falls back to `OPENAI_API_KEY` env var. |
| `inferenceConfig` | `{ temperature?, topP?, maxTokens?, stopSequences? }` | Optional inference parameters. |
| `cacheConfig` | `{ strategy: 'auto' \| 'anthropic' }` | Optional prompt caching (bedrock only). See [Prompt Caching](#prompt-caching). |

```typescript
import { Agent } from '@aws-blocks/bb-agent';
Expand Down Expand Up @@ -217,6 +218,21 @@ Override inference settings with spread:
model: { deployed: { ...BedrockModels.BALANCED, inferenceConfig: { temperature: 0.9, maxTokens: 8192 } } }
```

#### Prompt Caching

Prompt caching reuses the cached request prefix (tools + system prompt + prior turns) across requests, cutting input-token cost and latency — especially for agents with long system prompts, many tools, or multi-turn conversations. It's off by default and applies only to the `bedrock` provider (ignored by `openai-api` and `canned`).

```typescript
model: { deployed: { ...BedrockModels.BALANCED, cacheConfig: { strategy: 'auto' } } }
```

| Strategy | When to use |
|----------|-------------|
| `'auto'` | Recommended default. Strands places cache points automatically for known Bedrock model IDs — including all [`BedrockModels`](#bedrock-presets) presets. |
| `'anthropic'` | Use when `modelId` is an ARN application inference profile, where `'auto'` cannot detect the underlying model and won't enable caching. Same performance, no extra permissions. |

> Caching has per-model minimum token thresholds (e.g. ~1,024 tokens for Claude Sonnet) and cache entries expire after ~5 minutes of inactivity. See the [Strands caching docs](https://strandsagents.com/docs/user-guide/concepts/model-providers/amazon-bedrock/#caching).

#### Ollama Presets

Convenience shortcuts for local development using [Ollama](https://ollama.com/). Requires Ollama installed and running (`ollama serve`), model pulled (`ollama pull <model-id>`). Uses the default endpoint `http://localhost:11434/v1`.
Expand Down
13 changes: 13 additions & 0 deletions packages/bb-agent/src/index.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1041,6 +1041,19 @@ describe('BedrockModels presets', () => {
const model = await createStrandsModel(BedrockModels.BALANCED);
assert.ok(model, 'should create a model instance');
});

test('cacheConfig flows through createStrandsModel to BedrockModel', async () => {
const model = await createStrandsModel({ ...BedrockModels.BALANCED, cacheConfig: { strategy: 'auto' } });
assert.ok(model, 'should create a model instance');
const config = (model as { getConfig(): { cacheConfig?: { strategy: string } } }).getConfig();
assert.deepStrictEqual(config.cacheConfig, { strategy: 'auto' }, 'cacheConfig should reach the BedrockModel');
});

test('omitting cacheConfig leaves caching off', async () => {
const model = await createStrandsModel(BedrockModels.BALANCED);
const config = (model as { getConfig(): { cacheConfig?: { strategy: string } } }).getConfig();
assert.strictEqual(config.cacheConfig, undefined, 'caching should be off by default');
});
});

describe('OllamaModels presets', () => {
Expand Down
1 change: 1 addition & 0 deletions packages/bb-agent/src/model-factory.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,7 @@ export async function createStrandsModel(config?: ModelConfig, log?: ChildLogger
maxTokens: config.inferenceConfig.maxTokens,
stopSequences: config.inferenceConfig.stopSequences,
}),
...(config.cacheConfig && { cacheConfig: config.cacheConfig }),
});
}

Expand Down
24 changes: 24 additions & 0 deletions packages/bb-agent/src/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,30 @@ export interface ModelConfig {
apiKey?: string | (() => Promise<string>);
inferenceConfig?: InferenceConfig;
guardrails?: GuardrailsConfig;
/**
* Prompt caching for the `bedrock` provider. Off by default. Caching reuses the
* cached prefix (tools + system prompt + prior turns) across requests, cutting
* input-token cost and latency on multi-turn or long-system-prompt agents.
* Ignored by the `openai-api` and `canned` providers.
*
* - `{ strategy: 'auto' }` — let Strands place cache points at optimal positions
* (after tools, after the last user message) for known Bedrock model IDs. This is
* the recommended default for the {@link BedrockModels} presets.
* - `{ strategy: 'anthropic' }` — force-enable Anthropic-style caching. Use this
* when `modelId` is an ARN application inference profile, where `'auto'` cannot

@Simone319 Simone319 Jul 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happen if the strategy is 'anthropic' but the model id is not anthropic? Shall we place a check on modelId for this strategy?

@sQu4rks sQu4rks Aug 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the comments @Simone319

I believe the strategy: anthropic is specifically designed for cases where - from the ARN - it's unclear that the underlying model is an anthropic model. The ARN could look something like this arn:aws:bedrock:{region}:{account-number}:inference-profile/my-team-profile (docs) so we'd need another API call (and accompanying permissions) to retrieve the underlying model (via GetInferenceProfile (api docs) to then parse the models.

I also checked the case quickly via a script (see below). Since cacheConfig is passed to the Strands BedrockModel it's behavior is entirely strands.

import { Agent } from '@strands-agents/sdk';
import { createStrandsModel } from '../../dist/model-factory.js';

process.env.AWS_REGION ??= 'us-east-1';
const MODEL_ID = process.env.MODEL_ID ?? 'openai.gpt-oss-120b-1:0'; // a non-Anthropic model

// Long, stable prefix so we clear the ~1,024-token cache minimum.
const SYSTEM_PROMPT = [
  'You are "Atlas", a meticulous senior support engineer for the ACME Cloud platform.',
  'Follow these operating rules on every single response, without exception:',
  ...Array.from({ length: 40 }, (_, i) =>
    `Rule ${i + 1}: Be precise, cite the relevant ACME Cloud service by name, prefer the ` +
    'least-privilege remediation, never invent API names, and always end with a one-line summary.'),
].join('\n');

const TURNS = [
  'A user reports 503s from EdgeCDN after a deploy. First triage step?',
  'Now they also see elevated latency on ObjectStore. What next?',
];

const usageOf = (r) => {
  const u = r.metrics?.accumulatedUsage ?? {};
  return { input: u.inputTokens ?? 0, output: u.outputTokens ?? 0, cacheWrite: u.cacheWriteInputTokens ?? 0, cacheRead: u.cacheReadInputTokens ?? 0 };
};

async function runScenario(label, cacheConfig) {
  console.log(`\n--- ${label} ---`);
  try {
    const model = await createStrandsModel({ provider: 'bedrock', modelId: MODEL_ID, cacheConfig });
    const agent = new Agent({ model, systemPrompt: SYSTEM_PROMPT, printer: false });
    for (let i = 0; i < TURNS.length; i++) {
      const u = usageOf(await agent.invoke(TURNS[i]));
      console.log(`  turn ${i + 1}: input=${u.input} output=${u.output} cacheWrite=${u.cacheWrite} cacheRead=${u.cacheRead}`);
    }
  } catch (err) {
    console.log(`  ERROR: ${err?.name ?? 'Error'}: ${err?.message ?? err}`);
  }
}

// 'auto'      → Strands warns and disables caching for a non-Anthropic model
await runScenario("strategy: 'auto'", { strategy: 'auto' });
// 'anthropic' → forces Anthropic-style cache points onto a non-Anthropic model
await runScenario("strategy: 'anthropic'", { strategy: 'anthropic' });

With strategy: 'anthropic' on a non-Anthropic model, Strands forwards the Anthropic-style cache points and Bedrock rejects the request with a clear ModelError ("You invoked an unsupported model or your request did not allow prompt caching").

--- strategy: 'auto' ---
model_id=<openai.gpt-oss-120b-1:0> | cache_config is enabled but this model does not support automatic caching
  turn 1: input=1679 output=533 cacheWrite=0 cacheRead=0
  turn 2: input=3375 output=1558 cacheWrite=0 cacheRead=0

--- strategy: 'anthropic' ---
  ERROR: ModelError: You invoked an unsupported model or your request did not allow prompt caching. See the documentation for more information.

So my suggestion would be to handle the warnings/errors but let Bedrock do the verification.

* detect the underlying model and therefore won't enable caching.
*
* Caching has per-model minimum token thresholds (e.g. ~1,024 for Claude Sonnet)
* and cache entries expire after ~5 minutes of inactivity.
*
* @see https://strandsagents.com/docs/user-guide/concepts/model-providers/amazon-bedrock/#caching
*/
cacheConfig?: CacheConfig;
}

/** Prompt-caching strategy for the `bedrock` provider. @see {@link ModelConfig.cacheConfig} */
export interface CacheConfig {
strategy: 'auto' | 'anthropic';
}

export interface InferenceConfig {
Expand Down