Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ To use a skill in an experiment, reference its directory name in the experiment'

Both runtimes load skills lazily ([progressive disclosure](https://ai-sdk.dev/cookbook/guides/agent-skills)): only each skill's name+description is in the system prompt, and the agent pulls a skill's full instructions on demand. They differ only in how the body is fetched, because the tools-mode agent has no filesystem:

- **Local-stack (sandbox) mode:** skills are installed into the workspace with [Vercel's `skills` CLI](https://github.com/vercel-labs/skills) (baked into the sandbox image, sourced from the local `skills/` directory — never the network) under `.claude/skills/`. When a task matches, the agent reads `.claude/skills/<name>/SKILL.md` (and any files it references) with its file tools.
- **Local-stack (sandbox) mode:** skills are installed into the workspace with [Vercel's `skills` CLI](https://github.com/vercel-labs/skills) (baked into the sandbox image, sourced from the local `skills/` directory — never the network), into every project scope the CLI harnesses discover natively: `.claude/skills/` for Claude Code, `.agents/skills/` for Codex and OpenCode. Each CLI discovers the scope it reads and surfaces those skills to the model itself. The framework also still injects a listing naming them, so a CLI harness hears about them twice; removing that is a separate change.
- **Tools mode:** no filesystem, so a `load_skill` tool returns a skill's full instructions when the agent calls it with the skill's name.

## Framework Checks
Expand Down
2 changes: 2 additions & 0 deletions apps/framework/harness/run-eval.ts
Original file line number Diff line number Diff line change
Expand Up @@ -420,6 +420,7 @@ async function runOne(
: undefined;
await using session = disposable(
await exp.localStack.startSession({
agent: exp.agent.id,
cliVersion: ev.metadata.cliVersion,
localDir: ev.localDir,
includeServices: ev.metadata.services,
Expand Down Expand Up @@ -494,6 +495,7 @@ async function runOne(
await using cliSandbox = agentRunsInSandbox
? disposable(
await createBareSandbox({
agent: exp.agent.id,
skills: skillSources,
mounts: supabaseMcpServerMounts(),
})
Expand Down
5 changes: 5 additions & 0 deletions packages/core/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -430,6 +430,11 @@ export type SandboxMount = {
};

export type LocalStackSessionArgs = {
/**
* The agent harness this session serves. The prompt addendum builders take it
* but do not branch on it yet; a follow-up change gates their output on it.
*/
agent: AgentHarnessId;
/** Supabase CLI version this scenario requires, overriding the runtime default. */
cliVersion?: string;
/**
Expand Down
5 changes: 3 additions & 2 deletions packages/sandbox/src/agent-environment.ts
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ export interface AgentEnvironmentOptions {
cliVersion?: string;
/** Host directory whose contents seed the workspace. */
localDir?: string;
/** Skills to install into the sandbox (the agent reads them with its file tools). */
/** Skills to install into the sandbox (in every CLI harness's project scope). */
skills?: readonly SkillSource[];
/**
* Run the Supabase local stack. Present → local-stack mode; omitted → tools
Expand Down Expand Up @@ -92,7 +92,8 @@ export async function createAgentEnvironment(
} else if (options.localDir) {
await sandbox.copyToContainer(options.localDir, sandbox.workdir);
}
// Skills are installed in both modes; the agent reads SKILL.md with its file tools.
// Skills are installed in both modes, into every CLI harness's native
// project scope (see installSkills) so each agent discovers them itself.
const skills = await installSkills(sandbox, options.skills ?? []);
return { sandbox, skills, close: () => sandbox.stop() };
} catch (err) {
Expand Down
24 changes: 18 additions & 6 deletions packages/sandbox/src/bare-sandbox.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import type {
AgentHarnessId,
AgentSandbox,
SandboxMount,
SkillSource,
Expand All @@ -14,6 +15,21 @@ export interface BareSandboxHandle {
close(): Promise<void>;
}

export interface BareSandboxOptions {
/** Harness driving this sandbox. */
agent: AgentHarnessId;
/** Supabase CLI version baked into the sandbox image. */
cliVersion?: string;
/** Skills to install into the sandbox. */
skills?: readonly SkillSource[];
/**
* Extra host directories bind-mounted into the sandbox (read-only by
* default) — e.g. a local MCP server build the in-container agent must be
* able to launch. See `supabaseMcpServerMounts`.
*/
mounts?: readonly SandboxMount[];
}

/**
* The agent's execution environment for tools mode: the shared agent
* environment (image, tooling, skills) **without** the Supabase local stack.
Expand All @@ -24,11 +40,7 @@ export interface BareSandboxHandle {
* platform-lite via `host.docker.internal` on the default bridge).
*/
export async function createBareSandbox(
options: {
cliVersion?: string;
skills?: readonly SkillSource[];
mounts?: readonly SandboxMount[];
} = {}
options: BareSandboxOptions
): Promise<BareSandboxHandle> {
const env = await createAgentEnvironment({
cliVersion: options.cliVersion,
Expand All @@ -37,7 +49,7 @@ export async function createBareSandbox(
});
return {
sandbox: toAgentSandbox(env.sandbox),
promptAddendum: buildSkillsPrompt(env.skills),
promptAddendum: buildSkillsPrompt(options.agent, env.skills),
close: env.close,
};
}
6 changes: 5 additions & 1 deletion packages/sandbox/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,21 +19,25 @@ export type { SetupSupabaseSandboxOptions } from './supabase.js';
export {
buildLocalStackScoringContext,
buildLocalStackTools,
buildToolSurfaceAddendum,
localStackRuntime,
toAgentSandbox,
} from './local-stack-runtime.js';
export type { LocalStackRuntimeOptions } from './local-stack-runtime.js';
export {
SKILLS_CLI_VERSION,
SKILLS_INSTALL_AGENTS,
SKILLS_INSTALL_DIR,
SKILLS_INSTALL_DIRS,
buildSkillsAddCommand,
buildSkillsPrompt,
frontmatterDescription,
installSkills,
stripFrontmatter,
} from './skills.js';
export type { SkillEntry } from './skills.js';
export { createBareSandbox } from './bare-sandbox.js';
export type { BareSandboxHandle } from './bare-sandbox.js';
export type { BareSandboxHandle, BareSandboxOptions } from './bare-sandbox.js';
export { createAgentEnvironment } from './agent-environment.js';
export type {
AgentEnvironment,
Expand Down
39 changes: 27 additions & 12 deletions packages/sandbox/src/local-stack-runtime.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import { jsonSchema, tool, type ToolSet } from 'ai';
import { createClient } from '@supabase/supabase-js';
import {
supabaseMcpServer,
type AgentHarnessId,
type AgentSandbox,
type HostedLink,
type LocalStackRuntime,
Expand Down Expand Up @@ -73,6 +74,7 @@ export function localStackRuntime(
return {
id: 'local-stack',
async startSession({
agent,
cliVersion,
localDir,
includeServices,
Expand Down Expand Up @@ -105,22 +107,14 @@ export function localStackRuntime(

const mcpServers = await resolveMcpServers(options, hosted);

let baseAddendum =
'docker, psql, git, and curl are installed in the workspace. ' +
'Use the bash tool to run commands (the working directory is always the workspace root) ' +
'and the files tools to inspect and modify files.';

if (!skipCliInstall) {
baseAddendum = 'The Supabase CLI (`supabase`), ' + baseAddendum;
baseAddendum +=
' Services started with `supabase start` are reachable on their default 127.0.0.1 ports.';
}

return {
tools: buildLocalStackTools(sandbox),
sandbox: toAgentSandbox(sandbox),
mcpServers,
promptAddendum: [baseAddendum, buildSkillsPrompt(env.skills)]
promptAddendum: [
buildToolSurfaceAddendum(agent, { skipCliInstall }),
buildSkillsPrompt(agent, env.skills),
]
.filter(Boolean)
.join('\n\n'),
scoringContext: buildLocalStackScoringContext(sandbox, hosted),
Expand All @@ -135,6 +129,27 @@ export function localStackRuntime(
};
}

/**
* Describes the session's tool surface: the binaries installed in the workspace
* and the in-process `bash`/`files_*` tools from `buildLocalStackTools`.
*/
export function buildToolSurfaceAddendum(
agent: AgentHarnessId,
options: { skipCliInstall?: boolean } = {}
): string {
let addendum =
'docker, psql, git, and curl are installed in the workspace. ' +
'Use the bash tool to run commands (the working directory is always the workspace root) ' +
'and the files tools to inspect and modify files.';

if (!options.skipCliInstall) {
addendum = 'The Supabase CLI (`supabase`), ' + addendum;
addendum +=
' Services started with `supabase start` are reachable on their default 127.0.0.1 ports.';
}
return addendum;
}

/**
* Build the MCP server map for a session. An explicit `options.mcpServers`
* wins. Otherwise, when the eval links to a hosted project, expose a Supabase
Expand Down
Loading