diff --git a/CHANGELOG.md b/CHANGELOG.md index 511074ff..0294ba12 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,23 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Security + +- Recognized secrets are replaced with typed tokens before admitted records enter the archive, index, embeddings or summaries. The rules cover common provider keys, private keys, JWTs, connection strings and secret-named configuration fields. Record exclusions apply first, source transcripts stay unchanged, and safe neighboring records retain their original line numbers. Redaction is on by default and stops ingestion if its rules cannot load. See `docs/REDACTION.md` for coverage and limits. +- With redaction on, summaries use the admitted transcript instead of resuming the original session. Codex conversations stay on Codex; failures do not switch to Claude. Claude conversations keep their transcript-text route. + +### Added + +- `episodic-memory redact --rewrite` cleans data you indexed before upgrading. It redacts the archive and the search index in place, re-embeds only the messages that changed, and deletes summaries built from unredacted text so they regenerate. Use `--dry-run` to preview (it writes nothing, not even a schema migration), and add `--report` to list each value it would redact, by location, rule and shape, without printing the value. +- Custom redaction rules via `~/.config/superpowers/redaction-rules.json`, which extends the bundled defaults. Try rules with `episodic-memory redact --stdin`. See `docs/REDACTION.md`. +- New settings: `EPISODIC_MEMORY_REDACTION` (`on`/`off`), `EPISODIC_MEMORY_REDACTION_RULES`, and `EPISODIC_MEMORY_REDACTION_STRICT`. + +### Changed + +- `episodic-memory index` now parses the archived copy instead of the source transcript, and refreshes that copy when the source has grown. This is the same behavior `sync` already had. + ## [1.6.0] - 2026-09-08 Adds a fifth conversation source, an off switch for automatic syncing, and two fixes for real-world resource problems. diff --git a/README.md b/README.md index ee952317..6d258573 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,21 @@ Semantic search for Claude Code, Codex, Cursor, opencode, and Oh My Pi (OMP) conversations. Remember past discussions, decisions, and patterns. +## Recall admission + +Search rechecks the current record-exclusion policy and configured credential scanner, +including rows indexed before the policy changed. Excluded hits do not consume the +requested result limit. A stale whole-conversation summary is omitted if any of its +source records or tool calls is excluded; safe neighboring exchanges remain searchable. + +Conversation opt-out markers are read from decoded user text, including JSON escapes. +Assistant quotations and tool outputs do not opt out the conversation. Tool-result +blocks inside a user envelope are still tool output. Malformed legacy text retains +conservative marker handling, and unreadable or oversized records fail closed. +Ranged reads retain physical line coordinates without materializing unrelated message bodies. +Original transcripts are never rewritten. Source and policy changes during a search +abort the result rather than returning a partially validated view. + ## Testimonial From an AI coding assistant's perspective: @@ -214,7 +229,9 @@ In Codex and opencode, the skill guides the agent to use the episodic-memory MCP ## API Configuration -By default, episodic-memory uses your Claude Code authentication for Claude Code summarization. Codex-indexed sessions with a session ID are summarized through `codex app-server` by creating an ephemeral `thread/fork`, so the summary can use Codex session context and reasoning summaries without appending to the original rollout. +By default, episodic-memory uses your Claude Code authentication for Claude Code summarization. Codex-indexed sessions with a session ID are summarized through `codex app-server` by creating an ephemeral `thread/fork`, so the summary can use Codex session context and reasoning summaries without appending to the original rollout. A Codex transcript without a session ID is summarized from its admitted text in an isolated ephemeral `thread/start`; it is never sent to Claude. + +Claude summaries preserve the user-level `apiKeyHelper` and `env.ANTHROPIC_BASE_URL` from the selected Claude configuration directory. The helper remains with Claude Code; episodic-memory does not retrieve or store its credential. Explicit environment and episodic endpoint overrides take precedence. Hooks, plugins, project settings and MCP servers remain excluded from the summarizer. Missing settings leave native environment or OAuth authentication available; unreadable or malformed settings stop the call with a message that omits their contents. To route summarization through a custom Anthropic-compatible endpoint or override the model: @@ -222,7 +239,7 @@ To route summarization through a custom Anthropic-compatible endpoint or overrid # Override model (default: haiku) export EPISODIC_MEMORY_API_MODEL=opus -# Override fallback model on error (default: sonnet) +# Override the Claude fallback model on error (default: sonnet; not used for Codex) export EPISODIC_MEMORY_API_MODEL_FALLBACK=sonnet # Route through custom endpoint @@ -294,7 +311,7 @@ These pass through unchanged to episodic-memory's summarizer subprocess. Because These settings only affect episodic-memory's summarization calls, not your interactive Claude Code or Codex sessions. -Codex summarization requires `codex-cli 0.130.0` or newer. If Codex app-server summarization is unavailable, sync logs the reason and falls back to transcript-text summarization. +Codex summarization requires `codex-cli 0.130.0` or newer; isolated transcript-only summaries require `0.154.0` or newer. If the applicable Codex route is unavailable, sync records a retryable summary failure rather than switching to Claude or a metered Anthropic API. Exchanges can still be indexed and searched without a generated summary. ### What's Affected @@ -327,6 +344,18 @@ Add to `.claude/hooks/session-end`: episodic-memory sync ``` +### `episodic-memory redact` + +```bash +episodic-memory redact --rewrite --dry-run # what would change +episodic-memory redact --rewrite --dry-run --report # each hit: location, rule, value shape +episodic-memory redact --rewrite # redact the existing archive + index in place +episodic-memory redact --stdin < file.jsonl # try the rules on some text +episodic-memory redact --print-default-rules +``` + +See [docs/REDACTION.md](docs/REDACTION.md). + ### `episodic-memory stats` Display index statistics including conversation counts, date ranges, and project breakdown. @@ -372,9 +401,14 @@ This test uses your normal Claude Code auth and writes small test transcripts to Manual indexing tools for bulk operations and maintenance. See `episodic-memory index --help` for full options. Common operations: -- `--cleanup` - Index all unprocessed conversations -- `--verify` - Check index health -- `--repair` - Fix detected issues +- `--cleanup` - Index unprocessed exchanges; summaries may call an AI provider unless disabled +- `--verify` - Distinguish missing summaries, unindexed archives, appended exchanges, and archive mtime refreshes +- `--repair` - Re-embed and regenerate summaries for repair candidates; review the scope and provider cost first + +An archive mtime refresh without new exchanges is reported separately and is +not an outdated-index repair target. An unindexed archive is reported, but +that status alone does not select it for `--repair`; a separate overlapping +repair category may still cause it to be processed. ### `episodic-memory search` @@ -415,16 +449,39 @@ open output.html 4. **Index** - Stores in SQLite with sqlite-vec for fast similarity search 5. **Search** - Semantic search using vector similarity or exact text matching +## Secret Redaction + +Recognized secrets in your conversations (Azure client secrets, storage keys, SAS +signatures, connection-string passwords, private keys, JWTs, provider API keys, +`password: …` assignments) are replaced with typed tokens such as +`[REDACTED:azure-storage-key]` **before** anything is archived, indexed, +embedded, or sent to the summarizer. Shape rules preserve Git SHAs and GUIDs; +a value in a secret-named field is still redacted. The tokens are searchable too. + +Redaction is on by default and fails closed: if the rules can't be loaded, sync +won't run. After upgrading, run `episodic-memory redact --rewrite` once to clean +data you indexed before. Record exclusions and screening apply before redaction; +disabling redaction does not disable them. Summaries keep the original provider. + +| Variable | Default | Meaning | +|---|---|---| +| `EPISODIC_MEMORY_REDACTION` | `on` | `off` disables redaction | +| `EPISODIC_MEMORY_REDACTION_RULES` | `/redaction-rules.json` | Custom rules file (extends the defaults) | +| `EPISODIC_MEMORY_REDACTION_STRICT` | `1` | `0` continues unredacted (with a warning) when rules fail to load | + +See [docs/REDACTION.md](docs/REDACTION.md) for the rule list, the custom-rules +format, and what is and isn't covered. + ## Excluding Conversations -Conversations containing this marker anywhere in their content will be archived but not indexed: +Conversations with this marker in a decoded user instruction are excluded before new archive copying, parsing, and indexing: ``` DO NOT INDEX THIS CHAT ``` **Automatic exclusions:** -- Conversations where Claude generates summaries (marker in system prompt) +- Summarizer-generated conversations carrying the marker in their prompt - Meta-conversations about conversation processing **Use cases:** @@ -433,7 +490,117 @@ Conversations containing this marker anywhere in their content will be archived - Test or experimental sessions - Any conversation you don't want searchable -The marker can appear in any message (user or assistant) and excludes the entire conversation from the search index. +The marker in user instructions prevents new derived copies and indexing; an assistant quotation or tool result does not opt out the conversation. It does not delete an existing archive or index entry. + +### Persistent record exclusions + +`record-exclusions.json` in the existing configuration directory defines exact +records that must not enter new derived archives, parser output, summaries, or +database insertions. It is independent of `CONVERSATION_SEARCH_EXCLUDE_PROJECTS`; +that environment variable cannot replace or disable record exclusions. + +```json +{ + "version": 1, + "exchanges": [ + { + "session_id": "00000000-1111-4222-8333-444444444444", + "transcript": "rollout-00000000-1111-4222-8333-444444444444.jsonl", + "line_start": 4, + "line_end": 5 + } + ], + "tool_calls": [ + { + "session_id": "00000000-1111-4222-8333-444444444444", + "transcript": "rollout-00000000-1111-4222-8333-444444444444.jsonl", + "call_id": "call-example" + } + ] +} +``` + +Line ranges are inclusive physical JSONL coordinates in the original transcript. +Excluded records become empty lines in a derived copy, preserving later line +numbers. Session identity, exact transcript basename and native tool-call identity, not +a machine-specific absolute path, bind the policy. Sibling subagent transcripts +sharing a parent session are separate subjects. Safe text blocks remain when only a tool block +is excluded. Original transcripts are never changed. + +Malformed, unreadable, symlinked or oversized policy files reject admission. +When a policy is present, unchanged source timestamps do not bypass a fresh +admission decision. Source/archive aliasing and changing input are rejected. +The marker scan and record stream use the same descriptor; source identity is +checked during and after streaming. Direct database insertion checks both the +transcript identity and historical session metadata, which may name an ancestor. +The foreground sync CLI exits nonzero for per-file admission errors and does not +claim successful synchronization or start embedding migration after those errors. +Full, session and incremental indexing, repair, MCP reads and the display CLI use +the same admission boundary. Display pagination retains physical line numbers. +An absent policy retains ordinary behavior; deleting the policy intentionally +removes its protection, so maintain it as private operator configuration. + +Summarization with a record policy uses admitted transcript text, never a resumed +source session. Codex retains its selected provider through an ephemeral fresh +context and fails rather than silently falling back to another provider. Claude +receives the admitted text without session resume. Policy-enabled Codex summaries +require Codex 0.154.0 or newer. The isolated summary path disables inherited +tools, plugins, hooks, and skills, and refuses to proceed if the configured +MCP or skill boundary cannot be established. It uses a temporary empty working +directory; provider error payloads are suppressed and the directory is removed +after process termination. + +### Optional local secret screening + +Add an optional `screening` object to the same policy to reject newly detected +content before derived archive writes, reads, summaries and database insertion: + +```json +"screening": { + "engine": "gitleaks", + "executable": "/absolute/resolved/path/to/gitleaks", + "sha256": "<64 lowercase hex characters from the trusted executable>", + "timeout_ms": 2000, + "max_record_bytes": 1048576 +} +``` + +Keep the existing `version`, `exchanges` and `tool_calls` fields. Install Gitleaks +through a trusted package source, resolve its executable symlink and verify its provenance before pinning its +SHA-256. On migration or upgrade, explicitly replace the host-local path and +digest after verifying the new executable. The scanner must be a regular, +executable file owned by the current user or root and not group/world-writable. +Do not copy credential stores or an old host's executable-path assumption. + +Exact exclusions run first. Each remaining physical record is screened locally +with the pinned Gitleaks built-in rules. Nested JSON strings are decoded while +retaining key/value assignment context; separate exchange messages are screened +separately. A finding rejects the operation rather than silently dropping a new +message and breaking conversation pairing. Missing/changed executables, byte or +nesting limits, timeouts and malformed or contradictory reports fail closed. +Verification reports scanner rejections and scanner unavailability separately +from transcript corruption, even when a summary is missing. A broad repair +refuses to start while either screening failure remains. +No findings, input, scanner diagnostics or secret fingerprints appear in errors. +The scanner receives stdin in a private empty working directory with no inherited +environment/configuration and with inline allow comments disabled. Its temporary +directory is removed after execution. Original transcripts and previously +published archives remain unchanged when admission fails. + +Screening launches a bounded subprocess per record; it is not an unbounded bulk +collection service. Measure a finite representative batch before large imports. +No automatic sync or background job is enabled by this setting. Before enabling +it, replace all policy consumers with this version through their native lifecycle: +older strict-schema consumers reject the additional field. Installing a plugin +does not replace already-loaded code, so do not enable the field while protected +old consumers still require the previous schema. + +Without `screening`, this policy remains an exact exclusion mechanism. Even with +screening, pattern matching does not prove universal unknown-secret absence, +purge preexisting derived data, rewrite original +sessions, or retroactively filter already-loaded code. Old randomly generated +tool-row IDs need verified native source-call mapping before policy migration; +do not broaden those entries to whole sessions or claim a complete migration. ## MCP Server diff --git a/cli/episodic-memory.js b/cli/episodic-memory.js index 08ebccc8..45c61da1 100755 --- a/cli/episodic-memory.js +++ b/cli/episodic-memory.js @@ -44,6 +44,7 @@ COMMANDS: stats Show index statistics doctor Diagnose Claude Code or Codex integration issues import-cursor-history Export legacy Cursor conversations from state.vscdb for indexing + redact Re-run secret redaction over the archive and index (--rewrite) Run 'episodic-memory --help' for command-specific help. @@ -94,6 +95,10 @@ async function main() { await runScript(join(distDir, 'cursor-import-cli.js'), args); break; + case 'redact': + await runScript(join(distDir, 'redact-cli.js'), args); + break; + case '--help': case '-h': case undefined: diff --git a/cli/index-conversations.js b/cli/index-conversations.js index 4d5f5bf5..77ea6f10 100755 --- a/cli/index-conversations.js +++ b/cli/index-conversations.js @@ -50,16 +50,16 @@ USAGE: COMMANDS: (default) Index all conversations - --cleanup Process only unindexed conversations (fast, cheap) + --cleanup Process unindexed exchanges (may generate AI summaries) --session ID Index specific session (used by hook) - --verify Check index health - --repair Fix detected issues + --verify Classify index and summary state + --repair Re-embed and summarize repair candidates (review first) --rebuild Delete DB and re-index everything (requires confirmation) OPTIONS: --concurrency N Parallel summarization (1-16, default: 1) -c N Short form of --concurrency - --no-summaries Skip AI summary generation (free, but no summaries in results) + --no-summaries Skip AI summary provider calls --help, -h Show this help EXAMPLES: @@ -69,13 +69,13 @@ EXAMPLES: # Index with 8 parallel summarizations (8x faster) index-conversations --cleanup --concurrency 8 - # Index without AI summaries (free, fast) + # Index without AI summary provider calls index-conversations --cleanup --no-summaries # Check index health index-conversations --verify - # Fix any issues found + # Review candidates and provider cost before broad repair index-conversations --repair # Nuclear option (deletes everything, re-indexes) @@ -85,7 +85,7 @@ WORKFLOW: 1. Initial setup: index-conversations --cleanup 2. Ongoing: Auto-indexed by sessionEnd hook 3. Health check: index-conversations --verify (weekly) - 4. Recovery: index-conversations --repair (if issues found) + 4. Recovery: review exact verifier categories before repair SEE ALSO: INDEXING.md - Setup and maintenance guide diff --git a/dist/cursor-legacy.d.ts b/dist/cursor-legacy.d.ts index a035ba0f..77340197 100644 --- a/dist/cursor-legacy.d.ts +++ b/dist/cursor-legacy.d.ts @@ -1,3 +1,4 @@ +import { type Redactor } from './redaction.js'; export declare function getDefaultCursorVscdbPath(): string | undefined; /** * Collect composer IDs that already have live agent transcripts under @@ -14,6 +15,8 @@ export interface CursorLegacyImportOptions { force?: boolean; /** Report what would be exported without writing files. */ dryRun?: boolean; + /** `undefined` loads from the environment (strict mode throws on bad rules); `null` disables. */ + redactor?: Redactor | null; } export interface CursorLegacyImportResult { exported: number; diff --git a/dist/cursor-legacy.js b/dist/cursor-legacy.js index 811e5637..f977df10 100644 --- a/dist/cursor-legacy.js +++ b/dist/cursor-legacy.js @@ -3,6 +3,7 @@ import os from 'os'; import path from 'path'; import Database from 'better-sqlite3'; import { detectCursorCwd } from './parser.js'; +import { loadRedactor, redactJsonlLine } from './redaction.js'; /** * Import legacy Cursor conversations from Cursor's global SQLite store * (state.vscdb) into JSONL files compatible with the Cursor transcript parser. @@ -111,6 +112,7 @@ export function importCursorLegacy(options) { skippedEmpty: 0, errors: [], }; + const redactor = options.redactor === undefined ? loadRedactor() : options.redactor; const liveIds = options.liveTranscriptIds ?? new Set(); const db = new Database(options.dbPath, { readonly: true, fileMustExist: true }); try { @@ -200,13 +202,18 @@ export function importCursorLegacy(options) { if (!options.dryRun) { // Re-serialize with cwd now that it's known (it's derived from the // whole conversation's tool calls). - const finalLines = cwd + const withCwd = cwd ? lines.map(line => JSON.stringify({ ...JSON.parse(line), cwd })) : lines; + // The export dir is a plugin-owned plaintext copy, so redact it at + // write time like the archive (docs/REDACTION.md). + const finalLines = redactor + ? withCwd.map(line => redactJsonlLine(line, redactor, { source: 'cursor-legacy', path: outFile })) + : withCwd; fs.mkdirSync(path.dirname(outFile), { recursive: true }); fs.writeFileSync(outFile, finalLines.join('\n') + '\n', 'utf-8'); // Stamp the conversation's end time so mtime-based fallbacks and - // copyIfNewer comparisons reflect when it actually happened. + // archive freshness comparisons reflect when it actually happened. const mtime = toIsoTimestamp(composer.lastUpdatedAt) ?? lastTimestamp; const mtimeDate = new Date(mtime); fs.utimesSync(outFile, mtimeDate, mtimeDate); diff --git a/dist/db.d.ts b/dist/db.d.ts index 3f590fff..9d8ed98a 100644 --- a/dist/db.d.ts +++ b/dist/db.d.ts @@ -13,6 +13,12 @@ export declare function migrateSchema(db: Database.Database): void; * 3. Recreates the table with ON DELETE CASCADE and copies surviving rows. */ export declare function migrateToolCallsCascade(db: Database.Database): void; +/** + * Open the existing index read-only, without creating it, migrating it, or + * changing its journal mode. For dry runs that must not write. Returns null + * when there is no index yet. + */ +export declare function openDatabaseReadOnly(): Database.Database | null; export declare function initDatabase(): Database.Database; export declare function insertExchange(db: Database.Database, exchange: ConversationExchange, embedding: number[], toolNames?: string[]): void; export declare function getAllExchanges(db: Database.Database): Array<{ diff --git a/dist/db.js b/dist/db.js index 83f546ad..3fac2bec 100644 --- a/dist/db.js +++ b/dist/db.js @@ -1,4 +1,5 @@ import Database from 'better-sqlite3'; +import { admitExchange } from './record-admission.js'; import path from 'path'; import fs from 'fs'; import * as sqliteVec from 'sqlite-vec'; @@ -90,6 +91,17 @@ export function migrateToolCallsCascade(db) { db.pragma('foreign_keys = ON'); console.log(' tool_calls migration complete.'); } +/** + * Open the existing index read-only, without creating it, migrating it, or + * changing its journal mode. For dry runs that must not write. Returns null + * when there is no index yet. + */ +export function openDatabaseReadOnly() { + const dbPath = getDbPath(); + if (!fs.existsSync(dbPath)) + return null; + return new Database(dbPath, { readonly: true, fileMustExist: true }); +} export function initDatabase() { const dbPath = getDbPath(); // Ensure directory exists @@ -186,6 +198,10 @@ export function initDatabase() { return db; } export function insertExchange(db, exchange, embedding, toolNames) { + const admitted = admitExchange(exchange); + if (!admitted) + return; + exchange = admitted; const now = Date.now(); const stmt = db.prepare(` INSERT OR REPLACE INTO exchanges diff --git a/dist/index-cli.js b/dist/index-cli.js index 950a4c7d..4e16428b 100644 --- a/dist/index-cli.js +++ b/dist/index-cli.js @@ -7,16 +7,16 @@ import path from 'path'; import { getSyncLockPath } from './logging.js'; import { acquireFileLock, readLockHolder, releaseFileLock } from './file-lock.js'; const command = process.argv[2]; -// Serialize the whole CLI with `sync-cli`: every command can initialize or -// migrate the database, and rebuild also deletes the database and summaries. +// Serialize the whole CLI with `sync-cli`: writers mutate the index/archive, +// and verification must not claim a consistent result while one is active. const syncLockPath = getSyncLockPath(); const syncLock = acquireFileLock(syncLockPath); if (!syncLock) { const holder = readLockHolder(syncLockPath); const holderLabel = holder !== null ? `pid ${holder}` : 'another process'; - // stderr keeps this out of stdout consumers; status 0 so hooks don't fail. + // A skipped write may be retried by a hook; a skipped verifier is not a result. console.error(`episodic-memory: sync already running (${holderLabel}); skipping`); - process.exit(0); + process.exit(command === 'verify' ? 2 : 0); } const releaseSyncLockOnce = () => { if (releaseSyncLockOnce.done) @@ -63,15 +63,28 @@ async function main() { const issues = await verifyIndex(); console.log('\n=== Verification Results ==='); console.log(`Missing summaries: ${issues.missing.length}`); + console.log(`Unindexed archives: ${issues.unindexed.length}`); console.log(`Orphaned entries: ${issues.orphaned.length}`); console.log(`Outdated files: ${issues.outdated.length}`); + console.log(`Screening-rejected archives: ${issues.screeningRejected.length}`); + console.log(`Screening-unavailable archives: ${issues.screeningUnavailable.length}`); + console.log(`Archive refreshes (no new exchanges): ${issues.archiveRefreshes.length}`); console.log(`Corrupted files: ${issues.corrupted.length}`); if (issues.missing.length > 0) { console.log('\nMissing summaries:'); issues.missing.forEach(m => console.log(` ${m.path}`)); } - if (issues.missing.length + issues.orphaned.length + issues.outdated.length + issues.corrupted.length > 0) { - console.log('\nRun with --repair to fix these issues.'); + if (issues.unindexed.length > 0) { + console.log('\nUnindexed archives:'); + issues.unindexed.forEach(item => console.log(` ${item.path}`)); + } + if (issues.outdated.length > 0) { + console.log('\nArchives with appended exchanges:'); + issues.outdated.forEach(item => console.log(` ${item.path}`)); + } + if (issues.missing.length + issues.unindexed.length + issues.orphaned.length + issues.outdated.length + + issues.screeningRejected.length + issues.screeningUnavailable.length + issues.corrupted.length > 0) { + console.log('\nReview the reported categories before repair; --repair may regenerate embeddings and AI summaries.'); process.exit(1); } else { @@ -81,10 +94,19 @@ async function main() { case 'repair': console.log('Verifying conversation index...'); const repairIssues = await verifyIndex(); + if (repairIssues.screeningRejected.length + repairIssues.screeningUnavailable.length > 0) { + console.error('Record screening failures require review; repair was not started.'); + process.exitCode = 1; + break; + } + if (repairIssues.unindexed.length > 0) { + console.log(`${repairIssues.unindexed.length} unindexed archive(s) require separate review; --repair does not select them by this status alone.`); + process.exitCode = 1; + } if (repairIssues.missing.length + repairIssues.orphaned.length + repairIssues.outdated.length > 0) { await repairIndex(repairIssues); } - else { + else if (repairIssues.unindexed.length === 0) { console.log('✅ No issues to repair!'); } break; diff --git a/dist/indexer.js b/dist/indexer.js index 8afa1d35..24b853b8 100644 --- a/dist/indexer.js +++ b/dist/indexer.js @@ -1,4 +1,5 @@ import fs from 'fs'; +import { archiveAdmittedConversation, readRecordExclusions } from './record-admission.js'; import path from 'path'; import { initDatabase, insertExchange } from './db.js'; import { parseConversation } from './parser.js'; @@ -7,6 +8,7 @@ import { summarizeConversation } from './summarizer.js'; import { getArchiveDir, getExcludedProjects, getConversationSourceDirs, findJsonlFiles, statIfExists } from './paths.js'; import { formatErrorSentinel, shouldQueueForSummary } from './summary-sentinel.js'; import { getMaxMessageBytes, isOversizeExchange } from './message-size.js'; +import { FindingsTally, formatFindings, loadRedactor } from './redaction.js'; // Set max output tokens for Claude SDK (used by summarizer) process.env.CLAUDE_CODE_MAX_OUTPUT_TOKENS = '20000'; // Increase max listeners for concurrent API calls @@ -25,7 +27,18 @@ async function processBatch(items, processor, concurrency) { function sessionIdForSummary(exchanges) { return exchanges.find(exchange => exchange.sessionId)?.sessionId; } +// Resume/fork would summarize the unredacted source transcript; see sync.ts. +function summarizeOptions(redactor) { + return { allowResume: redactor === null }; +} +function logRedactions(tally) { + if (tally.total > 0) + console.log(` Redaction: ${formatFindings(tally.toArray())}`); +} export async function indexConversations(limitToProject, maxConversations, concurrency = 1, noSummaries = false) { + readRecordExclusions(); + const redactor = loadRedactor(); + const tally = new FindingsTally(); console.log('Initializing database...'); const db = initDatabase(); console.log('Loading embedding model...'); @@ -73,16 +86,17 @@ export async function indexConversations(limitToProject, maxConversations, concu // Source transcripts can vanish mid-run (Claude Code cleanup). Skip loudly. let exchanges; try { - // Copy to archive (ensure parent dirs exist for subagent files) - if (!fs.existsSync(archivePath)) { - fs.mkdirSync(path.dirname(archivePath), { recursive: true }); - fs.copyFileSync(sourcePath, archivePath); + if (await archiveAdmittedConversation(sourcePath, archivePath, undefined, redactor, tally)) { console.log(` Archived: ${file}`); } // Parse conversation - exchanges = await parseConversation(sourcePath, project, archivePath); + exchanges = await parseConversation(archivePath, project, archivePath); } catch (error) { + if (error.code !== 'ENOENT') { + db.close(); + throw error; + } console.log(` Skipped ${file} (read failed: ${error instanceof Error ? error.message : error})`); continue; } @@ -105,7 +119,7 @@ export async function indexConversations(limitToProject, maxConversations, concu console.log(` Generating ${needsSummary.length} summaries (concurrency: ${concurrency})...`); await processBatch(needsSummary, async (conv) => { try { - const summary = await summarizeConversation(conv.exchanges, sessionIdForSummary(conv.exchanges)); + const summary = await summarizeConversation(conv.exchanges, sessionIdForSummary(conv.exchanges), summarizeOptions(redactor)); fs.writeFileSync(conv.summaryPath, summary, 'utf-8'); const wordCount = summary.split(/\s+/).length; console.log(` ✓ ${conv.file}: ${wordCount} words`); @@ -145,6 +159,7 @@ export async function indexConversations(limitToProject, maxConversations, concu // Check if we hit the limit if (maxConversations && conversationsProcessed >= maxConversations) { console.log(`\nReached limit of ${maxConversations} conversations`); + logRedactions(tally); db.close(); console.log(`✅ Indexing complete! Conversations: ${conversationsProcessed}, Exchanges: ${totalExchanges}`); return; @@ -155,11 +170,15 @@ export async function indexConversations(limitToProject, maxConversations, concu if (oversizeSkipped > 0) { console.log(` Skipped ${oversizeSkipped} oversize exchange(s) (> ${maxMessageBytes} bytes; set EPISODIC_MEMORY_MAX_MESSAGE_BYTES to change) — likely embedded-transcript payloads (#139)`); } + logRedactions(tally); db.close(); console.log(`\n✅ Indexing complete! Conversations: ${conversationsProcessed}, Exchanges: ${totalExchanges}`); } export async function indexSession(sessionId, concurrency = 1, noSummaries = false) { + readRecordExclusions(); console.log(`Indexing session: ${sessionId}`); + const redactor = loadRedactor(); + const tally = new FindingsTally(); // Find the conversation file for this session const sourceDirs = getConversationSourceDirs(); const ARCHIVE_DIR = getArchiveDir(); @@ -187,15 +206,14 @@ export async function indexSession(sessionId, concurrency = 1, noSummaries = fal // Archive + parse — source may vanish mid-run (Claude Code cleanup). let exchanges; try { - if (!fs.existsSync(archivePath)) { - fs.mkdirSync(path.dirname(archivePath), { recursive: true }); - fs.copyFileSync(sourcePath, archivePath); - } - exchanges = await parseConversation(sourcePath, project, archivePath); + await archiveAdmittedConversation(sourcePath, archivePath, undefined, redactor, tally); + exchanges = await parseConversation(archivePath, project, archivePath); } catch (error) { - console.log(`Skipped ${file} (read failed: ${error instanceof Error ? error.message : error})`); db.close(); + if (error.code !== 'ENOENT') + throw error; + console.log(`Skipped ${file} (read failed: ${error instanceof Error ? error.message : error})`); break; } if (exchanges.length > 0) { @@ -204,7 +222,7 @@ export async function indexSession(sessionId, concurrency = 1, noSummaries = fal if (!noSummaries && shouldQueueForSummary(summaryPath)) { fs.mkdirSync(path.dirname(summaryPath), { recursive: true }); try { - const summary = await summarizeConversation(exchanges, sessionIdForSummary(exchanges)); + const summary = await summarizeConversation(exchanges, sessionIdForSummary(exchanges), summarizeOptions(redactor)); fs.writeFileSync(summaryPath, summary, 'utf-8'); console.log(`Summary: ${summary.split(/\s+/).length} words`); } @@ -235,6 +253,7 @@ export async function indexSession(sessionId, concurrency = 1, noSummaries = fal if (oversizeSkipped > 0) { console.log(` Skipped ${oversizeSkipped} oversize exchange(s) (> ${maxMessageBytes} bytes; set EPISODIC_MEMORY_MAX_MESSAGE_BYTES to change) — likely embedded-transcript payloads (#139)`); } + logRedactions(tally); console.log(`✅ Indexed session ${sessionId}: ${exchanges.length} exchanges`); } db.close(); @@ -249,11 +268,14 @@ export async function indexSession(sessionId, concurrency = 1, noSummaries = fal } } export async function indexUnprocessed(concurrency = 1, noSummaries = false) { + readRecordExclusions(); console.log('Finding unprocessed conversations...'); if (concurrency > 1) console.log(`Concurrency: ${concurrency}`); if (noSummaries) console.log('⚠️ Running in no-summaries mode (skipping AI summaries)'); + const redactor = loadRedactor(); + const tally = new FindingsTally(); const db = initDatabase(); await initEmbeddings(); const sourceDirs = getConversationSourceDirs(); @@ -280,15 +302,10 @@ export async function indexUnprocessed(concurrency = 1, noSummaries = false) { // Transcript JSONLs are append-only, so MAX(line_end) tells us where to resume. const hw = db.prepare('SELECT COALESCE(MAX(line_end), 0) as maxLine FROM exchanges WHERE archive_path = ?').get(archivePath); const maxIndexedLine = hw.maxLine; - // Ensure parent dirs exist for subagent files try { - fs.mkdirSync(path.dirname(archivePath), { recursive: true }); - // Refresh the archive when the source may have grown beyond what we've seen. - if (!fs.existsSync(archivePath) || maxIndexedLine > 0) { - fs.copyFileSync(sourcePath, archivePath); - } + await archiveAdmittedConversation(sourcePath, archivePath, undefined, redactor, tally); // Parse and filter to exchanges past the high-water mark - const exchanges = await parseConversation(sourcePath, project, archivePath); + const exchanges = await parseConversation(archivePath, project, archivePath); const newExchanges = maxIndexedLine > 0 ? exchanges.filter(e => e.lineStart > maxIndexedLine) : exchanges; @@ -297,12 +314,17 @@ export async function indexUnprocessed(concurrency = 1, noSummaries = false) { unprocessed.push({ project, file, sourcePath, archivePath, summaryPath, exchanges: newExchanges }); } catch (error) { + if (error.code !== 'ENOENT') { + db.close(); + throw error; + } console.log(` Skipped ${file} (read failed: ${error instanceof Error ? error.message : error})`); continue; } } } } // end sourceDir loop + logRedactions(tally); if (unprocessed.length === 0) { console.log('✅ All conversations are already processed!'); db.close(); @@ -316,7 +338,7 @@ export async function indexUnprocessed(concurrency = 1, noSummaries = false) { console.log(`Generating ${needsSummary.length} summaries (concurrency: ${concurrency})...\n`); await processBatch(needsSummary, async (conv) => { try { - const summary = await summarizeConversation(conv.exchanges, sessionIdForSummary(conv.exchanges)); + const summary = await summarizeConversation(conv.exchanges, sessionIdForSummary(conv.exchanges), summarizeOptions(redactor)); fs.writeFileSync(conv.summaryPath, summary, 'utf-8'); const wordCount = summary.split(/\s+/).length; console.log(` ✓ ${conv.project}/${conv.file}: ${wordCount} words`); diff --git a/dist/mcp-server.js b/dist/mcp-server.js index e3a9adac..fa6f73e4 100644 --- a/dist/mcp-server.js +++ b/dist/mcp-server.js @@ -469,9 +469,9 @@ var require_codegen = __commonJS({ } }; var Throw = class extends Node { - constructor(error61) { + constructor(error62) { super(); - this.error = error61; + this.error = error62; } render({ _n }) { return `throw ${this.error};` + _n; @@ -708,9 +708,9 @@ var require_codegen = __commonJS({ } }; var Catch = class extends BlockNode { - constructor(error61) { + constructor(error62) { super(); - this.error = error61; + this.error = error62; } render(opts) { return `catch(${this.error})` + super.render(opts); @@ -901,9 +901,9 @@ var require_codegen = __commonJS({ this._blockNode(node2); this.code(tryBody); if (catchCode) { - const error61 = this.name("e"); - this._currNode = node2.catch = new Catch(error61); - catchCode(error61); + const error62 = this.name("e"); + this._currNode = node2.catch = new Catch(error62); + catchCode(error62); } if (finallyCode) { this._currNode = node2.finally = new Finally(); @@ -912,8 +912,8 @@ var require_codegen = __commonJS({ return this._endBlockNode(Catch, Finally); } // `throw` statement - throw(error61) { - return this._leafNode(new Throw(error61)); + throw(error62) { + return this._leafNode(new Throw(error62)); } // start self-balancing block block(body, nodeCount) { @@ -1269,10 +1269,10 @@ var require_errors = __commonJS({ exports.keyword$DataError = { message: ({ keyword, schemaType }) => schemaType ? (0, codegen_1.str)`"${keyword}" keyword must be ${schemaType} ($data)` : (0, codegen_1.str)`"${keyword}" keyword is invalid ($data)` }; - function reportError(cxt, error61 = exports.keywordError, errorPaths, overrideAllErrors) { + function reportError(cxt, error62 = exports.keywordError, errorPaths, overrideAllErrors) { const { it } = cxt; const { gen, compositeRule, allErrors } = it; - const errObj = errorObjectCode(cxt, error61, errorPaths); + const errObj = errorObjectCode(cxt, error62, errorPaths); if (overrideAllErrors !== null && overrideAllErrors !== void 0 ? overrideAllErrors : compositeRule || allErrors) { addError(gen, errObj); } else { @@ -1280,10 +1280,10 @@ var require_errors = __commonJS({ } } exports.reportError = reportError; - function reportExtraError(cxt, error61 = exports.keywordError, errorPaths) { + function reportExtraError(cxt, error62 = exports.keywordError, errorPaths) { const { it } = cxt; const { gen, compositeRule, allErrors } = it; - const errObj = errorObjectCode(cxt, error61, errorPaths); + const errObj = errorObjectCode(cxt, error62, errorPaths); addError(gen, errObj); if (!(compositeRule || allErrors)) { returnErrors(it, names_1.default.vErrors); @@ -1334,19 +1334,19 @@ var require_errors = __commonJS({ schema: new codegen_1.Name("schema"), parentSchema: new codegen_1.Name("parentSchema") }; - function errorObjectCode(cxt, error61, errorPaths) { + function errorObjectCode(cxt, error62, errorPaths) { const { createErrors } = cxt.it; if (createErrors === false) return (0, codegen_1._)`{}`; - return errorObject(cxt, error61, errorPaths); + return errorObject(cxt, error62, errorPaths); } - function errorObject(cxt, error61, errorPaths = {}) { + function errorObject(cxt, error62, errorPaths = {}) { const { gen, it } = cxt; const keyValues = [ errorInstancePath(it, errorPaths), errorSchemaPath(cxt, errorPaths) ]; - extraErrorProps(cxt, error61, keyValues); + extraErrorProps(cxt, error62, keyValues); return gen.object(...keyValues); } function errorInstancePath({ errorPath }, { instancePath }) { @@ -1775,13 +1775,13 @@ var require_code2 = __commonJS({ exports.callValidateCode = callValidateCode; var newRegExp = (0, codegen_1._)`new RegExp`; function usePattern({ gen, it: { opts } }, pattern) { - const u3 = opts.unicodeRegExp ? "u" : ""; + const u = opts.unicodeRegExp ? "u" : ""; const { regExp } = opts.code; - const rx = regExp(pattern, u3); + const rx = regExp(pattern, u); return gen.scopeValue("pattern", { key: rx.toString(), ref: rx, - code: (0, codegen_1._)`${regExp.code === "new RegExp" ? newRegExp : (0, util_2.useFunc)(gen, regExp)}(${pattern}, ${u3})` + code: (0, codegen_1._)`${regExp.code === "new RegExp" ? newRegExp : (0, util_2.useFunc)(gen, regExp)}(${pattern}, ${u})` }); } exports.usePattern = usePattern; @@ -1878,13 +1878,13 @@ var require_keyword = __commonJS({ modifyData(cxt); reportErrs(() => cxt.error()); } else { - const ruleErrs = def.async ? validateAsync2() : validateSync(); + const ruleErrs = def.async ? validateAsync3() : validateSync(); if (def.modifying) modifyData(cxt); reportErrs(() => addErrs(cxt, ruleErrs)); } } - function validateAsync2() { + function validateAsync3() { const ruleErrs = gen.let("ruleErrs", null); gen.try(() => assignValid((0, codegen_1._)`await `), (e) => gen.assign(valid, false).if((0, codegen_1._)`${e} instanceof ${it.ValidationError}`, () => gen.assign(ruleErrs, (0, codegen_1._)`${e}.errors`), () => gen.throw(e))); return ruleErrs; @@ -2044,7 +2044,7 @@ var require_fast_deep_equal = __commonJS({ if (a === b2) return true; if (a && b2 && typeof a == "object" && typeof b2 == "object") { if (a.constructor !== b2.constructor) return false; - var length, i, keys; + var length, i, keys2; if (Array.isArray(a)) { length = a.length; if (length != b2.length) return false; @@ -2055,13 +2055,13 @@ var require_fast_deep_equal = __commonJS({ if (a.constructor === RegExp) return a.source === b2.source && a.flags === b2.flags; if (a.valueOf !== Object.prototype.valueOf) return a.valueOf() === b2.valueOf(); if (a.toString !== Object.prototype.toString) return a.toString() === b2.toString(); - keys = Object.keys(a); - length = keys.length; + keys2 = Object.keys(a); + length = keys2.length; if (length !== Object.keys(b2).length) return false; for (i = length; i-- !== 0; ) - if (!Object.prototype.hasOwnProperty.call(b2, keys[i])) return false; + if (!Object.prototype.hasOwnProperty.call(b2, keys2[i])) return false; for (i = length; i-- !== 0; ) { - var key = keys[i]; + var key = keys2[i]; if (!equal(a[key], b2[key])) return false; } return true; @@ -3258,8 +3258,8 @@ var require_utils = __commonJS({ } return ind; } - function removeDotSegments(path3) { - let input2 = path3; + function removeDotSegments(path4) { + let input2 = path4; const output2 = []; let nextSlash = -1; let len = 0; @@ -3333,6 +3333,37 @@ var require_utils = __commonJS({ } return output2.join(""); } + function removeDotSegmentsRelative(path4) { + if (path4 === "" || path4[0] === "/") { + return removeDotSegments(path4); + } + const segments = path4.split("/"); + const last = segments.length - 1; + const output2 = []; + let trailingDirectory = false; + for (let i = 0; i <= last; i++) { + const segment = segments[i]; + if (segment === "." || segment === "..") { + if (segment === "..") { + if (output2.length !== 0 && output2[output2.length - 1] !== "..") { + output2.pop(); + } else { + output2.push(".."); + } + } + trailingDirectory = i === last; + } else { + output2.push(segment); + } + } + if (trailingDirectory && output2[output2.length - 1] !== "..") { + output2.push(""); + } + if (output2.length === 0 || output2[0] === "" || output2[0].includes(":")) { + output2.unshift("."); + } + return output2.join("/"); + } var HOST_DELIMS = { "@": "%40", "/": "%2F", "?": "%3F", "#": "%23", ":": "%3A" }; var HOST_DELIM_RE = /[@/?#:]/g; var HOST_DELIM_NO_COLON_RE = /[@/?#]/g; @@ -3593,6 +3624,7 @@ var require_utils = __commonJS({ encodeFragment, escapePreservingEscapes, removeDotSegments, + removeDotSegmentsRelative, isIPv4, isUUID, normalizeIPv6, @@ -3668,8 +3700,8 @@ var require_schemes = __commonJS({ } if (wsComponent.resourceName) { const queryIndex = wsComponent.resourceName.indexOf("?"); - const path3 = queryIndex === -1 ? wsComponent.resourceName : wsComponent.resourceName.slice(0, queryIndex); - wsComponent.path = path3 && path3 !== "/" ? path3 : void 0; + const path4 = queryIndex === -1 ? wsComponent.resourceName : wsComponent.resourceName.slice(0, queryIndex); + wsComponent.path = path4 && path4 !== "/" ? path4 : void 0; wsComponent.query = queryIndex === -1 ? void 0 : wsComponent.resourceName.slice(queryIndex + 1); wsComponent.resourceName = void 0; } @@ -3816,16 +3848,18 @@ var require_schemes = __commonJS({ var require_fast_uri = __commonJS({ "node_modules/fast-uri/index.js"(exports, module) { "use strict"; - var { normalizeIPv6, removeDotSegments, recomposeAuthority, normalizePercentEncoding, normalizePathEncoding, serializePathEncoding, normalizeQueryFragmentEncoding, encodeQuery, encodeFragment, reescapeHostDelimiters, isIPv4, nonSimpleDomain } = require_utils(); + var { normalizeIPv6, removeDotSegments, removeDotSegmentsRelative, recomposeAuthority, normalizePercentEncoding, normalizePathEncoding, serializePathEncoding, normalizeQueryFragmentEncoding, encodeQuery, encodeFragment, reescapeHostDelimiters, isIPv4, nonSimpleDomain } = require_utils(); var { SCHEMES, getSchemeHandler } = require_schemes(); var VALID_SCHEME = /^[A-Za-z][A-Za-z0-9+.-]*$/u; var MALFORMED_SCHEME_ERROR = "URI scheme is malformed."; - function decodeValidScheme(scheme) { - const decodedScheme = unescape(String(scheme)); - if (!VALID_SCHEME.test(decodedScheme)) { + var MALFORMED_HOST_ERROR = "URI host is malformed."; + var FORBIDDEN_DOMAIN_HOST_CHARACTER = /[\0-\x20"#/:<>?@[\\\]^`{|}\x7F]/u; + function validateScheme(scheme) { + const stringScheme = String(scheme); + if (!VALID_SCHEME.test(stringScheme)) { throw new TypeError(MALFORMED_SCHEME_ERROR); } - return decodedScheme; + return stringScheme; } function normalize(uri, options) { if (typeof uri === "string") { @@ -3862,6 +3896,9 @@ var require_fast_uri = __commonJS({ const resolvedSchemeHandler = getSchemeHandler(options && options.scheme || resolved.scheme); const resolvedHost = resolved.host; const resolvedHostIsIP = resolvedHost !== void 0 && resolvedHost !== "" && (isIPv4(resolvedHost) || normalizeIPv6(resolvedHost).isIPV6); + if (hasForbiddenDomainHost(resolved, options || {}, resolvedSchemeHandler, resolvedHostIsIP)) { + throw new Error(MALFORMED_HOST_ERROR); + } canonicalizeHost(resolved, options || {}, resolvedSchemeHandler, resolvedHostIsIP); const encodedASCIIHost = resolvedHost && resolvedHost.indexOf("%") !== -1 && !/\P{ASCII}/u.test(resolvedHost); if (resolved.error && !encodedASCIIHost) { @@ -3903,14 +3940,15 @@ var require_fast_uri = __commonJS({ if (relative.path[0] === "/") { target.path = removeDotSegments(relative.path); } else { - if ((base.userinfo !== void 0 || base.host !== void 0 || base.port !== void 0) && !base.path) { + const baseHasAuthority = base.userinfo !== void 0 || base.host !== void 0 || base.port !== void 0; + if (baseHasAuthority && !base.path) { target.path = "/" + relative.path; } else if (!base.path) { target.path = relative.path; } else { target.path = base.path.slice(0, base.path.lastIndexOf("/") + 1) + relative.path; } - target.path = removeDotSegments(target.path); + target.path = base.scheme === void 0 && !baseHasAuthority && target.path[0] !== "/" ? removeDotSegmentsRelative(target.path) : removeDotSegments(target.path); } target.query = relative.query; } @@ -3948,12 +3986,13 @@ var require_fast_uri = __commonJS({ const options = Object.assign({}, opts); const uriTokens = []; if (component.scheme) { - component.scheme = decodeValidScheme(component.scheme); + component.scheme = validateScheme(component.scheme); } const schemeHandler = getSchemeHandler(options.scheme || component.scheme); if (schemeHandler && schemeHandler.serialize) schemeHandler.serialize(component, options); const hasAuthority = component.userinfo !== void 0 || component.host !== void 0 || component.port !== void 0; - const pathNoScheme = !options.skipEscape && component.scheme === void 0 && !hasAuthority; + const schemelessReference = (component.scheme === void 0 || options.reference === "suffix") && !hasAuthority; + const pathNoScheme = !options.skipEscape && schemelessReference; if (component.path !== void 0) { if (!options.skipEscape) { component.path = serializePathEncoding(component.path, pathNoScheme); @@ -3962,7 +4001,7 @@ var require_fast_uri = __commonJS({ } } if (options.reference !== "suffix" && component.scheme) { - component.scheme = decodeValidScheme(component.scheme); + component.scheme = validateScheme(component.scheme); uriTokens.push(component.scheme, ":"); } const authority = recomposeAuthority(component); @@ -3978,9 +4017,9 @@ var require_fast_uri = __commonJS({ if (component.path !== void 0) { let s = component.path; if (!options.absolutePath && (!schemeHandler || !schemeHandler.absolutePath)) { - s = removeDotSegments(s); + s = schemelessReference && s[0] !== "/" ? removeDotSegmentsRelative(s) : removeDotSegments(s); } - if (pathNoScheme) { + if (schemelessReference) { s = serializePathEncoding(s, true); } if (authority === void 0 && s[0] === "/" && s[1] === "/") { @@ -4027,16 +4066,34 @@ var require_fast_uri = __commonJS({ return hasMalformedPercentEncoding(matches[3]) || host !== void 0 && !isIPLiteral(host) && hasMalformedPercentEncoding(host) || hasMalformedPercentEncoding(matches[6]) || hasMalformedPercentEncoding(matches[7]) || hasMalformedPercentEncoding(matches[8]); } function canonicalizeHost(parsed, options, schemeHandler, isIP) { + if (hasForbiddenDomainHost(parsed, options, schemeHandler, isIP)) { + parsed.error = parsed.error || MALFORMED_HOST_ERROR; + return true; + } if (!options.unicodeSupport && (!schemeHandler || !schemeHandler.unicodeSupport) && parsed.host && !isIPLiteral(parsed.host) && (options.domainHost || schemeHandler && schemeHandler.domainHost) && isIP === false && nonSimpleDomain(parsed.host)) { + let url2; try { - parsed.host = new URL("http://" + parsed.host).hostname; + url2 = new URL("http://" + parsed.host); } catch (e) { parsed.error = parsed.error || "Host's domain name can not be converted to ASCII: " + e; return true; } + if (!isHostOnlyURL(url2)) { + parsed.error = parsed.error || MALFORMED_HOST_ERROR; + return true; + } + parsed.host = url2.hostname; } return false; } + function hasForbiddenDomainHost(parsed, options, schemeHandler, isIP) { + return Boolean( + parsed.host && isIP === false && !isIPLiteral(parsed.host) && (options.domainHost || schemeHandler && schemeHandler.domainHost) && FORBIDDEN_DOMAIN_HOST_CHARACTER.test(parsed.host) + ); + } + function isHostOnlyURL(url2) { + return url2.username === "" && url2.password === "" && url2.port === "" && url2.pathname === "/" && url2.search === "" && url2.hash === "" && url2.hostname !== ""; + } function parseWithStatus(uri, opts) { const options = Object.assign({}, opts); const parsed = { @@ -4091,9 +4148,8 @@ var require_fast_uri = __commonJS({ parsed.query = matches[7]; parsed.fragment = matches[8]; if (parsed.scheme !== void 0) { - const decodedScheme = unescape(parsed.scheme); - if (VALID_SCHEME.test(decodedScheme)) { - parsed.scheme = decodedScheme.toLowerCase(); + if (VALID_SCHEME.test(parsed.scheme)) { + parsed.scheme = parsed.scheme.toLowerCase(); } else { parsed.error = parsed.error || MALFORMED_SCHEME_ERROR; malformedScheme = true; @@ -4121,7 +4177,7 @@ var require_fast_uri = __commonJS({ malformedIPLiteral = hasIPLiteralBracket && (!bracketedIPLiteral || ipv6result.error === true); parsed.host = isIP ? ipv6result.host : ipv6result.host.toLowerCase(); if (malformedIPLiteral) { - parsed.error = parsed.error || "URI host is malformed."; + parsed.error = parsed.error || MALFORMED_HOST_ERROR; malformedAuthorityOrPort = true; } } else { @@ -4144,13 +4200,14 @@ var require_fast_uri = __commonJS({ if (!malformedIPLiteral) { malformedHost = canonicalizeHost(parsed, options, schemeHandler, isIP); } - if (!schemeHandler || schemeHandler && !schemeHandler.skipNormalize) { - if (uri.indexOf("%") !== -1) { - if (parsed.host !== void 0 && !malformedIPLiteral) { - const host = isIP ? parsed.host : normalizePercentEncoding(parsed.host, true); - parsed.host = reescapeHostDelimiters(host, isIP); - } + if (uri.indexOf("%") !== -1 && parsed.host !== void 0 && !malformedIPLiteral) { + let host = isIP ? parsed.host : normalizePercentEncoding(parsed.host, true); + if (!isIP) { + host = normalizePercentEncoding(host.toLowerCase()); } + parsed.host = isIP || options.scheme !== "null" ? reescapeHostDelimiters(host, isIP) : host; + } + if (!schemeHandler || schemeHandler && !schemeHandler.skipNormalize) { if (parsed.path) { parsed.path = normalizePathEncoding(parsed.path); } @@ -5010,7 +5067,7 @@ var require_limitNumber = __commonJS({ exclusiveMaximum: { okStr: "<", ok: ops.LT, fail: ops.GTE }, exclusiveMinimum: { okStr: ">", ok: ops.GT, fail: ops.LTE } }; - var error61 = { + var error62 = { message: ({ keyword, schemaCode }) => (0, codegen_1.str)`must be ${KWDs[keyword].okStr} ${schemaCode}`, params: ({ keyword, schemaCode }) => (0, codegen_1._)`{comparison: ${KWDs[keyword].okStr}, limit: ${schemaCode}}` }; @@ -5019,7 +5076,7 @@ var require_limitNumber = __commonJS({ type: "number", schemaType: "number", $data: true, - error: error61, + error: error62, code(cxt) { const { keyword, data, schemaCode } = cxt; cxt.fail$data((0, codegen_1._)`${data} ${KWDs[keyword].fail} ${schemaCode} || isNaN(${data})`); @@ -5035,7 +5092,7 @@ var require_multipleOf = __commonJS({ "use strict"; Object.defineProperty(exports, "__esModule", { value: true }); var codegen_1 = require_codegen(); - var error61 = { + var error62 = { message: ({ schemaCode }) => (0, codegen_1.str)`must be multiple of ${schemaCode}`, params: ({ schemaCode }) => (0, codegen_1._)`{multipleOf: ${schemaCode}}` }; @@ -5044,7 +5101,7 @@ var require_multipleOf = __commonJS({ type: "number", schemaType: "number", $data: true, - error: error61, + error: error62, code(cxt) { const { gen, data, schemaCode, it } = cxt; const prec = it.opts.multipleOfPrecision; @@ -5091,7 +5148,7 @@ var require_limitLength = __commonJS({ var codegen_1 = require_codegen(); var util_1 = require_util(); var ucs2length_1 = require_ucs2length(); - var error61 = { + var error62 = { message({ keyword, schemaCode }) { const comp = keyword === "maxLength" ? "more" : "fewer"; return (0, codegen_1.str)`must NOT have ${comp} than ${schemaCode} characters`; @@ -5103,7 +5160,7 @@ var require_limitLength = __commonJS({ type: "string", schemaType: "number", $data: true, - error: error61, + error: error62, code(cxt) { const { keyword, data, schemaCode, it } = cxt; const op = keyword === "maxLength" ? codegen_1.operators.GT : codegen_1.operators.LT; @@ -5123,7 +5180,7 @@ var require_pattern = __commonJS({ var code_1 = require_code2(); var util_1 = require_util(); var codegen_1 = require_codegen(); - var error61 = { + var error62 = { message: ({ schemaCode }) => (0, codegen_1.str)`must match pattern "${schemaCode}"`, params: ({ schemaCode }) => (0, codegen_1._)`{pattern: ${schemaCode}}` }; @@ -5132,15 +5189,15 @@ var require_pattern = __commonJS({ type: "string", schemaType: "string", $data: true, - error: error61, + error: error62, code(cxt) { const { gen, data, $data, schema, schemaCode, it } = cxt; - const u3 = it.opts.unicodeRegExp ? "u" : ""; + const u = it.opts.unicodeRegExp ? "u" : ""; if ($data) { const { regExp } = it.opts.code; const regExpCode = regExp.code === "new RegExp" ? (0, codegen_1._)`new RegExp` : (0, util_1.useFunc)(gen, regExp); const valid = gen.let("valid"); - gen.try(() => gen.assign(valid, (0, codegen_1._)`${regExpCode}(${schemaCode}, ${u3}).test(${data})`), () => gen.assign(valid, false)); + gen.try(() => gen.assign(valid, (0, codegen_1._)`${regExpCode}(${schemaCode}, ${u}).test(${data})`), () => gen.assign(valid, false)); cxt.fail$data((0, codegen_1._)`!${valid}`); } else { const regExp = (0, code_1.usePattern)(cxt, schema); @@ -5158,7 +5215,7 @@ var require_limitProperties = __commonJS({ "use strict"; Object.defineProperty(exports, "__esModule", { value: true }); var codegen_1 = require_codegen(); - var error61 = { + var error62 = { message({ keyword, schemaCode }) { const comp = keyword === "maxProperties" ? "more" : "fewer"; return (0, codegen_1.str)`must NOT have ${comp} than ${schemaCode} properties`; @@ -5170,7 +5227,7 @@ var require_limitProperties = __commonJS({ type: "object", schemaType: "number", $data: true, - error: error61, + error: error62, code(cxt) { const { keyword, data, schemaCode } = cxt; const op = keyword === "maxProperties" ? codegen_1.operators.GT : codegen_1.operators.LT; @@ -5189,7 +5246,7 @@ var require_required = __commonJS({ var code_1 = require_code2(); var codegen_1 = require_codegen(); var util_1 = require_util(); - var error61 = { + var error62 = { message: ({ params: { missingProperty } }) => (0, codegen_1.str)`must have required property '${missingProperty}'`, params: ({ params: { missingProperty } }) => (0, codegen_1._)`{missingProperty: ${missingProperty}}` }; @@ -5198,7 +5255,7 @@ var require_required = __commonJS({ type: "object", schemaType: "array", $data: true, - error: error61, + error: error62, code(cxt) { const { gen, schema, schemaCode, data, $data, it } = cxt; const { opts } = it; @@ -5269,7 +5326,7 @@ var require_limitItems = __commonJS({ "use strict"; Object.defineProperty(exports, "__esModule", { value: true }); var codegen_1 = require_codegen(); - var error61 = { + var error62 = { message({ keyword, schemaCode }) { const comp = keyword === "maxItems" ? "more" : "fewer"; return (0, codegen_1.str)`must NOT have ${comp} than ${schemaCode} items`; @@ -5281,7 +5338,7 @@ var require_limitItems = __commonJS({ type: "array", schemaType: "number", $data: true, - error: error61, + error: error62, code(cxt) { const { keyword, data, schemaCode } = cxt; const op = keyword === "maxItems" ? codegen_1.operators.GT : codegen_1.operators.LT; @@ -5312,7 +5369,7 @@ var require_uniqueItems = __commonJS({ var codegen_1 = require_codegen(); var util_1 = require_util(); var equal_1 = require_equal(); - var error61 = { + var error62 = { message: ({ params: { i, j: j2 } }) => (0, codegen_1.str)`must NOT have duplicate items (items ## ${j2} and ${i} are identical)`, params: ({ params: { i, j: j2 } }) => (0, codegen_1._)`{i: ${i}, j: ${j2}}` }; @@ -5321,7 +5378,7 @@ var require_uniqueItems = __commonJS({ type: "array", schemaType: "boolean", $data: true, - error: error61, + error: error62, code(cxt) { const { gen, data, $data, schema, parentSchema, schemaCode, it } = cxt; if (!$data && !schema) @@ -5378,14 +5435,14 @@ var require_const = __commonJS({ var codegen_1 = require_codegen(); var util_1 = require_util(); var equal_1 = require_equal(); - var error61 = { + var error62 = { message: "must be equal to constant", params: ({ schemaCode }) => (0, codegen_1._)`{allowedValue: ${schemaCode}}` }; var def = { keyword: "const", $data: true, - error: error61, + error: error62, code(cxt) { const { gen, data, $data, schemaCode, schema } = cxt; if ($data || schema && typeof schema == "object") { @@ -5407,7 +5464,7 @@ var require_enum = __commonJS({ var codegen_1 = require_codegen(); var util_1 = require_util(); var equal_1 = require_equal(); - var error61 = { + var error62 = { message: "must be equal to one of the allowed values", params: ({ schemaCode }) => (0, codegen_1._)`{allowedValues: ${schemaCode}}` }; @@ -5415,7 +5472,7 @@ var require_enum = __commonJS({ keyword: "enum", schemaType: "array", $data: true, - error: error61, + error: error62, code(cxt) { const { gen, data, $data, schema, schemaCode, it } = cxt; if (!$data && schema.length === 0) @@ -5494,7 +5551,7 @@ var require_additionalItems = __commonJS({ exports.validateAdditionalItems = void 0; var codegen_1 = require_codegen(); var util_1 = require_util(); - var error61 = { + var error62 = { message: ({ params: { len } }) => (0, codegen_1.str)`must NOT have more than ${len} items`, params: ({ params: { len } }) => (0, codegen_1._)`{limit: ${len}}` }; @@ -5503,7 +5560,7 @@ var require_additionalItems = __commonJS({ type: "array", schemaType: ["boolean", "object"], before: "uniqueItems", - error: error61, + error: error62, code(cxt) { const { parentSchema, it } = cxt; const { items } = parentSchema; @@ -5583,10 +5640,10 @@ var require_items = __commonJS({ }); function checkStrictTuple(sch) { const { opts, errSchemaPath } = it; - const l = schArr.length; - const fullTuple = l === sch.minItems && (l === sch.maxItems || sch[extraItems] === false); + const l3 = schArr.length; + const fullTuple = l3 === sch.minItems && (l3 === sch.maxItems || sch[extraItems] === false); if (opts.strictTuples && !fullTuple) { - const msg = `"${keyword}" is ${l}-tuple, but minItems or maxItems/${extraItems} are not specified or different at path "${errSchemaPath}"`; + const msg = `"${keyword}" is ${l3}-tuple, but minItems or maxItems/${extraItems} are not specified or different at path "${errSchemaPath}"`; (0, util_1.checkStrictMode)(it, msg, opts.strictTuples); } } @@ -5622,7 +5679,7 @@ var require_items2020 = __commonJS({ var util_1 = require_util(); var code_1 = require_code2(); var additionalItems_1 = require_additionalItems(); - var error61 = { + var error62 = { message: ({ params: { len } }) => (0, codegen_1.str)`must NOT have more than ${len} items`, params: ({ params: { len } }) => (0, codegen_1._)`{limit: ${len}}` }; @@ -5631,7 +5688,7 @@ var require_items2020 = __commonJS({ type: "array", schemaType: ["object", "boolean"], before: "uniqueItems", - error: error61, + error: error62, code(cxt) { const { schema, parentSchema, it } = cxt; const { prefixItems } = parentSchema; @@ -5655,7 +5712,7 @@ var require_contains = __commonJS({ Object.defineProperty(exports, "__esModule", { value: true }); var codegen_1 = require_codegen(); var util_1 = require_util(); - var error61 = { + var error62 = { message: ({ params: { min, max } }) => max === void 0 ? (0, codegen_1.str)`must contain at least ${min} valid item(s)` : (0, codegen_1.str)`must contain at least ${min} and no more than ${max} valid item(s)`, params: ({ params: { min, max } }) => max === void 0 ? (0, codegen_1._)`{minContains: ${min}}` : (0, codegen_1._)`{minContains: ${min}, maxContains: ${max}}` }; @@ -5665,7 +5722,7 @@ var require_contains = __commonJS({ schemaType: ["object", "boolean"], before: "uniqueItems", trackErrors: true, - error: error61, + error: error62, code(cxt) { const { gen, schema, parentSchema, data, it } = cxt; let min; @@ -5843,7 +5900,7 @@ var require_propertyNames = __commonJS({ Object.defineProperty(exports, "__esModule", { value: true }); var codegen_1 = require_codegen(); var util_1 = require_util(); - var error61 = { + var error62 = { message: "property name must be valid", params: ({ params }) => (0, codegen_1._)`{propertyName: ${params.propertyName}}` }; @@ -5851,7 +5908,7 @@ var require_propertyNames = __commonJS({ keyword: "propertyNames", type: "object", schemaType: ["object", "boolean"], - error: error61, + error: error62, code(cxt) { const { gen, schema, data, it } = cxt; if ((0, util_1.alwaysValidSchema)(it, schema)) @@ -5888,7 +5945,7 @@ var require_additionalProperties = __commonJS({ var codegen_1 = require_codegen(); var names_1 = require_names(); var util_1 = require_util(); - var error61 = { + var error62 = { message: "must NOT have additional properties", params: ({ params }) => (0, codegen_1._)`{additionalProperty: ${params.additionalProperty}}` }; @@ -5898,7 +5955,7 @@ var require_additionalProperties = __commonJS({ schemaType: ["boolean", "object"], allowUndefined: true, trackErrors: true, - error: error61, + error: error62, code(cxt) { const { gen, schema, parentSchema, data, errsCount, it } = cxt; if (!errsCount) @@ -6172,7 +6229,7 @@ var require_oneOf = __commonJS({ Object.defineProperty(exports, "__esModule", { value: true }); var codegen_1 = require_codegen(); var util_1 = require_util(); - var error61 = { + var error62 = { message: "must match exactly one schema in oneOf", params: ({ params }) => (0, codegen_1._)`{passingSchemas: ${params.passing}}` }; @@ -6180,7 +6237,7 @@ var require_oneOf = __commonJS({ keyword: "oneOf", schemaType: "array", trackErrors: true, - error: error61, + error: error62, code(cxt) { const { gen, schema, parentSchema, it } = cxt; if (!Array.isArray(schema)) @@ -6257,7 +6314,7 @@ var require_if = __commonJS({ Object.defineProperty(exports, "__esModule", { value: true }); var codegen_1 = require_codegen(); var util_1 = require_util(); - var error61 = { + var error62 = { message: ({ params }) => (0, codegen_1.str)`must match "${params.ifClause}" schema`, params: ({ params }) => (0, codegen_1._)`{failingKeyword: ${params.ifClause}}` }; @@ -6265,7 +6322,7 @@ var require_if = __commonJS({ keyword: "if", schemaType: ["object", "boolean"], trackErrors: true, - error: error61, + error: error62, code(cxt) { const { gen, parentSchema, it } = cxt; if (parentSchema.then === void 0 && parentSchema.else === void 0) { @@ -6391,7 +6448,7 @@ var require_format = __commonJS({ "use strict"; Object.defineProperty(exports, "__esModule", { value: true }); var codegen_1 = require_codegen(); - var error61 = { + var error62 = { message: ({ schemaCode }) => (0, codegen_1.str)`must match format "${schemaCode}"`, params: ({ schemaCode }) => (0, codegen_1._)`{format: ${schemaCode}}` }; @@ -6400,7 +6457,7 @@ var require_format = __commonJS({ type: ["number", "string"], schemaType: "string", $data: true, - error: error61, + error: error62, code(cxt, ruleType) { const { gen, data, $data, schema, schemaCode, it } = cxt; const { opts, errSchemaPath, schemaEnv, self } = it; @@ -6555,7 +6612,7 @@ var require_discriminator = __commonJS({ var compile_1 = require_compile(); var ref_error_1 = require_ref_error(); var util_1 = require_util(); - var error61 = { + var error62 = { message: ({ params: { discrError, tagName } }) => discrError === types_1.DiscrError.Tag ? `tag "${tagName}" must be string` : `value of tag "${tagName}" must be in oneOf`, params: ({ params: { discrError, tag, tagName } }) => (0, codegen_1._)`{error: ${discrError}, tag: ${tagName}, tagValue: ${tag}}` }; @@ -6563,7 +6620,7 @@ var require_discriminator = __commonJS({ keyword: "discriminator", type: "object", schemaType: "object", - error: error61, + error: error62, code(cxt) { const { gen, data, schema, parentSchema, it } = cxt; const { oneOf } = parentSchema; @@ -7095,7 +7152,7 @@ var require_limit = __commonJS({ formatExclusiveMaximum: { okStr: "<", ok: ops.LT, fail: ops.GTE }, formatExclusiveMinimum: { okStr: ">", ok: ops.GT, fail: ops.LTE } }; - var error61 = { + var error62 = { message: ({ keyword, schemaCode }) => (0, codegen_1.str)`should be ${KWDs[keyword].okStr} ${schemaCode}`, params: ({ keyword, schemaCode }) => (0, codegen_1._)`{comparison: ${KWDs[keyword].okStr}, limit: ${schemaCode}}` }; @@ -7104,7 +7161,7 @@ var require_limit = __commonJS({ type: "string", schemaType: "string", $data: true, - error: error61, + error: error62, code(cxt) { const { gen, data, schemaCode, keyword, it } = cxt; const { opts, self } = it; @@ -7181,12 +7238,12 @@ var require_dist = __commonJS({ throw new Error(`Unknown format "${name}"`); return f; }; - function addFormats(ajv, list, fs5, exportName) { + function addFormats(ajv, list, fs6, exportName) { var _a3; var _b; (_a3 = (_b = ajv.opts.code).formats) !== null && _a3 !== void 0 ? _a3 : _b.formats = (0, codegen_1._)`require("ajv-formats/dist/formats").${exportName}`; for (const f of list) - ajv.addFormat(f, fs5[f]); + ajv.addFormat(f, fs6[f]); } module.exports = exports = formatsPlugin; Object.defineProperty(exports, "__esModule", { value: true }); @@ -7226,6 +7283,7 @@ __export(core_exports2, { $ZodCheckMultipleOf: () => $ZodCheckMultipleOf, $ZodCheckNumberFormat: () => $ZodCheckNumberFormat, $ZodCheckOverwrite: () => $ZodCheckOverwrite, + $ZodCheckProperties: () => $ZodCheckProperties, $ZodCheckProperty: () => $ZodCheckProperty, $ZodCheckRegex: () => $ZodCheckRegex, $ZodCheckSizeEquals: () => $ZodCheckSizeEquals, @@ -7250,6 +7308,7 @@ __export(core_exports2, { $ZodFile: () => $ZodFile, $ZodFunction: () => $ZodFunction, $ZodGUID: () => $ZodGUID, + $ZodIBAN: () => $ZodIBAN, $ZodIPv4: () => $ZodIPv4, $ZodIPv6: () => $ZodIPv6, $ZodISODate: () => $ZodISODate, @@ -7351,6 +7410,7 @@ __export(core_exports2, { _gt: () => _gt, _gte: () => _gte, _guid: () => _guid, + _iban: () => _iban, _includes: () => _includes, _int: () => _int, _int32: () => _int32, @@ -7442,6 +7502,9 @@ __export(core_exports2, { _void: () => _void, _xid: () => _xid, _xor: () => _xor, + base64Charset: () => base64Charset, + base64urlCharset: () => base64urlCharset, + canParseURL: () => canParseURL, clone: () => clone, compile: () => compile, compileFn: () => compileFn, @@ -7468,6 +7531,7 @@ __export(core_exports2, { isValidBase64URL: () => isValidBase64URL, isValidCIDRv6: () => isValidCIDRv6, isValidCreditCard: () => isValidCreditCard, + isValidIBAN: () => isValidIBAN, isValidIPv6: () => isValidIPv6, isValidJWT: () => isValidJWT, locales: () => locales_exports, @@ -7478,7 +7542,8 @@ __export(core_exports2, { parseAsync: () => parseAsync, parseURLObject: () => parseURLObject, prettifyError: () => prettifyError, - process: () => process2, + process: () => processSchema, + processSchema: () => processSchema, regexes: () => regexes_exports, registry: () => registry, safeDecode: () => safeDecode, @@ -7498,7 +7563,9 @@ __export(core_exports2, { util: () => util_exports, validate: () => validate, validateAsync: () => validateAsync, - version: () => version + validateURL: () => validateURL, + version: () => version, + withParser: () => withParser }); // node_modules/zod/v4/core/util.js @@ -7530,6 +7597,7 @@ __export(util_exports, { createTransparentProxy: () => createTransparentProxy, defineLazy: () => defineLazy, defineLazyInternal: () => defineLazyInternal, + derived: () => derived, esc: () => esc, escapeRegex: () => escapeRegex, explicitlyAborted: () => explicitlyAborted, @@ -7567,9 +7635,11 @@ __export(util_exports, { promiseAllObject: () => promiseAllObject, propertyKeyTypes: () => propertyKeyTypes, randomString: () => randomString, + rawShape: () => rawShape, required: () => required, safeExtend: () => safeExtend, shallowClone: () => shallowClone, + sizeCaps: () => sizeCaps, slugify: () => slugify, stringifyPrimitive: () => stringifyPrimitive, toZod: () => toZod, @@ -7607,18 +7677,22 @@ function jsonStringifyReplacer(_2, value) { return value.toString(); return value; } -function cached(getter) { - const set2 = false; - return { - get value() { - if (!set2) { - const value = getter(); - Object.defineProperty(this, "value", { value }); - return value; - } - throw new Error("cached value already set"); +var Cached = class { + constructor(getter) { + this._getter = getter; + this._value = void 0; + } + get value() { + const getter = this._getter; + if (getter !== void 0) { + this._value = getter(); + this._getter = void 0; } - }; + return this._value; + } +}; +function cached(getter) { + return new Cached(getter); } function nullish(input2) { return input2 === null || input2 === void 0; @@ -7637,9 +7711,9 @@ function floatSafeRemainder(val, step) { return ratio - roundedRatio; } var EVALUATING = /* @__PURE__ */ Symbol("evaluating"); -function defineLazy(object3, key, getter) { +function defineLazy(object4, key, getter) { let value = void 0; - Object.defineProperty(object3, key, { + Object.defineProperty(object4, key, { get() { if (value === EVALUATING) { return void 0; @@ -7651,7 +7725,7 @@ function defineLazy(object3, key, getter) { return value; }, set(v2) { - Object.defineProperty(object3, key, { + Object.defineProperty(object4, key, { value: v2 // configurable: true, }); @@ -7670,6 +7744,56 @@ function assignProp(target, prop, value) { configurable: true }); } +function rawShape(def) { + const desc = Object.getOwnPropertyDescriptor(def, "shape"); + return desc?.get ? desc.get.raw : desc?.value; +} +function sourceShape(schema) { + return rawShape(schema._zod.def) ?? schema._zod.def.shape; +} +function deferProp(target, key, getter) { + Object.defineProperty(target, key, { + get() { + const value = getter(); + assignProp(this, key, value); + return value; + }, + enumerable: true, + configurable: true + }); +} +function putProp(target, key, value) { + if (key in target) + assignProp(target, key, value); + else + target[key] = value; +} +function mirrorShape(target, source, keys2, wrap) { + const raw = sourceShape(source); + for (const key of keys2) { + const desc = Object.getOwnPropertyDescriptor(raw, key); + if (!desc.enumerable) + continue; + if (desc.get) { + deferProp(target, key, () => { + const value = source._zod.def.shape[key]; + return wrap ? wrap(value, key) : value; + }); + } else + putProp(target, key, wrap ? wrap(desc.value, key) : desc.value); + } +} +function mirrorProps(target, source) { + for (const key of Reflect.ownKeys(source)) { + const desc = Object.getOwnPropertyDescriptor(source, key); + if (!desc.enumerable) + continue; + if (desc.get) + deferProp(target, key, () => source[key]); + else + putProp(target, key, desc.value); + } +} function mergeDefs(...defs) { const mergedDescriptors = {}; for (const def of defs) { @@ -7681,18 +7805,18 @@ function mergeDefs(...defs) { function cloneDef(schema) { return mergeDefs(schema._zod.def); } -function getElementAtPath(obj, path3) { - if (!path3) +function getElementAtPath(obj, path4) { + if (!path4) return obj; - return path3.reduce((acc, key) => acc?.[key], obj); + return path4.reduce((acc, key) => acc?.[key], obj); } function promiseAllObject(promisesObj) { - const keys = Object.keys(promisesObj); - const promises = keys.map((key) => promisesObj[key]); + const keys2 = Object.keys(promisesObj); + const promises = keys2.map((key) => promisesObj[key]); return Promise.all(promises).then((results) => { const resolvedObj = {}; - for (let i = 0; i < keys.length; i++) { - resolvedObj[keys[i]] = results[i]; + for (let i = 0; i < keys2.length; i++) { + resolvedObj[keys2[i]] = results[i]; } return resolvedObj; }); @@ -7908,23 +8032,21 @@ function pick(schema, mask) { if (hasChecks) { throw new Error(".pick() cannot be used on object schemas containing refinements"); } - const def = mergeDefs(schema._zod.def, { - get shape() { - const newShape = {}; - for (const key of Reflect.ownKeys(mask)) { - if (!Object.prototype.hasOwnProperty.call(currDef.shape, key)) { - throw new Error(`Unrecognized key: "${String(key)}"`); - } - if (!mask[key]) - continue; - assignProp(newShape, key, currDef.shape[key]); - } - assignProp(this, "shape", newShape); - return newShape; - }, - checks: [] - }); - return clone(schema, def); + const newShape = {}; + mirrorShape(newShape, schema, maskedKeys(schema, mask)); + return clone(schema, mergeDefs(currDef, { shape: newShape, checks: [] })); +} +function maskedKeys(schema, mask) { + const raw = sourceShape(schema); + const keys2 = []; + for (const key of Reflect.ownKeys(mask)) { + if (!Object.getOwnPropertyDescriptor(raw, key)?.enumerable) { + throw new Error(`Unrecognized key: "${String(key)}"`); + } + if (mask[key]) + keys2.push(key); + } + return keys2; } function omit(schema, mask) { const currDef = schema._zod.def; @@ -7933,23 +8055,10 @@ function omit(schema, mask) { if (hasChecks) { throw new Error(".omit() cannot be used on object schemas containing refinements"); } - const def = mergeDefs(schema._zod.def, { - get shape() { - const newShape = { ...schema._zod.def.shape }; - for (const key of Reflect.ownKeys(mask)) { - if (!Object.prototype.hasOwnProperty.call(currDef.shape, key)) { - throw new Error(`Unrecognized key: "${String(key)}"`); - } - if (!mask[key]) - continue; - delete newShape[key]; - } - assignProp(this, "shape", newShape); - return newShape; - }, - checks: [] - }); - return clone(schema, def); + const omitted = new Set(maskedKeys(schema, mask)); + const newShape = {}; + mirrorShape(newShape, schema, Reflect.ownKeys(sourceShape(schema)).filter((key) => !omitted.has(key))); + return clone(schema, mergeDefs(currDef, { shape: newShape, checks: [] })); } function extend(schema, shape) { if (!isPlainObject(shape)) { @@ -7958,34 +8067,26 @@ function extend(schema, shape) { const checks = schema._zod.def.checks; const hasChecks = checks && checks.length > 0; if (hasChecks) { - const existingShape = schema._zod.def.shape; + const existingShape = sourceShape(schema); for (const key of Reflect.ownKeys(shape)) { if (Object.getOwnPropertyDescriptor(existingShape, key) !== void 0) { throw new Error("Cannot overwrite keys on object schemas containing refinements. Use `.safeExtend()` instead."); } } } - const def = mergeDefs(schema._zod.def, { - get shape() { - const _shape = { ...schema._zod.def.shape, ...shape }; - assignProp(this, "shape", _shape); - return _shape; - } - }); - return clone(schema, def); + return clone(schema, mergeDefs(schema._zod.def, { shape: extended(schema, shape) })); +} +function extended(schema, shape) { + const newShape = {}; + mirrorShape(newShape, schema, Reflect.ownKeys(sourceShape(schema))); + mirrorProps(newShape, shape); + return newShape; } function safeExtend(schema, shape) { if (!isPlainObject(shape)) { throw new Error("Invalid input to safeExtend: expected a plain object"); } - const def = mergeDefs(schema._zod.def, { - get shape() { - const _shape = { ...schema._zod.def.shape, ...shape }; - assignProp(this, "shape", _shape); - return _shape; - } - }); - return clone(schema, def); + return clone(schema, mergeDefs(schema._zod.def, { shape: extended(schema, shape) })); } function merge(a, b2) { if (!b2?._zod?.def) { @@ -7994,12 +8095,11 @@ function merge(a, b2) { if (a._zod.def.checks?.length) { throw new Error(".merge() cannot be used on object schemas containing refinements. Use .safeExtend() instead."); } + const newShape = {}; + mirrorShape(newShape, a, Reflect.ownKeys(sourceShape(a))); + mirrorShape(newShape, b2, Reflect.ownKeys(sourceShape(b2))); const def = mergeDefs(a._zod.def, { - get shape() { - const _shape = { ...a._zod.def.shape, ...b2._zod.def.shape }; - assignProp(this, "shape", _shape); - return _shape; - }, + shape: newShape, get catchall() { return b2._zod.def.catchall; }, @@ -8014,67 +8114,19 @@ function partial(Class2, schema, mask, name = "partial") { if (hasChecks) { throw new Error(`.${name}() cannot be used on object schemas containing refinements`); } - const def = mergeDefs(schema._zod.def, { - get shape() { - const oldShape = schema._zod.def.shape; - const shape = { ...oldShape }; - if (mask) { - for (const key of Reflect.ownKeys(mask)) { - if (!Object.prototype.hasOwnProperty.call(oldShape, key)) { - throw new Error(`Unrecognized key: "${String(key)}"`); - } - if (!mask[key]) - continue; - shape[key] = Class2 ? new Class2({ - type: "optional", - innerType: oldShape[key] - }) : oldShape[key]; - } - } else { - for (const key of Reflect.ownKeys(oldShape)) { - shape[key] = Class2 ? new Class2({ - type: "optional", - innerType: oldShape[key] - }) : oldShape[key]; - } - } - assignProp(this, "shape", shape); - return shape; - }, - checks: [] - }); - return clone(schema, def); + const selected = mask ? new Set(maskedKeys(schema, mask)) : void 0; + const newShape = {}; + mirrorShape(newShape, schema, Reflect.ownKeys(sourceShape(schema)), Class2 && ((value, key) => selected && !selected.has(key) ? value : new Class2({ type: "optional", innerType: value }))); + return clone(schema, mergeDefs(schema._zod.def, { shape: newShape, checks: [] })); } function required(Class2, schema, mask) { - const def = mergeDefs(schema._zod.def, { - get shape() { - const oldShape = schema._zod.def.shape; - const shape = { ...oldShape }; - if (mask) { - for (const key of Reflect.ownKeys(mask)) { - if (!Object.prototype.hasOwnProperty.call(shape, key)) { - throw new Error(`Unrecognized key: "${String(key)}"`); - } - if (!mask[key]) - continue; - shape[key] = new Class2({ - type: "nonoptional", - innerType: oldShape[key] - }); - } - } else { - for (const key of Reflect.ownKeys(oldShape)) { - shape[key] = new Class2({ - type: "nonoptional", - innerType: oldShape[key] - }); - } - } - assignProp(this, "shape", shape); - return shape; - } - }); - return clone(schema, def); + const selected = mask ? new Set(maskedKeys(schema, mask)) : void 0; + const newShape = {}; + mirrorShape(newShape, schema, Reflect.ownKeys(sourceShape(schema)), (value, key) => ( + // overwrite with non-optional + selected && !selected.has(key) ? value : new Class2({ type: "nonoptional", innerType: value }) + )); + return clone(schema, mergeDefs(schema._zod.def, { shape: newShape })); } function aborted(x2, startIndex = 0) { if (x2.aborted === true) @@ -8096,17 +8148,29 @@ function explicitlyAborted(x2, startIndex = 0) { } return false; } -function prefixIssues(path3, issues) { +function prefixIssues(path4, issues) { return issues.map((iss) => { var _a3; (_a3 = iss).path ?? (_a3.path = []); - iss.path.unshift(path3); + iss.path.unshift(path4); return iss; }); } function unwrapMessage(message) { return typeof message === "string" ? message : message?.message; } +function sizeCaps(defChecks) { + const caps = []; + for (const ch of defChecks ?? []) { + const d2 = ch._zod.def; + if (!/_(length|size)$|_equals$/.test(d2.check)) + break; + const max = d2.maximum ?? d2.length ?? d2.size; + if (max >= 0) + caps.push([max, ch]); + } + return caps; +} function attachSchema(issues, start, inst) { var _a3; for (let i = start; i < issues.length; i++) { @@ -8124,13 +8188,18 @@ function finalizeIssue(iss, ctx, config2) { } const schemaError = iss.schema !== iss.inst ? iss.schema?._zod.def?.error : void 0; const message = iss.message ? iss.message : unwrapMessage(iss.inst?._zod.def?.error?.(iss)) ?? unwrapMessage(schemaError?.(iss)) ?? unwrapMessage(ctx?.error?.(iss)) ?? unwrapMessage(config2.customError?.(iss)) ?? unwrapMessage(config2.localeError?.(iss)) ?? "Invalid input"; - const { inst: _inst, schema: _schema, continue: _continue, input: _input, ...rest } = iss; - rest.path ?? (rest.path = []); - rest.message = message; + const full = {}; + for (const k2 of Object.keys(iss)) { + if (k2 === "inst" || k2 === "schema" || k2 === "continue" || k2 === "input" || k2 === "__proto__") + continue; + full[k2] = iss[k2]; + } + full.path ?? (full.path = []); + full.message = message; if (ctx?.reportInput) { - rest.input = _input; + full.input = iss.input; } - return rest; + return full; } function getSizableOrigin(input2) { if (input2 instanceof Set) @@ -8257,6 +8326,23 @@ function own(inst, key, value, enumerable = true) { function hide(inst, key, value) { return own(inst, key, value, false); } +// @__NO_SIDE_EFFECTS__ +function derived(computes, table) { + for (const key in computes) { + const compute = computes[key]; + Object.defineProperty(table, key, { + configurable: true, + enumerable: true, + get() { + return own(this, key, compute(this)); + }, + set(value) { + own(this, key, value); + } + }); + } + return table; +} function defineBound(proto, key, fn) { Object.defineProperty(proto, key, { configurable: true, @@ -8384,8 +8470,7 @@ function $constructor(name, initializer3, proto, params) { } finally { _zodDesc.value = void 0; } - } - if (inst._zod.traits.has(name)) { + } else if (inst._zod.traits.has(name)) { return; } inst._zod.traits.add(name); @@ -8476,16 +8561,12 @@ var _messageDesc = { enumerable: true, configurable: true }; -var _zodDesc2 = { value: void 0, enumerable: false }; var _issuesDesc = { value: void 0, enumerable: false }; var _installedToString = /* @__PURE__ */ new WeakSet([Object.prototype, Error.prototype]); var initializer = (inst, def) => { inst.name = "$ZodError"; - _zodDesc2.value = inst._zod; - Object.defineProperty(inst, "_zod", _zodDesc2); _issuesDesc.value = def; Object.defineProperty(inst, "issues", _issuesDesc); - _zodDesc2.value = void 0; _issuesDesc.value = void 0; Object.defineProperty(inst, "message", _messageDesc); const proto = Object.getPrototypeOf(inst); @@ -8519,10 +8600,10 @@ function node(obj, key, make) { } return obj[key]; } -function flattenError(error61, mapper = (issue2) => issue2.message) { +function flattenError(error62, mapper = (issue2) => issue2.message) { const fieldErrors = {}; const formErrors = []; - for (const sub of error61.issues) { + for (const sub of error62.issues) { if (sub.path.length > 0) { node(fieldErrors, sub.path[0], () => []).push(mapper(sub)); } else { @@ -8531,18 +8612,18 @@ function flattenError(error61, mapper = (issue2) => issue2.message) { } return { formErrors, fieldErrors }; } -function formatError(error61, mapper = (issue2) => issue2.message) { +function formatError(error62, mapper = (issue2) => issue2.message) { const fieldErrors = { _errors: [] }; - const processError = (error62, path3 = []) => { - for (const issue2 of error62.issues) { + const processError = (error63, path4 = []) => { + for (const issue2 of error63.issues) { if (issue2.code === "invalid_union" && issue2.errors.length) { - issue2.errors.map((issues) => processError({ issues }, [...path3, ...issue2.path])); + issue2.errors.map((issues) => processError({ issues }, [...path4, ...issue2.path])); } else if (issue2.code === "invalid_key") { - processError({ issues: issue2.issues }, [...path3, ...issue2.path]); + processError({ issues: issue2.issues }, [...path4, ...issue2.path]); } else if (issue2.code === "invalid_element") { - processError({ issues: issue2.issues }, [...path3, ...issue2.path]); + processError({ issues: issue2.issues }, [...path4, ...issue2.path]); } else { - const fullpath = [...path3, ...issue2.path]; + const fullpath = [...path4, ...issue2.path]; if (fullpath.length === 0) { fieldErrors._errors.push(mapper(issue2)); } else { @@ -8576,22 +8657,22 @@ function formatError(error61, mapper = (issue2) => issue2.message) { } } }; - processError(error61); + processError(error62); return fieldErrors; } -function treeifyError(error61, mapper = (issue2) => issue2.message) { +function treeifyError(error62, mapper = (issue2) => issue2.message) { const result = { errors: [] }; - const processError = (error62, path3 = []) => { + const processError = (error63, path4 = []) => { var _a3; - for (const issue2 of error62.issues) { + for (const issue2 of error63.issues) { if (issue2.code === "invalid_union" && issue2.errors.length) { - issue2.errors.map((issues) => processError({ issues }, [...path3, ...issue2.path])); + issue2.errors.map((issues) => processError({ issues }, [...path4, ...issue2.path])); } else if (issue2.code === "invalid_key") { - processError({ issues: issue2.issues }, [...path3, ...issue2.path]); + processError({ issues: issue2.issues }, [...path4, ...issue2.path]); } else if (issue2.code === "invalid_element") { - processError({ issues: issue2.issues }, [...path3, ...issue2.path]); + processError({ issues: issue2.issues }, [...path4, ...issue2.path]); } else { - const fullpath = [...path3, ...issue2.path]; + const fullpath = [...path4, ...issue2.path]; if (fullpath.length === 0) { result.errors.push(mapper(issue2)); continue; @@ -8625,13 +8706,13 @@ function treeifyError(error61, mapper = (issue2) => issue2.message) { } } }; - processError(error61); + processError(error62); return result; } function toDotPath(_path) { const segs = []; - const path3 = _path.map((seg) => typeof seg === "object" ? seg.key : seg); - for (const seg of path3) { + const path4 = _path.map((seg) => typeof seg === "object" ? seg.key : seg); + for (const seg of path4) { if (typeof seg === "number") segs.push(`[${seg}]`); else if (typeof seg === "symbol") @@ -8646,9 +8727,9 @@ function toDotPath(_path) { } return segs.join(""); } -function prettifyError(error61) { +function prettifyError(error62) { const lines = []; - const issues = [...error61.issues].sort((a, b2) => (a.path ?? []).length - (b2.path ?? []).length); + const issues = [...error62.issues].sort((a, b2) => (a.path ?? []).length - (b2.path ?? []).length); for (const issue2 of issues) { lines.push(`\u2716 ${issue2.message}`); if (issue2.path?.length) @@ -8700,33 +8781,50 @@ var _safeParse = (_Err) => (schema, value, _ctx) => { if (result instanceof Promise) { throw new $ZodAsyncError(); } - return result.issues.length ? { - success: false, - error: new (_Err ?? $ZodError)(result.issues.map((iss) => finalizeIssue(iss, ctx, config()))) - } : { success: true, data: result.value }; + return result.issues.length ? failure(_Err, result.issues, ctx) : { success: true, data: result.value }; }; var safeParse = /* @__PURE__ */ _safeParse($ZodRealError); +function failure(Err, issues, ctx) { + let error62; + return { + success: false, + get error() { + if (!error62) { + error62 = new Err(issues.map((iss) => finalizeIssue(iss, ctx, config()))); + issues = void 0; + ctx = void 0; + } + return error62; + }, + set error(e) { + error62 = e; + issues = void 0; + ctx = void 0; + } + }; +} var _safeParseAsync = (_Err) => async (schema, value, _ctx) => { const ctx = _ctx ? { ..._ctx, async: true } : { async: true }; let result = schema._zod.run({ value, issues: [] }, ctx); if (result instanceof Promise) result = await result; - return result.issues.length ? { - success: false, - error: new _Err(result.issues.map((iss) => finalizeIssue(iss, ctx, config()))) - } : { success: true, data: result.value }; + return result.issues.length ? failure(_Err, result.issues, ctx) : { success: true, data: result.value }; }; var safeParseAsync = /* @__PURE__ */ _safeParseAsync($ZodRealError); var COMPILE_INVALID = /* @__PURE__ */ Symbol.for("zod.compile.invalid"); var COMPILE_FALLBACK = /* @__PURE__ */ Symbol.for("zod.compile.fallback"); var validate = ((schema, value, _ctx) => { const validator = schema._zod.bag.validator; - if (validator !== void 0 && validator(value) !== COMPILE_INVALID) - return true; + if (validator !== void 0) { + if (validator(value) !== COMPILE_INVALID) + return true; + if (validator.definite === true && _ctx === void 0) + return false; + } return validateFallback(schema, value, _ctx); }); function validateFallback(schema, value, _ctx) { - const ctx = _ctx ? { ..._ctx, async: false } : { async: false }; + const ctx = _ctx ? { ..._ctx, async: false, abortEarly: true } : { async: false, abortEarly: true }; const fallbackRun = schema._zod.bag.fallbackRun; let result; if (fallbackRun) { @@ -8741,7 +8839,7 @@ function validateFallback(schema, value, _ctx) { return result.issues.length === 0; } var validateAsync = async (schema, value, _ctx) => { - const ctx = _ctx ? { ..._ctx, async: true } : { async: true }; + const ctx = _ctx ? { ..._ctx, async: true, abortEarly: true } : { async: true, abortEarly: true }; let result = schema._zod.run({ value, issues: [] }, ctx); if (result instanceof Promise) result = await result; @@ -8803,6 +8901,7 @@ var safeDecodeAsync = /* @__PURE__ */ _safeDecodeAsync($ZodRealError); // node_modules/zod/v4/core/regexes.js var regexes_exports = {}; __export(regexes_exports, { + anyString: () => anyString, base64: () => base64, base64url: () => base64url, bigint: () => bigint, @@ -8813,6 +8912,7 @@ __export(regexes_exports, { creditCard: () => creditCard, cuid: () => cuid, cuid2: () => cuid2, + currencyCode: () => currencyCode, date: () => date, datetime: () => datetime, domain: () => domain, @@ -8826,6 +8926,7 @@ __export(regexes_exports, { hostname: () => hostname, html5Email: () => html5Email, httpProtocol: () => httpProtocol, + iban: () => iban, idnEmail: () => idnEmail, integer: () => integer, ipv4: () => ipv4, @@ -8885,13 +8986,13 @@ var uuid = (version2) => { var uuid4 = /* @__PURE__ */ uuid(4); var uuid6 = /* @__PURE__ */ uuid(6); var uuid7 = /* @__PURE__ */ uuid(7); -var email = /^(?!\.)(?!.*\.\.)([A-Za-z0-9_'+\-\.]*)[A-Za-z0-9_+-]@([A-Za-z0-9][A-Za-z0-9\-]*\.)+[A-Za-z]{2,}$/; +var email = /^(?:[A-Za-z0-9_'+\-]+\.)*[A-Za-z0-9_'+\-]*[A-Za-z0-9_+-]@(?:[A-Za-z0-9][A-Za-z0-9\-]*\.)+[A-Za-z]{2,}$/; var html5Email = /^[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$/; var rfc5322Email = /^(([^<>()\[\]\\.,;:\s@"]+(\.[^<>()\[\]\\.,;:\s@"]+)*)|(".+"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/; var unicodeEmail = /^[^\s@"]{1,64}@[^\s@]{1,255}$/u; var idnEmail = unicodeEmail; var browserEmail = /^[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$/; -var _emoji = `^[\\p{Extended_Pictographic}\\p{Emoji_Component}]+$`; +var _emoji = `^(?=[\\s\\S]*[\\p{Extended_Pictographic}\\p{Regional_Indicator}\\u20E3])[\\p{Extended_Pictographic}\\p{Emoji_Component}]+$`; function emoji() { return new RegExp(_emoji, "u"); } @@ -8904,12 +9005,14 @@ var mac = (delimiter) => { var cidrv4 = /^((25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9])\.){3}(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9])\/([0-9]|[1-2][0-9]|3[0-2])$/; var cidrv6 = /^(([0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:))\/(12[0-8]|1[01][0-9]|[1-9]?[0-9])$/; var base64 = /^$|^(?:[0-9a-zA-Z+/]{4})*(?:(?:[0-9a-zA-Z+/]{2}==)|(?:[0-9a-zA-Z+/]{3}=))?$/; -var base64url = /^[A-Za-z0-9_-]*$/; +var base64url = /^(?:[A-Za-z0-9_-]{4})*(?:[A-Za-z0-9_-]{2,3})?$/; var hostname = /^(?=.{1,253}\.?$)[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[-0-9a-zA-Z]{0,61}[0-9a-zA-Z])?)*\.?$/; var domain = /^(?=.{1,253}$)([a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?\.)+[a-zA-Z]{2,63}$/; var httpProtocol = /^https?$/; var e164 = /^\+[1-9]\d{6,14}$/; var creditCard = /^\d(?:[ -]?\d){11,18}$/; +var currencyCode = /^(?:AED|AFN|ALL|AMD|AOA|ARS|AUD|AWG|AZN|BAM|BBD|BDT|BHD|BIF|BMD|BND|BOB|BOV|BRL|BSD|BTN|BWP|BYN|BZD|CAD|CDF|CHE|CHF|CHW|CLF|CLP|CNY|COP|COU|CRC|CUP|CVE|CZK|DJF|DKK|DOP|DZD|EGP|ERN|ETB|EUR|FJD|FKP|GBP|GEL|GHS|GIP|GMD|GNF|GTQ|GYD|HKD|HNL|HTG|HUF|IDR|ILS|INR|IQD|IRR|ISK|JMD|JOD|JPY|KES|KGS|KHR|KMF|KPW|KRW|KWD|KYD|KZT|LAK|LBP|LKR|LRD|LSL|LYD|MAD|MDL|MGA|MKD|MMK|MNT|MOP|MRU|MUR|MVR|MWK|MXN|MXV|MYR|MZN|NAD|NGN|NIO|NOK|NPR|NZD|OMR|PAB|PEN|PGK|PHP|PKR|PLN|PYG|QAR|RON|RSD|RUB|RWF|SAR|SBD|SCR|SDG|SEK|SGD|SHP|SLE|SOS|SRD|SSP|STN|SVC|SYP|SZL|THB|TJS|TMT|TND|TOP|TRY|TTD|TWD|TZS|UAH|UGX|USD|USN|UYI|UYU|UYW|UZS|VED|VES|VND|VUV|WST|XAD|XAF|XAG|XAU|XBA|XBB|XBC|XBD|XCD|XCG|XDR|XOF|XPD|XPF|XPT|XSU|XTS|XUA|XXX|YER|ZAR|ZMW|ZWG)$/; +var iban = /^[A-Z]{2}(?!00|01|99)\d{2}[A-Z0-9]{11,30}$/; var dateSource = `(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))`; function anchor(source) { return new RegExp(`^${source}$`); @@ -8931,6 +9034,7 @@ function datetime(args) { const timeRegex = args.local ? `${qualified}|${timeSource({ precision: args.precision })}` : qualified; return new RegExp(`^${dateSource}T(?:${timeRegex})$`); } +var anyString = /^[\s\S]{0,}$/; var string = (params) => { const regex = params ? `[\\s\\S]{${params?.minimum ?? 0},${params?.maximum ?? ""}}` : `[\\s\\S]*`; return new RegExp(`^${regex}$`); @@ -8989,16 +9093,6 @@ var numericOriginMap = { var $ZodCheckLessThan = /* @__PURE__ */ $constructor("$ZodCheckLessThan", (inst, def) => { $ZodCheck.init(inst, def); const origin = numericOriginMap[typeof def.value]; - inst._zod.onattach.push((inst2) => { - const bag = inst2._zod.bag; - const curr = (def.inclusive ? bag.maximum : bag.exclusiveMaximum) ?? Number.POSITIVE_INFINITY; - if (def.value < curr) { - if (def.inclusive) - bag.maximum = def.value; - else - bag.exclusiveMaximum = def.value; - } - }); inst._zod.check = (payload) => { if (def.inclusive ? payload.value <= def.value : payload.value < def.value) { return; @@ -9017,16 +9111,6 @@ var $ZodCheckLessThan = /* @__PURE__ */ $constructor("$ZodCheckLessThan", (inst, var $ZodCheckGreaterThan = /* @__PURE__ */ $constructor("$ZodCheckGreaterThan", (inst, def) => { $ZodCheck.init(inst, def); const origin = numericOriginMap[typeof def.value]; - inst._zod.onattach.push((inst2) => { - const bag = inst2._zod.bag; - const curr = (def.inclusive ? bag.minimum : bag.exclusiveMinimum) ?? Number.NEGATIVE_INFINITY; - if (def.value > curr) { - if (def.inclusive) - bag.minimum = def.value; - else - bag.exclusiveMinimum = def.value; - } - }); inst._zod.check = (payload) => { if (def.inclusive ? payload.value >= def.value : payload.value > def.value) { return; @@ -9044,10 +9128,6 @@ var $ZodCheckGreaterThan = /* @__PURE__ */ $constructor("$ZodCheckGreaterThan", }); var $ZodCheckMultipleOf = /* @__PURE__ */ $constructor("$ZodCheckMultipleOf", (inst, def) => { $ZodCheck.init(inst, def); - inst._zod.onattach.push((inst2) => { - var _a3; - (_a3 = inst2._zod.bag).multipleOf ?? (_a3.multipleOf = def.value); - }); inst._zod.check = (payload) => { if (typeof payload.value !== typeof def.value) throw new Error("Cannot mix number and bigint in multiple_of check."); @@ -9073,14 +9153,6 @@ var $ZodCheckNumberFormat = /* @__PURE__ */ $constructor("$ZodCheckNumberFormat" const isInt = def.format?.includes("int"); const origin = isInt ? "int" : "number"; const [minimum, maximum] = NUMBER_FORMAT_RANGES[def.format]; - inst._zod.onattach.push((inst2) => { - const bag = inst2._zod.bag; - bag.format = def.format; - bag.minimum = minimum; - bag.maximum = maximum; - if (isInt) - bag.pattern = integer; - }); inst._zod.check = (payload) => { const input2 = payload.value; if (isInt) { @@ -9149,12 +9221,6 @@ var $ZodCheckNumberFormat = /* @__PURE__ */ $constructor("$ZodCheckNumberFormat" var $ZodCheckBigIntFormat = /* @__PURE__ */ $constructor("$ZodCheckBigIntFormat", (inst, def) => { $ZodCheck.init(inst, def); const [minimum, maximum] = BIGINT_FORMAT_RANGES[def.format]; - inst._zod.onattach.push((inst2) => { - const bag = inst2._zod.bag; - bag.format = def.format; - bag.minimum = minimum; - bag.maximum = maximum; - }); inst._zod.check = (payload) => { const input2 = payload.value; if (input2 < minimum) { @@ -9185,11 +9251,6 @@ var $ZodCheckMaxSize = /* @__PURE__ */ $constructor("$ZodCheckMaxSize", (inst, d var _a3; $ZodCheck.init(inst, def); (_a3 = inst._zod.def).when ?? (_a3.when = _whenHasSize); - inst._zod.onattach.push((inst2) => { - const curr = inst2._zod.bag.maximum ?? Number.POSITIVE_INFINITY; - if (def.maximum < curr) - inst2._zod.bag.maximum = def.maximum; - }); inst._zod.check = (payload) => { const input2 = payload.value; const size = input2.size; @@ -9210,11 +9271,6 @@ var $ZodCheckMinSize = /* @__PURE__ */ $constructor("$ZodCheckMinSize", (inst, d var _a3; $ZodCheck.init(inst, def); (_a3 = inst._zod.def).when ?? (_a3.when = _whenHasSize); - inst._zod.onattach.push((inst2) => { - const curr = inst2._zod.bag.minimum ?? Number.NEGATIVE_INFINITY; - if (def.minimum > curr) - inst2._zod.bag.minimum = def.minimum; - }); inst._zod.check = (payload) => { const input2 = payload.value; const size = input2.size; @@ -9235,12 +9291,6 @@ var $ZodCheckSizeEquals = /* @__PURE__ */ $constructor("$ZodCheckSizeEquals", (i var _a3; $ZodCheck.init(inst, def); (_a3 = inst._zod.def).when ?? (_a3.when = _whenHasSize); - inst._zod.onattach.push((inst2) => { - const bag = inst2._zod.bag; - bag.minimum = def.size; - bag.maximum = def.size; - bag.size = def.size; - }); inst._zod.check = (payload) => { const input2 = payload.value; const size = input2.size; @@ -9262,11 +9312,6 @@ var $ZodCheckMaxLength = /* @__PURE__ */ $constructor("$ZodCheckMaxLength", (ins var _a3; $ZodCheck.init(inst, def); (_a3 = inst._zod.def).when ?? (_a3.when = _whenHasLength); - inst._zod.onattach.push((inst2) => { - const curr = inst2._zod.bag.maximum ?? Number.POSITIVE_INFINITY; - if (def.maximum < curr) - inst2._zod.bag.maximum = def.maximum; - }); inst._zod.check = (payload) => { const input2 = payload.value; const units = input2.length; @@ -9289,11 +9334,6 @@ var $ZodCheckMinLength = /* @__PURE__ */ $constructor("$ZodCheckMinLength", (ins var _a3; $ZodCheck.init(inst, def); (_a3 = inst._zod.def).when ?? (_a3.when = _whenHasLength); - inst._zod.onattach.push((inst2) => { - const curr = inst2._zod.bag.minimum ?? Number.NEGATIVE_INFINITY; - if (def.minimum > curr) - inst2._zod.bag.minimum = def.minimum; - }); inst._zod.check = (payload) => { const input2 = payload.value; const units = input2.length; @@ -9316,12 +9356,6 @@ var $ZodCheckLengthEquals = /* @__PURE__ */ $constructor("$ZodCheckLengthEquals" var _a3; $ZodCheck.init(inst, def); (_a3 = inst._zod.def).when ?? (_a3.when = _whenHasLength); - inst._zod.onattach.push((inst2) => { - const bag = inst2._zod.bag; - bag.minimum = def.length; - bag.maximum = def.length; - bag.length = def.length; - }); inst._zod.check = (payload) => { const input2 = payload.value; const units = input2.length; @@ -9344,14 +9378,6 @@ var $ZodCheckLengthEquals = /* @__PURE__ */ $constructor("$ZodCheckLengthEquals" var $ZodCheckStringFormat = /* @__PURE__ */ $constructor("$ZodCheckStringFormat", (inst, def) => { var _a3, _b; $ZodCheck.init(inst, def); - inst._zod.onattach.push((inst2) => { - const bag = inst2._zod.bag; - bag.format = def.format; - if (def.pattern) { - bag.patterns ?? (bag.patterns = /* @__PURE__ */ new Set()); - bag.patterns.add(def.pattern); - } - }); if (def.pattern) (_a3 = inst._zod).check ?? (_a3.check = (payload) => { def.pattern.lastIndex = 0; @@ -9401,11 +9427,6 @@ var $ZodCheckIncludes = /* @__PURE__ */ $constructor("$ZodCheckIncludes", (inst, const escapedRegex = escapeRegex(def.includes); const pattern = new RegExp(typeof def.position === "number" ? `^.{${def.position},}${escapedRegex}` : escapedRegex); def.pattern = pattern; - inst._zod.onattach.push((inst2) => { - const bag = inst2._zod.bag; - bag.patterns ?? (bag.patterns = /* @__PURE__ */ new Set()); - bag.patterns.add(pattern); - }); inst._zod.check = (payload) => { if (payload.value.includes(def.includes, def.position)) return; @@ -9424,11 +9445,6 @@ var $ZodCheckStartsWith = /* @__PURE__ */ $constructor("$ZodCheckStartsWith", (i $ZodCheck.init(inst, def); const pattern = new RegExp(`^${escapeRegex(def.prefix)}.*`); def.pattern ?? (def.pattern = pattern); - inst._zod.onattach.push((inst2) => { - const bag = inst2._zod.bag; - bag.patterns ?? (bag.patterns = /* @__PURE__ */ new Set()); - bag.patterns.add(pattern); - }); inst._zod.check = (payload) => { if (payload.value.startsWith(def.prefix)) return; @@ -9447,11 +9463,6 @@ var $ZodCheckEndsWith = /* @__PURE__ */ $constructor("$ZodCheckEndsWith", (inst, $ZodCheck.init(inst, def); const pattern = new RegExp(`.*${escapeRegex(def.suffix)}$`); def.pattern ?? (def.pattern = pattern); - inst._zod.onattach.push((inst2) => { - const bag = inst2._zod.bag; - bag.patterns ?? (bag.patterns = /* @__PURE__ */ new Set()); - bag.patterns.add(pattern); - }); inst._zod.check = (payload) => { if (payload.value.endsWith(def.suffix)) return; @@ -9485,12 +9496,37 @@ var $ZodCheckProperty = /* @__PURE__ */ $constructor("$ZodCheckProperty", (inst, return; }; }); +var $ZodCheckProperties = /* @__PURE__ */ $constructor("$ZodCheckProperties", (inst, def) => { + $ZodCheck.init(inst, def); + hide(inst, Symbol.iterator, function* () { + yield inst; + }); + let entries; + inst._zod.check = (payload) => { + if (payload.value == null) { + payload.issues.push({ expected: "object", code: "invalid_type", input: payload.value, inst }); + return void 0; + } + entries ?? (entries = Reflect.ownKeys(def.shape).map((key) => [key, def.shape[key]])); + const input2 = payload.value; + let proms; + for (const [key, schema] of entries) { + const result = schema._zod.run({ value: input2[key], issues: [] }, {}); + if (result instanceof Promise) { + proms ?? (proms = []); + proms.push(result.then((result2) => handleCheckPropertyResult(result2, payload, key))); + } else { + handleCheckPropertyResult(result, payload, key); + } + } + if (proms) + return Promise.all(proms).then(() => void 0); + return void 0; + }; +}); var $ZodCheckMimeType = /* @__PURE__ */ $constructor("$ZodCheckMimeType", (inst, def) => { $ZodCheck.init(inst, def); const mimeSet = new Set(def.mime); - inst._zod.onattach.push((inst2) => { - inst2._zod.bag.mime = def.mime; - }); inst._zod.check = (payload) => { if (mimeSet.has(payload.value.type)) return; @@ -9518,10 +9554,14 @@ var Doc = class { this.args = args; this.closed = closed; } + // the compiler catches a child's throw and keeps writing into this doc, so the indent has to unwind with it indented(fn) { this.indent += 1; - fn(this); - this.indent -= 1; + try { + fn(this); + } finally { + this.indent -= 1; + } } write(arg) { if (typeof arg === "function") { @@ -9550,8 +9590,8 @@ ${content.join("\n")} // node_modules/zod/v4/core/versions.js var version = { major: 4, - minor: 5, - patch: 4 + minor: 6, + patch: 7 }; // node_modules/zod/v4/core/schemas.js @@ -9664,15 +9704,22 @@ var $ZodType = /* @__PURE__ */ $constructor("$ZodType", (inst, def) => { own(this, "~standard", value); } }); -var toStandardResult = (r) => r.success ? { value: r.data } : { issues: r.error?.issues }; +var toStandardResult = (r, ctx) => r.issues.length ? { issues: r.issues.map((iss) => finalizeIssue(iss, ctx, config())) } : { value: r.value }; +async function validateAsync2(inst, value) { + const ctx = { async: true }; + return toStandardResult(await inst._zod.run({ value, issues: [] }, ctx), ctx); +} function standardProps(inst) { return { validate: (value) => { + const ctx = { async: false }; try { - return toStandardResult(safeParse(inst, value)); + const r = inst._zod.run({ value, issues: [] }, ctx); + if (!(r instanceof Promise)) + return toStandardResult(r, ctx); } catch (_2) { - return safeParseAsync(inst, value).then(toStandardResult); } + return validateAsync2(inst, value); }, vendor: "zod", version: 1 @@ -9680,7 +9727,7 @@ function standardProps(inst) { } var $ZodString = /* @__PURE__ */ $constructor("$ZodString", (inst, def) => { $ZodType.init(inst, def); - inst._zod.pattern = [...inst?._zod.bag?.patterns ?? []].pop() ?? string(inst._zod.bag); + inst._zod.pattern = def.pattern ?? anyString; inst._zod.parse = (payload, _2) => { if (def.coerce) try { @@ -9732,11 +9779,32 @@ var $ZodEmail = /* @__PURE__ */ $constructor("$ZodEmail", (inst, def) => { }); var URL_BAD_FORMAT = 1; var URL_UNPARSEABLE = 2; +function canParseURL(input2) { + try { + if (typeof URL !== "undefined" && typeof URL.canParse === "function") + return URL.canParse(input2); + new URL(input2); + return true; + } catch { + return false; + } +} +function validateURL(trimmed, def) { + if (!("normalize" in def) && !("hostname" in def) && !("protocol" in def)) { + return canParseURL(trimmed) || URL_UNPARSEABLE; + } + return parseURLObject(trimmed, def); +} function parseURLObject(trimmed, def) { if (!def.normalize && def.protocol?.source === httpProtocol.source && !/^https?:\/\//i.test(trimmed)) { return URL_BAD_FORMAT; } try { + if (typeof URL !== "undefined") { + const URLStatic = URL; + if (typeof URLStatic.parse === "function") + return URLStatic.parse(trimmed) ?? URL_UNPARSEABLE; + } return new URL(trimmed); } catch { return URL_UNPARSEABLE; @@ -9759,7 +9827,7 @@ var $ZodURL = /* @__PURE__ */ $constructor("$ZodURL", (inst, def) => { inst._zod.check = (payload) => { try { const trimmed = payload.value.trim(); - const url2 = parseURLObject(trimmed, def); + const url2 = validateURL(trimmed, def); if (url2 === URL_BAD_FORMAT) { payload.issues.push({ code: "invalid_format", @@ -9781,6 +9849,10 @@ var $ZodURL = /* @__PURE__ */ $constructor("$ZodURL", (inst, def) => { }); return; } + if (url2 === true) { + payload.value = stripTabAndNewline(trimmed); + return; + } if (def.hostname && !urlHostnameOk(url2, def.hostname)) { payload.issues.push({ code: "invalid_format", @@ -9849,12 +9921,6 @@ var $ZodKSUID = /* @__PURE__ */ $constructor("$ZodKSUID", (inst, def) => { var $ZodISODateTime = /* @__PURE__ */ $constructor("$ZodISODateTime", (inst, def) => { def.pattern ?? (def.pattern = datetime(def)); $ZodStringFormat.init(inst, def); - if (def.local || def.precision === -1) { - inst._zod.bag.laxFormat = true; - inst._zod.onattach.push((s) => { - s._zod.bag.laxFormat = true; - }); - } }); var $ZodISODate = /* @__PURE__ */ $constructor("$ZodISODate", (inst, def) => { def.pattern ?? (def.pattern = date); @@ -9871,23 +9937,16 @@ var $ZodISODuration = /* @__PURE__ */ $constructor("$ZodISODuration", (inst, def var $ZodIPv4 = /* @__PURE__ */ $constructor("$ZodIPv4", (inst, def) => { def.pattern ?? (def.pattern = ipv4); $ZodStringFormat.init(inst, def); - inst._zod.bag.format = `ipv4`; }); var ipv6Alphabet = /^[0-9a-fA-F:.]+$/; function isValidIPv6(value) { if (!ipv6Alphabet.test(value)) return false; - try { - new URL(`http://[${value}]`); - return true; - } catch { - return false; - } + return canParseURL(`http://[${value}]`); } var $ZodIPv6 = /* @__PURE__ */ $constructor("$ZodIPv6", (inst, def) => { def.pattern ?? (def.pattern = ipv6); $ZodStringFormat.init(inst, def); - inst._zod.bag.format = `ipv6`; inst._zod.check = (payload) => { if (!isValidIPv6(payload.value)) { payload.issues.push({ @@ -9903,7 +9962,6 @@ var $ZodIPv6 = /* @__PURE__ */ $constructor("$ZodIPv6", (inst, def) => { var $ZodMAC = /* @__PURE__ */ $constructor("$ZodMAC", (inst, def) => { def.pattern ?? (def.pattern = mac(def.delimiter)); $ZodStringFormat.init(inst, def); - inst._zod.bag.format = `mac`; }); var $ZodCIDRv4 = /* @__PURE__ */ $constructor("$ZodCIDRv4", (inst, def) => { def.pattern ?? (def.pattern = cidrv4); @@ -9952,10 +10010,10 @@ function isValidBase64(data) { return false; } } +var base64Charset = /^[0-9a-zA-Z+/]*={0,2}$/; var $ZodBase64 = /* @__PURE__ */ $constructor("$ZodBase64", (inst, def) => { - def.pattern ?? (def.pattern = base64); + def.pattern ?? (def.pattern = base64Charset); $ZodStringFormat.init(inst, def); - inst._zod.bag.contentEncoding = "base64"; inst._zod.check = (payload) => { if (isValidBase64(payload.value)) return; @@ -9968,17 +10026,17 @@ var $ZodBase64 = /* @__PURE__ */ $constructor("$ZodBase64", (inst, def) => { }); }; }); +var base64urlCharset = /^[A-Za-z0-9_-]*$/; function isValidBase64URL(data) { - if (!base64url.test(data)) + if (!base64urlCharset.test(data)) return false; const base643 = data.replace(/[-_]/g, (c) => c === "-" ? "+" : "/"); const padded = base643.padEnd(Math.ceil(base643.length / 4) * 4, "="); return isValidBase64(padded); } var $ZodBase64URL = /* @__PURE__ */ $constructor("$ZodBase64URL", (inst, def) => { - def.pattern ?? (def.pattern = base64url); + def.pattern ?? (def.pattern = base64urlCharset); $ZodStringFormat.init(inst, def); - inst._zod.bag.contentEncoding = "base64url"; inst._zod.check = (payload) => { if (isValidBase64URL(payload.value)) return; @@ -10001,7 +10059,7 @@ function isLuhnAlgo(digits) { let bit = 1; let sum = 0; while (length) { - const value = +digits[--length]; + const value = digits.charCodeAt(--length) - 48; bit ^= 1; sum += bit ? [0, 2, 4, 6, 8, 1, 3, 5, 7, 9][value] : value; } @@ -10027,6 +10085,39 @@ var $ZodCreditCard = /* @__PURE__ */ $constructor("$ZodCreditCard", (inst, def) }); }; }); +function isIso7064Mod97(iban3) { + let remainder = 0; + const len = iban3.length; + for (let i = 4; i < len; i++) { + const code = iban3.charCodeAt(i); + remainder = (code >= 65 ? remainder * 100 + (code - 55) : remainder * 10 + (code - 48)) % 97; + } + for (let i = 0; i < 4; i++) { + const code = iban3.charCodeAt(i); + remainder = (code >= 65 ? remainder * 100 + (code - 55) : remainder * 10 + (code - 48)) % 97; + } + return remainder === 1; +} +function isValidIBAN(input2) { + if (!iban.test(input2)) + return false; + return isIso7064Mod97(input2); +} +var $ZodIBAN = /* @__PURE__ */ $constructor("$ZodIBAN", (inst, def) => { + def.pattern ?? (def.pattern = iban); + $ZodStringFormat.init(inst, def); + inst._zod.check = (payload) => { + if (isValidIBAN(payload.value)) + return; + payload.issues.push({ + code: "invalid_format", + format: "iban", + input: payload.value, + inst, + continue: !def.abort + }); + }; +}); function isValidJWT(token, algorithm = null) { try { const tokensParts = token.split("."); @@ -10077,7 +10168,7 @@ var $ZodCustomStringFormat = /* @__PURE__ */ $constructor("$ZodCustomStringForma }); var $ZodNumber = /* @__PURE__ */ $constructor("$ZodNumber", (inst, def) => { $ZodType.init(inst, def); - inst._zod.pattern = inst._zod.bag.pattern ?? number; + inst._zod.pattern = number; inst._zod.parse = (payload, _ctx) => { if (def.coerce) try { @@ -10256,6 +10347,31 @@ var $ZodDate = /* @__PURE__ */ $constructor("$ZodDate", (inst, def) => { return payload; }; }); +function capSize(inst, size) { + const caps = sizeCaps(inst._zod.def.checks); + if (!caps.length) + return; + const cap = Math.min(...caps.map((c) => c[0])); + const parse3 = inst._zod.parse; + inst._zod.parse = (payload, ctx) => { + const n = size(payload.value); + if (n > cap && !ctx?.skipChecks && ctx?.direction !== "backward") { + for (const [max, ch] of sizeCaps(inst._zod.def.checks)) { + if (n > max && ch._zod.def.when(payload)) { + const start = payload.issues.length; + ch._zod.check(payload); + payload.issues[start].continue = false; + attachSchema(payload.issues, start, inst); + return payload; + } + } + } + return parse3(payload, ctx); + }; +} +var arraySize = (input2) => Array.isArray(input2) ? input2.length : -1; +var mapSize = (input2) => input2 instanceof Map ? input2.size : -1; +var setSize = (input2) => input2 instanceof Set ? input2.size : -1; function handleArrayResult(result, final, index) { if (result.issues.length) { final.issues.push(...prefixIssues(index, result.issues)); @@ -10279,6 +10395,7 @@ var $ZodArray = /* @__PURE__ */ $constructor("$ZodArray", (inst, def) => { } payload.value = memo2 ? memo2.alloc(inst, payload, Array(input2.length), ctx) : Array(input2.length); const proms = []; + const abortEarly = ctx?.abortEarly; for (let i = 0; i < input2.length; i++) { const item = input2[i]; const result = def.element._zod.run({ @@ -10289,6 +10406,8 @@ var $ZodArray = /* @__PURE__ */ $constructor("$ZodArray", (inst, def) => { proms.push(result.then((result2) => handleArrayResult(result2, payload, i))); } else { handleArrayResult(result, payload, i); + if (abortEarly && result.issues.length !== 0 && aborted(result)) + break; } } if (proms.length) { @@ -10296,6 +10415,7 @@ var $ZodArray = /* @__PURE__ */ $constructor("$ZodArray", (inst, def) => { } return payload; }; + capSize(inst, arraySize); }); function handlePropertyResult(result, final, key, input2, optin, optout) { const isPresent = key in input2; @@ -10321,7 +10441,7 @@ function handlePropertyResult(result, final, key, input2, optin, optout) { return; } if (result.value === void 0) { - if (isPresent) { + if (isPresent || optin === "defaulted" && !isOptionalOut) { final.value[key] = void 0; } } else { @@ -10330,10 +10450,10 @@ function handlePropertyResult(result, final, key, input2, optin, optout) { } var NO_SYMBOL_KEYS = []; function normalizeDef(def) { - const keys = Object.keys(def.shape); + const keys2 = Object.keys(def.shape); const ownSymbols = Object.getOwnPropertySymbols(def.shape); const symbolKeys = ownSymbols.length ? ownSymbols : NO_SYMBOL_KEYS; - const allKeys = symbolKeys.length ? [...keys, ...symbolKeys] : keys; + const allKeys = symbolKeys.length ? [...keys2, ...symbolKeys] : keys2; for (const k2 of allKeys) { if (!def.shape?.[k2]?._zod?.traits?.has("$ZodType")) { throw new Error(`Invalid element at key "${String(k2)}": expected a Zod schema`); @@ -10345,19 +10465,25 @@ function normalizeDef(def) { allKeys, symbolKeys, // string-only: handleCatchall matches it against `for...in`, which never yields a symbol - keySet: new Set(keys), - numKeys: keys.length, + keySet: new Set(keys2), + numKeys: keys2.length, optionalKeys: new Set(okeys) }; } -function handleCatchall(proms, input2, payload, ctx, def, inst) { +function handleCatchall(proms, input2, payload, ctx, def, inst, abortEarly) { const unrecognized = []; const keySet = def.keySet; const _catchall = def.catchall._zod; const t = _catchall.def.type; const optin = _catchall.optin; const optout = _catchall.optout; + let seen = 0; for (const key in input2) { + if (abortEarly && payload.issues.length !== seen) { + if (aborted(payload, seen)) + break; + seen = payload.issues.length; + } if (keySet.has(key)) continue; if (key === "__proto__") { @@ -10392,23 +10518,19 @@ function handleCatchall(proms, input2, payload, ctx, def, inst) { return payload; }); } -var propShapes = /* @__PURE__ */ new WeakMap(); var $ZodObject = /* @__PURE__ */ $constructor("$ZodObject", (inst, def) => { $ZodType.init(inst, def); const desc = Object.getOwnPropertyDescriptor(def, "shape"); - if (!desc?.get) { - const sh = def.shape; - propShapes.set(def, sh); - Object.defineProperty(def, "shape", { - get: () => { - const newSh = { ...sh }; - Object.defineProperty(def, "shape", { - value: newSh - }); - propShapes.set(def, newSh); - return newSh; - } - }); + const sh = desc?.get ? desc.get.raw : def.shape ?? {}; + if (sh) { + const get = () => { + const newSh = { ...sh }; + Object.defineProperty(def, "shape", { value: newSh }); + get.raw = newSh; + return newSh; + }; + get.raw = sh; + Object.defineProperty(def, "shape", { get }); } const _normalized = cached(() => normalizeDef(def)); defineLazyInternal(inst, "propValues", (zod) => { @@ -10448,7 +10570,14 @@ var $ZodObject = /* @__PURE__ */ $constructor("$ZodObject", (inst, def) => { payload.value = memo2 ? memo2.alloc(inst, payload, {}, ctx) : {}; const proms = []; const shape = value.shape; + const abortEarly = ctx?.abortEarly; + let seen = payload.issues.length; for (const key of value.allKeys) { + if (abortEarly && payload.issues.length !== seen) { + if (aborted(payload, seen)) + break; + seen = payload.issues.length; + } if (key === "__proto__") continue; const el = shape[key]; @@ -10464,7 +10593,7 @@ var $ZodObject = /* @__PURE__ */ $constructor("$ZodObject", (inst, def) => { if (!catchall) { return proms.length ? Promise.all(proms).then(() => payload) : payload; } - return handleCatchall(proms, input2, payload, ctx, _normalized.value, inst); + return handleCatchall(proms, input2, payload, ctx, _normalized.value, inst, abortEarly === true); }; }); var $ZodObjectJIT = /* @__PURE__ */ $constructor("$ZodObjectJIT", (inst, def) => { @@ -10478,10 +10607,16 @@ var $ZodObjectJIT = /* @__PURE__ */ $constructor("$ZodObjectJIT", (inst, def) => const doc = new Doc(["payload", "ctx"], { shape, inst, memo: memo2, syms }); const parseStr = (k2) => `shape[${k2}]._zod.run({ value: input[${k2}], issues: [] }, ctx)`; const prefixStr = (id, k2) => ` + let ${id}_ab = false; for (let i = 0; i < ${id}.issues.length; i++) { const iss = ${id}.issues[i]; iss.path = iss.path ? [${k2}, ...iss.path] : [${k2}]; payload.issues.push(iss); + if (iss.continue !== true) ${id}_ab = true; + } + if (${id}_ab && ctx && ctx.abortEarly) { + payload.value = newResult; + return payload; }`; doc.write(`const input = payload.value;`); const ids = /* @__PURE__ */ Object.create(null); @@ -10527,6 +10662,10 @@ var $ZodObjectJIT = /* @__PURE__ */ $constructor("$ZodObjectJIT", (inst, def) => input: undefined, path: [${k2}] }); + if (ctx && ctx.abortEarly) { + payload.value = newResult; + return payload; + } } if (${id}_present) { @@ -10538,16 +10677,16 @@ var $ZodObjectJIT = /* @__PURE__ */ $constructor("$ZodObjectJIT", (inst, def) => doc.write(` if (${id}.issues.length) {${prefixStr(id, k2)} } - - if (${id}.value === undefined) { - if (${isPresent}) { - newResult[${k2}] = undefined; - } + `); + if (optin === "defaulted") { + doc.write(`newResult[${k2}] = ${id}.value;`); } else { + doc.write(` + if (${id}.value !== undefined || ${isPresent}) { newResult[${k2}] = ${id}.value; } - `); + } } } doc.write(`payload.value = newResult;`); @@ -10579,7 +10718,7 @@ var $ZodObjectJIT = /* @__PURE__ */ $constructor("$ZodObjectJIT", (inst, def) => payload = fastpass(payload, ctx); if (!catchall) return payload; - return handleCatchall([], input2, payload, ctx, value, inst); + return handleCatchall([], input2, payload, ctx, value, inst, ctx?.abortEarly === true); } return superParse(payload, ctx); }; @@ -10711,16 +10850,31 @@ function getDiscriminatedOption(union2, value) { const internals = union2._zod; let map2 = internals.bag.optionsMap; if (!map2) { - map2 = /* @__PURE__ */ new Map(); - const { options, discriminator } = internals.def; - for (const option of options) { - for (const v2 of option._zod.propValues?.[discriminator] ?? []) - if (!map2.has(v2)) - map2.set(v2, option); - } + map2 = discriminatorMap(internals.def); internals.bag.optionsMap = map2; } - return map2.get(value); + const option = map2.get(value); + if (option === null) + throw new Error(`Ambiguous discriminator value "${String(value)}"`); + return option; +} +function discriminatorMap(def) { + const map2 = /* @__PURE__ */ new Map(); + for (const option of def.options) { + const values = option._zod.propValues?.[def.discriminator]; + if (!values || values.size === 0) + throw new Error(`Invalid discriminated union option at index "${def.options.indexOf(option)}"`); + for (const value of values) { + if (map2.has(value)) { + if (value !== void 0) + throw new Error(`Duplicate discriminator value "${String(value)}"`); + map2.set(value, null); + } else { + map2.set(value, option); + } + } + } + return map2; } var $ZodDiscriminatedUnion = /* @__PURE__ */ $constructor("$ZodDiscriminatedUnion", (inst, def) => { def.inclusive = false; @@ -10728,10 +10882,13 @@ var $ZodDiscriminatedUnion = /* @__PURE__ */ $constructor("$ZodDiscriminatedUnio const _super = inst._zod.parse; defineLazyInternal(inst, "propValues", (zod) => { const propValues = {}; + let undefinedCount = 0; for (const option of zod.def.options) { const pv = option._zod.propValues; if (!pv || Object.keys(pv).length === 0) throw new Error(`Invalid discriminated union option at index "${zod.def.options.indexOf(option)}"`); + if (pv[zod.def.discriminator]?.has(void 0)) + undefinedCount++; for (const [k2, v2] of Object.entries(pv)) { if (!Object.prototype.hasOwnProperty.call(propValues, k2)) { assignProp(propValues, k2, /* @__PURE__ */ new Set()); @@ -10741,30 +10898,17 @@ var $ZodDiscriminatedUnion = /* @__PURE__ */ $constructor("$ZodDiscriminatedUnio } } } + if (!zod.def.unionFallback && undefinedCount > 1) + propValues[zod.def.discriminator]?.delete(void 0); return propValues; }); def.options.forEach((option, i) => { - const propShape = propShapes.get(option._zod.def); + const propShape = rawShape(option._zod.def); if (propShape && !Object.prototype.hasOwnProperty.call(propShape, def.discriminator)) { throw new Error(`Invalid discriminated union option at index "${i}"`); } }); - const disc = cached(() => { - const opts = def.options; - const map2 = /* @__PURE__ */ new Map(); - for (const o of opts) { - const values = o._zod.propValues?.[def.discriminator]; - if (!values || values.size === 0) - throw new Error(`Invalid discriminated union option at index "${def.options.indexOf(o)}"`); - for (const v2 of values) { - if (map2.has(v2)) { - throw new Error(`Duplicate discriminator value "${String(v2)}"`); - } - map2.set(v2, o); - } - } - return map2; - }); + const disc = cached(() => discriminatorMap(def)); inst._zod.parse = (payload, ctx) => { const input2 = payload.value; if (!isObject(input2)) { @@ -10776,8 +10920,9 @@ var $ZodDiscriminatedUnion = /* @__PURE__ */ $constructor("$ZodDiscriminatedUnio }); return payload; } - const opt = disc.value.get(input2?.[def.discriminator]); - if (opt) { + const value = input2?.[def.discriminator]; + const opt = disc.value.get(value); + if (opt && (value !== void 0 || ctx.direction !== "backward")) { return opt._zod.run(payload, ctx); } if (def.unionFallback || ctx.direction === "backward") { @@ -10788,7 +10933,7 @@ var $ZodDiscriminatedUnion = /* @__PURE__ */ $constructor("$ZodDiscriminatedUnio errors: [], note: "No matching discriminator", discriminator: def.discriminator, - options: Array.from(disc.value.keys()), + options: Array.from(disc.value.keys()).filter((value2) => disc.value.get(value2) !== null), input: input2, path: [def.discriminator], inst @@ -10864,19 +11009,19 @@ function handleIntersectionResults(result, left, right) { let unrecIssue; const keyIssues = /* @__PURE__ */ new Map(); const collect = (iss, side) => { - let keys; + let keys2; if (iss.code === "unrecognized_keys" && !iss.path?.length) { unrecIssue ?? (unrecIssue = iss); - keys = iss.keys; + keys2 = iss.keys; } else if (iss.code === "invalid_key" && iss.origin === "record" && iss.path?.length === 1) { const k2 = String(iss.path[0]); if (!keyIssues.has(k2)) keyIssues.set(k2, iss); - keys = [k2]; + keys2 = [k2]; } else { return false; } - for (const k2 of keys) { + for (const k2 of keys2) { if (!unrecKeys.has(k2)) unrecKeys.set(k2, {}); unrecKeys.get(k2)[side] = true; @@ -10954,6 +11099,8 @@ var $ZodTuple = /* @__PURE__ */ $constructor("$ZodTuple", (inst, def) => { } } const itemResults = new Array(items.length); + const abortEarly = def.rest ? ctx?.abortEarly : void 0; + let itemAborted = false; for (let i = 0; i < items.length; i++) { const r = items[i]._zod.run({ value: input2[i], issues: [] }, ctx); if (r instanceof Promise) { @@ -10962,12 +11109,20 @@ var $ZodTuple = /* @__PURE__ */ $constructor("$ZodTuple", (inst, def) => { })); } else { itemResults[i] = r; + if (abortEarly && !itemAborted && r.issues.length) + itemAborted = aborted(r); } } - if (def.rest) { + if (def.rest && !itemAborted) { let i = items.length - 1; const rest = input2.slice(items.length); + let seen = payload.issues.length; for (const el of rest) { + if (abortEarly && payload.issues.length !== seen) { + if (aborted(payload, seen)) + break; + seen = payload.issues.length; + } i++; const result = def.rest._zod.run({ value: el, issues: [] }, ctx); if (result instanceof Promise) { @@ -11195,7 +11350,14 @@ var $ZodMap = /* @__PURE__ */ $constructor("$ZodMap", (inst, def) => { } const proms = []; payload.value = memo2 ? memo2.alloc(inst, payload, /* @__PURE__ */ new Map(), ctx) : /* @__PURE__ */ new Map(); + const abortEarly = ctx?.abortEarly; + let seen = payload.issues.length; for (const [key, value] of input2) { + if (abortEarly && payload.issues.length !== seen) { + if (aborted(payload, seen)) + break; + seen = payload.issues.length; + } const keyResult = def.keyType._zod.run({ value: key, issues: [] }, ctx); const valueResult = def.valueType._zod.run({ value, issues: [] }, ctx); if (keyResult instanceof Promise || valueResult instanceof Promise) { @@ -11210,6 +11372,7 @@ var $ZodMap = /* @__PURE__ */ $constructor("$ZodMap", (inst, def) => { return Promise.all(proms).then(() => payload); return payload; }; + capSize(inst, mapSize); }); function handleMapResult(keyResult, valueResult, final, key, input2, inst, ctx) { if (keyResult.issues.length) { @@ -11258,7 +11421,14 @@ var $ZodSet = /* @__PURE__ */ $constructor("$ZodSet", (inst, def) => { } const proms = []; payload.value = memo2 ? memo2.alloc(inst, payload, /* @__PURE__ */ new Set(), ctx) : /* @__PURE__ */ new Set(); + const abortEarly = ctx?.abortEarly; + let seen = payload.issues.length; for (const item of input2) { + if (abortEarly && payload.issues.length !== seen) { + if (aborted(payload, seen)) + break; + seen = payload.issues.length; + } const result = def.valueType._zod.run({ value: item, issues: [] }, ctx); if (result instanceof Promise) { proms.push(result.then((result2) => handleSetResult(result2, payload))); @@ -11269,6 +11439,7 @@ var $ZodSet = /* @__PURE__ */ $constructor("$ZodSet", (inst, def) => { return Promise.all(proms).then(() => payload); return payload; }; + capSize(inst, setSize); }); function handleSetResult(result, final) { if (result.issues.length) { @@ -11281,8 +11452,10 @@ var $ZodEnum = /* @__PURE__ */ $constructor("$ZodEnum", (inst, def) => { const values = getEnumValues(def.entries); const valuesSet = new Set(values); inst._zod.values = valuesSet; - const patternValues = values.filter((k2) => propertyKeyTypes.has(typeof k2)); - inst._zod.pattern = new RegExp(patternValues.length ? `^(${patternValues.map((o) => escapeRegex(o.toString())).join("|")})$` : "^[^\\s\\S]$"); + defineLazyInternal(inst, "pattern", (zod) => { + const patternValues = getEnumValues(zod.def.entries).filter((k2) => propertyKeyTypes.has(typeof k2)); + return new RegExp(patternValues.length ? `^(${patternValues.map((o) => escapeRegex(o.toString())).join("|")})$` : "^[^\\s\\S]$"); + }); inst._zod.parse = (payload, _ctx) => { const input2 = payload.value; if (valuesSet.has(input2)) { @@ -11301,7 +11474,10 @@ var $ZodLiteral = /* @__PURE__ */ $constructor("$ZodLiteral", (inst, def) => { $ZodType.init(inst, def); const values = new Set(def.values); inst._zod.values = values; - inst._zod.pattern = new RegExp(def.values.length ? `^(${def.values.map((o) => typeof o === "string" ? escapeRegex(o) : o ? escapeRegex(o.toString()) : String(o)).join("|")})$` : "^[^\\s\\S]$"); + defineLazyInternal(inst, "pattern", (zod) => { + const vals = zod.def.values; + return new RegExp(vals.length ? `^(${vals.map((o) => typeof o === "string" ? escapeRegex(o) : o ? escapeRegex(o.toString()) : String(o)).join("|")})$` : "^[^\\s\\S]$"); + }); inst._zod.parse = (payload, _ctx) => { const input2 = payload.value; if (values.has(input2)) { @@ -11639,20 +11815,62 @@ function handleReadonlyResult(payload) { payload.value = Object.freeze(payload.value); return payload; } +function leafPattern(schema) { + const def = schema._zod.def; + let pattern = def.pattern; + let isInt = !!def.format?.includes("int"); + let minimum; + let maximum; + for (const ch of def.checks ?? []) { + const d2 = ch._zod.def; + if (d2.pattern) + pattern = d2.pattern; + isInt || (isInt = !!d2.format?.includes("int")); + const lo = d2.minimum ?? d2.length; + const hi = d2.maximum ?? d2.length; + if (lo !== void 0 && (minimum === void 0 || lo > minimum)) + minimum = lo; + if (hi !== void 0 && (maximum === void 0 || hi < maximum)) + maximum = hi; + } + if (pattern) + return pattern.source; + if (minimum !== void 0 && maximum !== void 0 && minimum > maximum) + return "(?!)"; + if (minimum !== void 0 || maximum !== void 0) + return string({ minimum, maximum }).source; + const own2 = schema._zod.pattern; + return (isInt && own2 === number ? integer : own2)?.source; +} +function partPattern(schema) { + const def = schema._zod.def; + const own2 = schema._zod.pattern?.source; + const inner = def.innerType ?? schema._zod.innerType; + if (inner) { + const before = inner._zod.pattern?.source; + const after = partPattern(inner); + if (own2 && before && after && after !== before) { + return own2.replace(cleanRegex(before), () => cleanRegex(after)); + } + return own2; + } + if (def.options) { + const sources = def.options.map(partPattern); + if (sources.every(Boolean)) + return `^(${sources.map((s) => cleanRegex(s)).join("|")})$`; + } + return leafPattern(schema); +} var $ZodTemplateLiteral = /* @__PURE__ */ $constructor("$ZodTemplateLiteral", (inst, def) => { $ZodType.init(inst, def); const regexParts = []; for (const part of def.parts) { if (typeof part === "object" && part !== null) { - if (!part._zod.pattern) { + const source = partPattern(part); + if (!source) { throw new Error(`Invalid template literal part, no pattern found: ${[...part._zod.traits].shift()}`); } - const source = part._zod.pattern instanceof RegExp ? part._zod.pattern.source : part._zod.pattern; - if (!source) - throw new Error(`Invalid template literal part: ${part._zod.traits}`); - const start = source.startsWith("^") ? 1 : 0; - const end = source.endsWith("$") ? source.length - 1 : source.length; - regexParts.push(source.slice(start, end)); + regexParts.push(cleanRegex(source)); } else if (part === null || primitiveTypes.has(typeof part)) { regexParts.push(escapeRegex(`${part}`)); } else { @@ -11827,28 +12045,53 @@ var $ZodCyclicError = class extends Error { }; var STATE = "~memo"; var NO_ISSUES = []; +function isRef(value) { + return value !== null && typeof value === "object"; +} function cloneIssues(issues) { return issues.map((iss) => iss.path ? { ...iss, path: iss.path.slice() } : { ...iss }); } var recursive = /* @__PURE__ */ new WeakMap(); -function isRecursive(inst, stack) { +var NONE = 0; +var ASSUMED = 1; +var PROVEN = 2; +function isRecursive(inst, stack, resolve) { const cached2 = recursive.get(inst); if (cached2 !== void 0) - return cached2; + return cached2 ? PROVEN : NONE; if (stack.has(inst)) - return true; + return PROVEN; stack.add(inst); - let result = false; + let result = NONE; const check2 = (child) => { - if (!result && child?._zod && isRecursive(child, stack)) - result = true; + if (result !== PROVEN && child?._zod) { + const answer = isRecursive(child, stack, resolve); + if (answer > result) + result = answer; + } + }; + const shape = (sh, spread) => { + let answer = NONE; + for (const key of Reflect.ownKeys(sh)) { + const desc = Object.getOwnPropertyDescriptor(sh, key); + if (spread && !desc.enumerable) + continue; + const child = desc.get ? ASSUMED : desc.value?._zod ? isRecursive(desc.value, stack, resolve) : NONE; + if (child > answer) + answer = child; + } + return answer; + }; + const merge2 = (answer) => { + if (answer > result) + result = answer; }; const def = inst._zod.def; const kind = def.type; switch (kind) { case "object": { - for (const key of Reflect.ownKeys(def.shape)) - check2(def.shape[key]); + const raw = rawShape(def); + merge2(raw ? shape(raw, true) : ASSUMED); check2(def.catchall); break; } @@ -11895,10 +12138,12 @@ function isRecursive(inst, stack) { check2(def.input); check2(def.output); break; - // reading `_zod.innerType` resolves the getter once and caches it - case "lazy": - check2(inst._zod.innerType); + // `$ZodLazy` caches its inner on the def, so a resolved edge is followed exactly + case "lazy": { + const inner = def._cachedInner ?? (resolve ? inst._zod.innerType : void 0); + merge2(inner ? isRecursive(inner, stack, false) : ASSUMED); break; + } // a leaf by choice: `parts` are regex fragments, not data positions case "template_literal": // leaves @@ -11940,16 +12185,20 @@ function isRecursive(inst, stack) { } } stack.delete(inst); - recursive.set(inst, result); - return result; + return settle(inst, result); +} +function settle(inst, answer) { + if (answer !== ASSUMED) + recursive.set(inst, answer === PROVEN); + return answer; } function isRecursiveSchema(inst) { - return isRecursive(inst, /* @__PURE__ */ new Set()); + return isRecursive(inst, /* @__PURE__ */ new Set(), true) !== NONE; } function bucketFor(state, inst) { let bucket = state.buckets.get(inst); if (!bucket) { - bucket = /* @__PURE__ */ new Map(); + bucket = /* @__PURE__ */ new WeakMap(); state.buckets.set(inst, bucket); } return bucket; @@ -11985,6 +12234,7 @@ var memo = { attach(inst) { var _a3; let isRecursiveInst; + let rechecked = false; let lastCtx; let lastBucket; (_a3 = inst._zod).deferred ?? (_a3.deferred = []); @@ -11992,20 +12242,24 @@ var memo = { const base = inst._zod.parse; const wrapped = (payload, ctx) => { if (isRecursiveInst === void 0) { - isRecursiveInst = isRecursive(inst, /* @__PURE__ */ new Set()); - if (!isRecursiveInst) { + const walked = isRecursive(inst, /* @__PURE__ */ new Set(), false); + if (walked === NONE) { inst._zod.parse = base; if (inst._zod.run === wrapped) inst._zod.run = base; return base(payload, ctx); } + if (walked === PROVEN || rechecked) + isRecursiveInst = true; + else + rechecked = true; } const input2 = payload.value; - if (input2 === null || typeof input2 !== "object") + if (!isRef(input2)) return base(payload, ctx); let state = ctx[STATE]; if (!state) { - state = { buckets: /* @__PURE__ */ new Map(), backEdges: void 0 }; + state = { buckets: /* @__PURE__ */ new WeakMap(), backEdges: void 0 }; ctx[STATE] = state; } let bucket; @@ -12024,7 +12278,7 @@ var memo = { payload.issues.push(...cloneIssues(hit.issues)); } else { payload.memo = true; - state.backEdges ?? (state.backEdges = /* @__PURE__ */ new Set()); + state.backEdges ?? (state.backEdges = /* @__PURE__ */ new WeakSet()); state.backEdges.add(hit.value); } return payload; @@ -12056,7 +12310,7 @@ function memoizer() { } function isBackEdge(ctx, value) { const backEdges = ctx[STATE]?.backEdges; - return backEdges !== void 0 && value !== null && typeof value === "object" && backEdges.has(value); + return backEdges !== void 0 && isRef(value) && backEdges.has(value); } // node_modules/zod/v4/locales/index.js @@ -12113,6 +12367,7 @@ __export(locales_exports, { sl: () => sl_default, sv: () => sv_default, ta: () => ta_default, + tg: () => tg_default, th: () => th_default, tk: () => tk_default, tr: () => tr_default, @@ -12167,6 +12422,8 @@ var error = () => { json_string: "\u0646\u064E\u0635 \u0639\u0644\u0649 \u0647\u064A\u0626\u0629 JSON", e164: "\u0631\u0642\u0645 \u0647\u0627\u062A\u0641 \u0628\u0645\u0639\u064A\u0627\u0631 E.164", credit_card: "\u0631\u0642\u0645 \u0628\u0637\u0627\u0642\u0629 \u0627\u0644\u0627\u0626\u062A\u0645\u0627\u0646", + currency_code: "\u0631\u0645\u0632 \u0627\u0644\u0639\u0645\u0644\u0629", + iban: "IBAN", jwt: "JWT", template_literal: "\u0645\u062F\u062E\u0644" }; @@ -12277,6 +12534,8 @@ var error2 = () => { json_string: "JSON string", e164: "E.164 number", credit_card: "kredit kart\u0131 n\xF6mr\u0259si", + currency_code: "valyuta kodu", + iban: "IBAN", jwt: "JWT", template_literal: "input" }; @@ -12436,6 +12695,8 @@ var error3 = () => { json_string: "JSON \u0440\u0430\u0434\u043E\u043A", e164: "\u043D\u0443\u043C\u0430\u0440 E.164", credit_card: "\u043D\u0443\u043C\u0430\u0440 \u043A\u0440\u044D\u0434\u044B\u0442\u043D\u0430\u0439 \u043A\u0430\u0440\u0442\u044B", + currency_code: "\u043A\u043E\u0434 \u0432\u0430\u043B\u044E\u0442\u044B", + iban: "IBAN", jwt: "JWT", template_literal: "\u0443\u0432\u043E\u0434" }; @@ -12553,6 +12814,8 @@ var error4 = () => { json_string: "JSON \u043D\u0438\u0437", e164: "E.164 \u043D\u043E\u043C\u0435\u0440", credit_card: "\u043D\u043E\u043C\u0435\u0440 \u043D\u0430 \u043A\u0440\u0435\u0434\u0438\u0442\u043D\u0430 \u043A\u0430\u0440\u0442\u0430", + currency_code: "\u043A\u043E\u0434 \u043D\u0430 \u0432\u0430\u043B\u0443\u0442\u0430", + iban: "IBAN", jwt: "JWT", template_literal: "\u0432\u0445\u043E\u0434" }; @@ -12677,6 +12940,8 @@ var error5 = () => { json_string: "JSON \u09B8\u09CD\u099F\u09CD\u09B0\u09BF\u0982", e164: "E.164 \u09A8\u09AE\u09CD\u09AC\u09B0", credit_card: "\u0995\u09CD\u09B0\u09C7\u09A1\u09BF\u099F \u0995\u09BE\u09B0\u09CD\u09A1 \u09A8\u09AE\u09CD\u09AC\u09B0", + currency_code: "\u09AE\u09C1\u09A6\u09CD\u09B0\u09BE \u0995\u09CB\u09A1", + iban: "IBAN", jwt: "JWT", template_literal: "\u0987\u09A8\u09AA\u09C1\u099F" }; @@ -12789,6 +13054,8 @@ var error6 = () => { json_string: "cadena JSON", e164: "n\xFAmero E.164", credit_card: "n\xFAmero de targeta de cr\xE8dit", + currency_code: "codi de moneda", + iban: "IBAN", jwt: "JWT", template_literal: "entrada" }; @@ -12901,6 +13168,8 @@ var error7 = () => { json_string: "\u062F\u06D5\u0642\u06CC JSON", e164: "\u0698\u0645\u0627\u0631\u06D5\u06CC E.164", credit_card: "\u0698\u0645\u0627\u0631\u06D5\u06CC \u06A9\u0627\u0631\u062A\u06CC \u06A9\u0631\u06CE\u062F\u06CC\u062A", + currency_code: "\u06A9\u06C6\u062F\u06CC \u062F\u0631\u0627\u0648", + iban: "IBAN", jwt: "JWT", template_literal: "\u062A\u06CE\u06A9\u0631\u062F\u06D5" }; @@ -13032,6 +13301,8 @@ var error8 = () => { json_string: "\u0159et\u011Bzec ve form\xE1tu JSON", e164: "\u010D\xEDslo E.164", credit_card: "\u010D\xEDslo kreditn\xED karty", + currency_code: "k\xF3d m\u011Bny", + iban: "IBAN", jwt: "JWT", template_literal: "vstup" }; @@ -13147,6 +13418,8 @@ var error9 = () => { json_string: "JSON-streng", e164: "E.164-nummer", credit_card: "kreditkortnummer", + currency_code: "valutakode", + iban: "IBAN", jwt: "JWT", template_literal: "input" }; @@ -13266,6 +13539,8 @@ var error10 = () => { json_string: "JSON-String", e164: "E.164-Nummer", credit_card: "Kreditkartennummer", + currency_code: "W\xE4hrungscode", + iban: "IBAN", jwt: "JWT", template_literal: "Eingabe" }; @@ -13378,6 +13653,8 @@ var error11 = () => { json_string: "\u03C3\u03C5\u03BC\u03B2\u03BF\u03BB\u03BF\u03C3\u03B5\u03B9\u03C1\u03AC JSON", e164: "\u03B1\u03C1\u03B9\u03B8\u03BC\u03CC\u03C2 E.164", credit_card: "\u03B1\u03C1\u03B9\u03B8\u03BC\u03CC\u03C2 \u03C0\u03B9\u03C3\u03C4\u03C9\u03C4\u03B9\u03BA\u03AE\u03C2 \u03BA\u03AC\u03C1\u03C4\u03B1\u03C2", + currency_code: "\u03BA\u03C9\u03B4\u03B9\u03BA\u03CC\u03C2 \u03BD\u03BF\u03BC\u03AF\u03C3\u03BC\u03B1\u03C4\u03BF\u03C2", + iban: "IBAN", jwt: "JWT", template_literal: "\u03B5\u03AF\u03C3\u03BF\u03B4\u03BF\u03C2" }; @@ -13488,7 +13765,9 @@ var error12 = () => { base64url: "base64url-encoded string", json_string: "JSON string", e164: "E.164 number", + currency_code: "currency code", credit_card: "credit card number", + iban: "IBAN", jwt: "JWT", template_literal: "input" }; @@ -13612,6 +13891,8 @@ var error13 = () => { json_string: "JSON-karaktraro", e164: "E.164-nombro", credit_card: "kreditkarta numero", + currency_code: "valuta kodo", + iban: "IBAN", jwt: "JWT", template_literal: "enigo" }; @@ -13724,6 +14005,8 @@ var error14 = () => { json_string: "cadena JSON", e164: "n\xFAmero E.164", credit_card: "n\xFAmero de tarjeta de cr\xE9dito", + currency_code: "c\xF3digo de moneda", + iban: "IBAN", jwt: "JWT", template_literal: "entrada" }; @@ -13860,6 +14143,8 @@ var error15 = () => { json_string: "JSON \u0631\u0634\u062A\u0647", e164: "E.164 \u0639\u062F\u062F", credit_card: "\u0634\u0645\u0627\u0631\u0647 \u06A9\u0627\u0631\u062A \u0627\u0639\u062A\u0628\u0627\u0631\u06CC", + currency_code: "\u06A9\u062F \u0627\u0631\u0632", + iban: "IBAN", jwt: "JWT", template_literal: "\u0648\u0631\u0648\u062F\u06CC" }; @@ -13982,6 +14267,8 @@ var error16 = () => { json_string: "JSON-merkkijono", e164: "E.164-luku", credit_card: "luottokortin numero", + currency_code: "valuuttakoodi", + iban: "IBAN", jwt: "JWT", template_literal: "templaattimerkkijono" }; @@ -14093,6 +14380,8 @@ var error17 = () => { json_string: "cha\xEEne de caract\xE8res JSON", e164: "num\xE9ro au format E.164", credit_card: "num\xE9ro de carte de cr\xE9dit", + currency_code: "code de devise", + iban: "IBAN", jwt: "JWT", template_literal: "entr\xE9e" }; @@ -14222,6 +14511,8 @@ var error18 = () => { json_string: "cha\xEEne JSON", e164: "num\xE9ro E.164", credit_card: "num\xE9ro de carte de cr\xE9dit", + currency_code: "code de devise", + iban: "IBAN", jwt: "JWT", template_literal: "entr\xE9e" }; @@ -14333,6 +14624,8 @@ var error19 = () => { json_string: "JSON \u0AB8\u0ACD\u0A9F\u0ACD\u0AB0\u0ABF\u0A82\u0A97", e164: "E.164 \u0AA8\u0A82\u0AAC\u0AB0", credit_card: "\u0A95\u0ACD\u0AB0\u0AC7\u0AA1\u0ABF\u0A9F \u0A95\u0ABE\u0AB0\u0ACD\u0AA1 \u0AA8\u0A82\u0AAC\u0AB0", + currency_code: "\u0A9A\u0AB2\u0AA3 \u0A95\u0ACB\u0AA1", + iban: "IBAN", jwt: "JWT", template_literal: "\u0A87\u0AA8\u0AAA\u0AC1\u0A9F" }; @@ -14481,6 +14774,8 @@ var error20 = () => { json_string: { label: "\u05DE\u05D7\u05E8\u05D5\u05D6\u05EA JSON", gender: "f" }, e164: { label: "\u05DE\u05E1\u05E4\u05E8 E.164", gender: "m" }, credit_card: { label: "\u05DE\u05E1\u05E4\u05E8 \u05DB\u05E8\u05D8\u05D9\u05E1 \u05D0\u05E9\u05E8\u05D0\u05D9", gender: "m" }, + currency_code: { label: "\u05E7\u05D5\u05D3 \u05DE\u05D8\u05D1\u05E2", gender: "m" }, + iban: { label: "IBAN", gender: "m" }, jwt: { label: "JWT", gender: "m" }, template_literal: { label: "\u05E7\u05DC\u05D8", gender: "m" }, ends_with: { label: "\u05E7\u05DC\u05D8", gender: "m" }, @@ -14645,6 +14940,8 @@ var error21 = () => { json_string: "JSON \u0938\u094D\u091F\u094D\u0930\u093F\u0902\u0917", e164: "E.164 \u0938\u0902\u0916\u094D\u092F\u093E", credit_card: "\u0915\u094D\u0930\u0947\u0921\u093F\u091F \u0915\u093E\u0930\u094D\u0921 \u0938\u0902\u0916\u094D\u092F\u093E", + currency_code: "\u092E\u0941\u0926\u094D\u0930\u093E \u0915\u094B\u0921", + iban: "IBAN", jwt: "JWT", template_literal: "\u0907\u0928\u092A\u0941\u091F" }; @@ -14754,6 +15051,8 @@ var error22 = () => { json_string: "JSON tekst", e164: "E.164 broj", credit_card: "broj kreditne kartice", + currency_code: "kod valute", + iban: "IBAN", jwt: "JWT", template_literal: "unos" }; @@ -14880,6 +15179,8 @@ var error23 = () => { json_string: "JSON string", e164: "E.164 sz\xE1m", credit_card: "hitelk\xE1rtyasz\xE1m", + currency_code: "p\xE9nznemk\xF3d", + iban: "IBAN", jwt: "JWT", template_literal: "bemenet" }; @@ -15032,6 +15333,8 @@ var error24 = () => { json_string: "JSON \u057F\u0578\u0572", e164: "E.164 \u0570\u0561\u0574\u0561\u0580", credit_card: "\u056F\u0580\u0565\u0564\u056B\u057F \u0584\u0561\u0580\u057F\u056B \u0570\u0561\u0574\u0561\u0580", + currency_code: "\u0561\u0580\u056A\u0578\u0582\u0575\u0569\u056B \u056F\u0578\u0564", + iban: "IBAN", jwt: "JWT", template_literal: "\u0574\u0578\u0582\u057F\u0584" }; @@ -15149,6 +15452,8 @@ var error25 = () => { json_string: "string JSON", e164: "angka E.164", credit_card: "nomor kartu kredit", + currency_code: "kode mata uang", + iban: "IBAN", jwt: "JWT", template_literal: "input" }; @@ -15259,6 +15564,8 @@ var error26 = () => { json_string: "JSON strengur", e164: "E.164 t\xF6lugildi", credit_card: "kreditkortan\xFAmer", + currency_code: "gjaldmi\xF0ilsk\xF3\xF0i", + iban: "IBAN", jwt: "JWT", template_literal: "gildi" }; @@ -15372,6 +15679,8 @@ var error27 = () => { json_string: "stringa JSON", e164: "numero E.164", credit_card: "numero di carta di credito", + currency_code: "codice valuta", + iban: "IBAN", jwt: "JWT", template_literal: "input" }; @@ -15484,6 +15793,8 @@ var error28 = () => { json_string: "JSON\u6587\u5B57\u5217", e164: "E.164\u756A\u53F7", credit_card: "\u30AF\u30EC\u30B8\u30C3\u30C8\u30AB\u30FC\u30C9\u756A\u53F7", + currency_code: "\u901A\u8CA8\u30B3\u30FC\u30C9", + iban: "IBAN", jwt: "JWT", template_literal: "\u5165\u529B\u5024" }; @@ -15595,6 +15906,8 @@ var error29 = () => { json_string: "JSON \u10D5\u10D4\u10DA\u10D8", e164: "E.164 \u10DC\u10DD\u10DB\u10D4\u10E0\u10D8", credit_card: "\u10E1\u10D0\u10D9\u10E0\u10D4\u10D3\u10D8\u10E2\u10DD \u10D1\u10D0\u10E0\u10D0\u10D7\u10D8\u10E1 \u10DC\u10DD\u10DB\u10D4\u10E0\u10D8", + currency_code: "\u10D5\u10D0\u10DA\u10E3\u10E2\u10D8\u10E1 \u10D9\u10DD\u10D3\u10D8", + iban: "IBAN", jwt: "JWT", template_literal: "\u10E8\u10D4\u10E7\u10D5\u10D0\u10DC\u10D0" }; @@ -15711,6 +16024,8 @@ var error30 = () => { json_string: "\u1781\u17D2\u179F\u17C2\u17A2\u1780\u17D2\u179F\u179A JSON", e164: "\u179B\u17C1\u1781 E.164", credit_card: "\u179B\u17C1\u1781\u1794\u17D0\u178E\u17D2\u178E\u17A5\u178E\u1791\u17B6\u1793", + currency_code: "\u1780\u17BC\u178A\u179A\u17BC\u1794\u17B7\u1799\u1794\u17D0\u178E\u17D2\u178E", + iban: "IBAN", jwt: "JWT", template_literal: "\u1791\u17B7\u1793\u17D2\u1793\u1793\u17D0\u1799\u1794\u1789\u17D2\u1785\u17BC\u179B" }; @@ -15830,6 +16145,8 @@ var error31 = () => { json_string: "JSON\u0CB8\u0CCD\u0C9F\u0CCD\u0CB0\u0CBF\u0C82\u0C97\u0CCD", e164: "E.164 \u0CB8\u0C82\u0C96\u0CCD\u0CAF\u0CC6", credit_card: "\u0C95\u0CCD\u0CB0\u0CC6\u0CA1\u0CBF\u0C9F\u0CCD \u0C95\u0CBE\u0CB0\u0CCD\u0CA1\u0CCD \u0CB8\u0C82\u0C96\u0CCD\u0CAF\u0CC6", + currency_code: "\u0C95\u0CB0\u0CC6\u0CA8\u0CCD\u0CB8\u0CBF \u0C95\u0CCB\u0CA1\u0CCD", + iban: "IBAN", jwt: "JWT", template_literal: "\u0C87\u0CA8\u0CCD\u0CAA\u0CC1\u0C9F\u0CCD" }; @@ -15944,6 +16261,8 @@ var error32 = () => { json_string: "JSON \uBB38\uC790\uC5F4", e164: "E.164 \uBC88\uD638", credit_card: "\uC2E0\uC6A9\uCE74\uB4DC \uBC88\uD638", + currency_code: "\uD1B5\uD654 \uCF54\uB4DC", + iban: "IBAN", jwt: "JWT", template_literal: "\uC785\uB825" }; @@ -16141,6 +16460,8 @@ var error33 = () => { json_string: "JSON eilut\u0117", e164: "E.164 numeris", credit_card: "kredito kortel\u0117s numeris", + currency_code: "valiutos kodas", + iban: "IBAN", jwt: "JWT", template_literal: "\u012Fvestis" }; @@ -16265,6 +16586,8 @@ var error34 = () => { json_string: "JSON \u043D\u0438\u0437\u0430", e164: "E.164 \u0431\u0440\u043E\u0458", credit_card: "\u0431\u0440\u043E\u0458 \u043D\u0430 \u043A\u0440\u0435\u0434\u0438\u0442\u043D\u0430 \u043A\u0430\u0440\u0442\u0438\u0447\u043A\u0430", + currency_code: "\u043A\u043E\u0434 \u043D\u0430 \u0432\u0430\u043B\u0443\u0442\u0430", + iban: "IBAN", jwt: "JWT", template_literal: "\u0432\u043D\u0435\u0441" }; @@ -16378,6 +16701,8 @@ var error35 = () => { json_string: "string JSON", e164: "nombor E.164", credit_card: "nombor kad kredit", + currency_code: "kod mata wang", + iban: "IBAN", jwt: "JWT", template_literal: "input" }; @@ -16489,6 +16814,8 @@ var error36 = () => { json_string: "JSON \u0938\u094D\u091F\u094D\u0930\u093F\u0919", e164: "E.164 \u0928\u092E\u094D\u092C\u0930", credit_card: "\u0915\u094D\u0930\u0947\u0921\u093F\u091F \u0915\u093E\u0930\u094D\u0921 \u0928\u092E\u094D\u092C\u0930", + currency_code: "\u092E\u0941\u0926\u094D\u0930\u093E \u0915\u094B\u0921", + iban: "IBAN", jwt: "JWT", template_literal: "\u0907\u0928\u092A\u0941\u091F" }; @@ -16599,6 +16926,8 @@ var error37 = () => { json_string: "JSON string", e164: "E.164-nummer", credit_card: "creditcardnummer", + currency_code: "valutacode", + iban: "IBAN", jwt: "JWT", template_literal: "invoer" }; @@ -16713,6 +17042,8 @@ var error38 = () => { json_string: "JSON-streng", e164: "E.164-nummer", credit_card: "kredittkortnummer", + currency_code: "valutakode", + iban: "IBAN", jwt: "JWT", template_literal: "input" }; @@ -16825,6 +17156,8 @@ var error39 = () => { json_string: "JSON-streng", e164: "E.164-nummer", credit_card: "kredittkortnummer", + currency_code: "valutakode", + iban: "IBAN", jwt: "JWT", template_literal: "input" }; @@ -16937,6 +17270,8 @@ var error40 = () => { json_string: "JSON metin", e164: "E.164 say\u0131s\u0131", credit_card: "i'tib\xE2r kart\u0131 numaras\u0131", + currency_code: "para birimi kodu", + iban: "IBAN", jwt: "JWT", template_literal: "giren" }; @@ -17050,6 +17385,8 @@ var error41 = () => { json_string: "JSON \u0645\u062A\u0646", e164: "\u062F E.164 \u0634\u0645\u06D0\u0631\u0647", credit_card: "\u062F \u06A9\u0631\u06CC\u0689\u06CC\u067C \u06A9\u0627\u0631\u062A \u0634\u0645\u06CC\u0631\u0647", + currency_code: "\u062F \u0627\u0633\u0639\u0627\u0631\u0648 \u06A9\u0648\u0689", + iban: "IBAN", jwt: "JWT", template_literal: "\u0648\u0631\u0648\u062F\u064A" }; @@ -17168,6 +17505,8 @@ var error42 = () => { json_string: "ci\u0105g znak\xF3w w formacie JSON", e164: "liczba E.164", credit_card: "numer karty kredytowej", + currency_code: "kod waluty", + iban: "IBAN", jwt: "JWT", template_literal: "wej\u015Bcie" }; @@ -17281,6 +17620,8 @@ var error43 = () => { json_string: "o texto JSON", e164: "o n\xFAmero E.164", credit_card: "o n\xFAmero de cart\xE3o de cr\xE9dito", + currency_code: "o c\xF3digo de moeda", + iban: "o IBAN", jwt: "o JWT", template_literal: "a entrada" }; @@ -17358,8 +17699,8 @@ var error43 = () => { case "not_multiple_of": return `N\xFAmero inv\xE1lido: deve ser m\xFAltiplo de ${issue2.divisor}`; case "unrecognized_keys": { - const plural = issue2.keys.length > 1 ? "s" : ""; - return `Chave${plural} inv\xE1lida${plural}: ${joinValues(issue2.keys, ", ")}`; + const plural2 = issue2.keys.length > 1 ? "s" : ""; + return `Chave${plural2} inv\xE1lida${plural2}: ${joinValues(issue2.keys, ", ")}`; } case "invalid_key": return `Entrada inv\xE1lida n${translateOriginWithArticle(issue2.origin, "definite")}`; @@ -17423,6 +17764,8 @@ var error44 = () => { json_string: "o texto JSON", e164: "o n\xFAmero E.164", credit_card: "o n\xFAmero de cart\xE3o de cr\xE9dito", + currency_code: "o c\xF3digo de moeda", + iban: "o IBAN", jwt: "o JWT", template_literal: "a entrada" }; @@ -17501,8 +17844,8 @@ var error44 = () => { case "not_multiple_of": return `N\xFAmero inv\xE1lido: deve ser m\xFAltiplo de ${issue2.divisor}`; case "unrecognized_keys": { - const plural = issue2.keys.length > 1 ? "s" : ""; - return `Chave${plural} inv\xE1lida${plural}: ${joinValues(issue2.keys, ", ")}`; + const plural2 = issue2.keys.length > 1 ? "s" : ""; + return `Chave${plural2} inv\xE1lida${plural2}: ${joinValues(issue2.keys, ", ")}`; } case "invalid_key": return `Entrada inv\xE1lida n${translateOriginWithArticle(issue2.origin, "definite")}`; @@ -17566,6 +17909,8 @@ var error45 = () => { json_string: "\u0219ir JSON", e164: "num\u0103r E.164", credit_card: "num\u0103r de card de credit", + currency_code: "cod valutar", + iban: "IBAN", jwt: "JWT", template_literal: "intrare" }; @@ -17737,6 +18082,8 @@ var error46 = () => { json_string: "JSON \u0441\u0442\u0440\u043E\u043A\u0430", e164: "\u043D\u043E\u043C\u0435\u0440 E.164", credit_card: "\u043D\u043E\u043C\u0435\u0440 \u043A\u0440\u0435\u0434\u0438\u0442\u043D\u043E\u0439 \u043A\u0430\u0440\u0442\u044B", + currency_code: "\u043A\u043E\u0434 \u0432\u0430\u043B\u044E\u0442\u044B", + iban: "IBAN", jwt: "JWT", template_literal: "\u0432\u0432\u043E\u0434" }; @@ -17854,6 +18201,8 @@ var error47 = () => { json_string: "re\u0165azec vo form\xE1te JSON", e164: "\u010D\xEDslo E.164", credit_card: "\u010D\xEDslo kreditnej karty", + currency_code: "k\xF3d meny", + iban: "IBAN", jwt: "JWT", template_literal: "vstup" }; @@ -17969,6 +18318,8 @@ var error48 = () => { json_string: "JSON niz", e164: "E.164 \u0161tevilka", credit_card: "\u0161tevilka kreditne kartice", + currency_code: "koda valute", + iban: "IBAN", jwt: "JWT", template_literal: "vnos" }; @@ -18082,6 +18433,8 @@ var error49 = () => { json_string: "JSON-str\xE4ng", e164: "E.164-nummer", credit_card: "kreditkortsnummer", + currency_code: "valutakod", + iban: "IBAN", jwt: "JWT", template_literal: "mall-literal" }; @@ -18196,6 +18549,8 @@ var error50 = () => { json_string: "JSON \u0B9A\u0BB0\u0BAE\u0BCD", e164: "E.164 \u0B8E\u0BA3\u0BCD", credit_card: "\u0B95\u0B9F\u0BA9\u0BCD \u0B85\u0B9F\u0BCD\u0B9F\u0BC8 \u0B8E\u0BA3\u0BCD", + currency_code: "\u0BA8\u0BBE\u0BA3\u0BAF\u0B95\u0BCD \u0B95\u0BC1\u0BB1\u0BBF\u0BAF\u0BC0\u0B9F\u0BC1", + iban: "IBAN", jwt: "JWT", template_literal: "input" }; @@ -18269,8 +18624,125 @@ function ta_default() { }; } -// node_modules/zod/v4/locales/th.js +// node_modules/zod/v4/locales/tg.js var error51 = () => { + const Sizable = { + string: { unit: "\u0430\u043B\u043E\u043C\u0430\u0442", verb: "\u0434\u043E\u0448\u0442\u0430 \u0431\u043E\u0448\u0430\u0434" }, + file: { unit: "\u0431\u0430\u0439\u0442", verb: "\u0434\u043E\u0448\u0442\u0430 \u0431\u043E\u0448\u0430\u0434" }, + array: { unit: "\u0443\u043D\u0441\u0443\u0440", verb: "\u0434\u043E\u0448\u0442\u0430 \u0431\u043E\u0448\u0430\u0434" }, + set: { unit: "\u0443\u043D\u0441\u0443\u0440", verb: "\u0434\u043E\u0448\u0442\u0430 \u0431\u043E\u0448\u0430\u0434" }, + map: { unit: "\u0441\u0430\u0431\u0442", verb: "\u0434\u043E\u0448\u0442\u0430 \u0431\u043E\u0448\u0430\u0434" } + }; + function getSizing(origin) { + return Sizable[origin] ?? null; + } + const FormatDictionary = { + regex: "\u0432\u0443\u0440\u0443\u0434", + email: "\u0441\u0443\u0440\u043E\u0493\u0430\u0438 email", + url: "URL", + emoji: "\u044D\u043C\u043E\u04B7\u0438", + uuid: "UUID", + uuidv4: "UUIDv4", + uuidv6: "UUIDv6", + nanoid: "nanoid", + guid: "GUID", + cuid: "cuid", + cuid2: "cuid2", + ulid: "ULID", + xid: "XID", + ksuid: "KSUID", + datetime: "\u0441\u0430\u043D\u0430\u0432\u0443 \u0432\u0430\u049B\u0442\u0438 ISO", + date: "\u0441\u0430\u043D\u0430\u0438 ISO", + time: "\u0432\u0430\u049B\u0442\u0438 ISO", + duration: "\u0434\u0430\u0432\u043E\u043C\u043D\u043E\u043A\u0438\u0438 ISO", + ipv4: "\u0441\u0443\u0440\u043E\u0493\u0430\u0438 IPv4", + ipv6: "\u0441\u0443\u0440\u043E\u0493\u0430\u0438 IPv6", + mac: "\u0441\u0443\u0440\u043E\u0493\u0430\u0438 MAC", + cidrv4: "\u043C\u0430\u04B3\u0434\u0443\u0434\u0430\u0438 IPv4", + cidrv6: "\u043C\u0430\u04B3\u0434\u0443\u0434\u0430\u0438 IPv6", + base64: "\u0441\u0430\u0442\u0440\u0438 \u0434\u0430\u0440 \u0444\u043E\u0440\u043C\u0430\u0442\u0438 base64", + base64url: "\u0441\u0430\u0442\u0440\u0438 \u0434\u0430\u0440 \u0444\u043E\u0440\u043C\u0430\u0442\u0438 base64url", + json_string: "\u0441\u0430\u0442\u0440\u0438 JSON", + e164: "\u0440\u0430\u049B\u0430\u043C\u0438 E.164", + credit_card: "\u0440\u0430\u049B\u0430\u043C\u0438 \u043A\u043E\u0440\u0442\u0438 \u043A\u0440\u0435\u0434\u0438\u0442\u04E3", + currency_code: "\u0440\u0430\u043C\u0437\u0438 \u0430\u0441\u044A\u043E\u0440", + iban: "IBAN", + jwt: "JWT", + template_literal: "\u0432\u0443\u0440\u0443\u0434" + }; + const TypeDictionary = { + nan: "NaN", + number: "\u0440\u0430\u049B\u0430\u043C", + string: "\u0441\u0430\u0442\u0440", + array: "\u043C\u0430\u0441\u0441\u0438\u0432", + object: "\u043E\u0431\u044A\u0435\u043A\u0442", + date: "\u0441\u0430\u043D\u0430" + }; + return (issue2) => { + switch (issue2.code) { + case "invalid_type": { + const expected = TypeDictionary[issue2.expected] ?? issue2.expected; + const receivedType = parsedType(issue2.input); + const received = TypeDictionary[receivedType] ?? receivedType; + return `\u0412\u0443\u0440\u0443\u0434\u0438 \u043D\u043E\u0434\u0443\u0440\u0443\u0441\u0442: ${expected} \u0438\u043D\u0442\u0438\u0437\u043E\u0440 \u043C\u0435\u0440\u0430\u0444\u0442, ${received} \u0433\u0438\u0440\u0438\u0444\u0442\u0430 \u0448\u0443\u0434`; + } + case "invalid_value": + if (issue2.values.length === 1) + return `\u0412\u0443\u0440\u0443\u0434\u0438 \u043D\u043E\u0434\u0443\u0440\u0443\u0441\u0442: ${stringifyPrimitive(issue2.values[0])} \u0438\u043D\u0442\u0438\u0437\u043E\u0440 \u043C\u0435\u0440\u0430\u0444\u0442`; + return `\u0418\u043D\u0442\u0438\u0445\u043E\u0431\u0438 \u043D\u043E\u0434\u0443\u0440\u0443\u0441\u0442: \u044F\u043A\u0435 \u0430\u0437 ${joinValues(issue2.values, "|")} \u0438\u043D\u0442\u0438\u0437\u043E\u0440 \u043C\u0435\u0440\u0430\u0444\u0442`; + case "too_big": { + const adj = issue2.inclusive ? "<=" : "<"; + const sizing = getSizing(issue2.origin); + if (sizing) + return `\u0425\u0435\u043B\u0435 \u043A\u0430\u043B\u043E\u043D: ${issue2.origin ?? "\u049B\u0438\u043C\u0430\u0442"} \u0431\u043E\u044F\u0434 ${adj}${issue2.maximum.toString()} ${sizing.unit} ${sizing.verb}`; + return `\u0425\u0435\u043B\u0435 \u043A\u0430\u043B\u043E\u043D: ${issue2.origin ?? "\u049B\u0438\u043C\u0430\u0442"} \u0431\u043E\u044F\u0434 ${adj}${issue2.maximum.toString()} \u0431\u043E\u0448\u0430\u0434`; + } + case "too_small": { + const adj = issue2.inclusive ? ">=" : ">"; + const sizing = getSizing(issue2.origin); + if (sizing) + return `\u0425\u0435\u043B\u0435 \u0445\u0443\u0440\u0434: ${issue2.origin} \u0431\u043E\u044F\u0434 ${adj}${issue2.minimum.toString()} ${sizing.unit} ${sizing.verb}`; + return `\u0425\u0435\u043B\u0435 \u0445\u0443\u0440\u0434: ${issue2.origin} \u0431\u043E\u044F\u0434 ${adj}${issue2.minimum.toString()} \u0431\u043E\u0448\u0430\u0434`; + } + case "invalid_format": { + const _issue = issue2; + if (_issue.format === "starts_with") + return `\u0421\u0430\u0442\u0440\u0438 \u043D\u043E\u0434\u0443\u0440\u0443\u0441\u0442: \u0431\u043E\u044F\u0434 \u0431\u043E "${_issue.prefix}" \u043E\u0493\u043E\u0437 \u0448\u0430\u0432\u0430\u0434`; + if (_issue.format === "ends_with") + return `\u0421\u0430\u0442\u0440\u0438 \u043D\u043E\u0434\u0443\u0440\u0443\u0441\u0442: \u0431\u043E\u044F\u0434 \u0431\u043E "${_issue.suffix}" \u0430\u043D\u04B7\u043E\u043C \u0451\u0431\u0430\u0434`; + if (_issue.format === "includes") + return `\u0421\u0430\u0442\u0440\u0438 \u043D\u043E\u0434\u0443\u0440\u0443\u0441\u0442: \u0431\u043E\u044F\u0434 "${_issue.includes}"-\u0440\u043E \u0434\u0430\u0440 \u0431\u0430\u0440 \u0433\u0438\u0440\u0430\u0434`; + if (_issue.format === "regex") + return `\u0421\u0430\u0442\u0440\u0438 \u043D\u043E\u0434\u0443\u0440\u0443\u0441\u0442: \u0431\u043E\u044F\u0434 \u0431\u0430 \u043D\u0430\u043C\u0443\u043D\u0430\u0438 ${_issue.pattern} \u043C\u0443\u0432\u043E\u0444\u0438\u049B\u0430\u0442 \u043A\u0443\u043D\u0430\u0434`; + return `${FormatDictionary[_issue.format] ?? issue2.format}-\u0438 \u043D\u043E\u0434\u0443\u0440\u0443\u0441\u0442`; + } + case "not_multiple_of": + return `\u0420\u0430\u049B\u0430\u043C\u0438 \u043D\u043E\u0434\u0443\u0440\u0443\u0441\u0442: \u0431\u043E\u044F\u0434 \u0431\u0430 ${issue2.divisor} \u0431\u0435 \u0431\u0430\u049B\u0438\u044F \u0442\u0430\u049B\u0441\u0438\u043C \u0448\u0430\u0432\u0430\u0434`; + case "unrecognized_keys": + return `\u041A\u0430\u043B\u0438\u0434${issue2.keys.length > 1 ? "\u04B3\u043E\u0438" : "\u0438"} \u043D\u043E\u043C\u0430\u044A\u043B\u0443\u043C: ${joinValues(issue2.keys, ", ")}`; + case "invalid_key": + return `\u041A\u0430\u043B\u0438\u0434\u0438 \u043D\u043E\u0434\u0443\u0440\u0443\u0441\u0442 \u0434\u0430\u0440 ${issue2.origin}`; + case "invalid_union": + if (issue2.options && Array.isArray(issue2.options) && issue2.options.length > 0) { + const opts = issue2.options.map((o) => `'${o}'`).join(" | "); + return `\u049A\u0438\u043C\u0430\u0442\u0438 \u043D\u043E\u0434\u0443\u0440\u0443\u0441\u0442\u0438 \u0434\u0438\u0441\u043A\u0440\u0438\u043C\u0438\u043D\u0430\u0442\u043E\u0440: ${opts} \u0438\u043D\u0442\u0438\u0437\u043E\u0440 \u043C\u0435\u0440\u0430\u0444\u0442`; + } + return "\u0412\u0443\u0440\u0443\u0434\u0438 \u043D\u043E\u0434\u0443\u0440\u0443\u0441\u0442"; + case "invalid_element": + return `\u049A\u0438\u043C\u0430\u0442\u0438 \u043D\u043E\u0434\u0443\u0440\u0443\u0441\u0442 \u0434\u0430\u0440 ${issue2.origin}`; + default: + return `\u0412\u0443\u0440\u0443\u0434\u0438 \u043D\u043E\u0434\u0443\u0440\u0443\u0441\u0442`; + } + }; +}; +function tg_default() { + return { + localeError: error51() + }; +} + +// node_modules/zod/v4/locales/th.js +var error52 = () => { const Sizable = { string: { unit: "\u0E15\u0E31\u0E27\u0E2D\u0E31\u0E01\u0E29\u0E23", verb: "\u0E04\u0E27\u0E23\u0E21\u0E35" }, file: { unit: "\u0E44\u0E1A\u0E15\u0E4C", verb: "\u0E04\u0E27\u0E23\u0E21\u0E35" }, @@ -18310,6 +18782,8 @@ var error51 = () => { json_string: "\u0E02\u0E49\u0E2D\u0E04\u0E27\u0E32\u0E21\u0E41\u0E1A\u0E1A JSON", e164: "\u0E40\u0E1A\u0E2D\u0E23\u0E4C\u0E42\u0E17\u0E23\u0E28\u0E31\u0E1E\u0E17\u0E4C\u0E23\u0E30\u0E2B\u0E27\u0E48\u0E32\u0E07\u0E1B\u0E23\u0E30\u0E40\u0E17\u0E28 (E.164)", credit_card: "\u0E2B\u0E21\u0E32\u0E22\u0E40\u0E25\u0E02\u0E1A\u0E31\u0E15\u0E23\u0E40\u0E04\u0E23\u0E14\u0E34\u0E15", + currency_code: "\u0E23\u0E2B\u0E31\u0E2A\u0E2A\u0E01\u0E38\u0E25\u0E40\u0E07\u0E34\u0E19", + iban: "IBAN", jwt: "\u0E42\u0E17\u0E40\u0E04\u0E19 JWT", template_literal: "\u0E02\u0E49\u0E2D\u0E21\u0E39\u0E25\u0E17\u0E35\u0E48\u0E1B\u0E49\u0E2D\u0E19" }; @@ -18379,12 +18853,12 @@ var error51 = () => { }; function th_default() { return { - localeError: error51() + localeError: error52() }; } // node_modules/zod/v4/locales/tk.js -var error52 = () => { +var error53 = () => { const Sizable = { string: { unit: "simwol", verb: "bolmaly" }, file: { unit: "ba\xFDt", verb: "bolmaly" }, @@ -18424,6 +18898,8 @@ var error52 = () => { json_string: "JSON setiri", e164: "E.164 nomeri", credit_card: "kredit kartyny\u0148 nomeri", + currency_code: "wal\xFDuta kody", + iban: "IBAN", jwt: "JWT", template_literal: "\u015Fablon" }; @@ -18485,12 +18961,12 @@ var error52 = () => { }; function tk_default() { return { - localeError: error52() + localeError: error53() }; } // node_modules/zod/v4/locales/tr.js -var error53 = () => { +var error54 = () => { const Sizable = { string: { unit: "karakter", verb: "olmal\u0131" }, file: { unit: "bayt", verb: "olmal\u0131" }, @@ -18530,6 +19006,8 @@ var error53 = () => { json_string: "JSON dizesi", e164: "E.164 say\u0131s\u0131", credit_card: "kredi kart\u0131 numaras\u0131", + currency_code: "para birimi kodu", + iban: "IBAN", jwt: "JWT", template_literal: "\u015Eablon dizesi" }; @@ -18594,12 +19072,12 @@ var error53 = () => { }; function tr_default() { return { - localeError: error53() + localeError: error54() }; } // node_modules/zod/v4/locales/uk.js -var error54 = () => { +var error55 = () => { const Sizable = { string: { unit: "\u0441\u0438\u043C\u0432\u043E\u043B\u0456\u0432", verb: "\u043C\u0430\u0442\u0438\u043C\u0435" }, file: { unit: "\u0431\u0430\u0439\u0442\u0456\u0432", verb: "\u043C\u0430\u0442\u0438\u043C\u0435" }, @@ -18639,6 +19117,8 @@ var error54 = () => { json_string: "\u0440\u044F\u0434\u043E\u043A JSON", e164: "\u043D\u043E\u043C\u0435\u0440 E.164", credit_card: "\u043D\u043E\u043C\u0435\u0440 \u043A\u0440\u0435\u0434\u0438\u0442\u043D\u043E\u0457 \u043A\u0430\u0440\u0442\u043A\u0438", + currency_code: "\u043A\u043E\u0434 \u0432\u0430\u043B\u044E\u0442\u0438", + iban: "IBAN", jwt: "JWT", template_literal: "\u0432\u0445\u0456\u0434\u043D\u0456 \u0434\u0430\u043D\u0456" }; @@ -18706,7 +19186,7 @@ var error54 = () => { }; function uk_default() { return { - localeError: error54() + localeError: error55() }; } @@ -18716,7 +19196,7 @@ function ua_default() { } // node_modules/zod/v4/locales/ur.js -var error55 = () => { +var error56 = () => { const Sizable = { string: { unit: "\u062D\u0631\u0648\u0641", verb: "\u06C1\u0648\u0646\u0627" }, file: { unit: "\u0628\u0627\u0626\u0679\u0633", verb: "\u06C1\u0648\u0646\u0627" }, @@ -18756,6 +19236,8 @@ var error55 = () => { json_string: "\u062C\u06D2 \u0627\u06CC\u0633 \u0627\u0648 \u0627\u06CC\u0646 \u0633\u0679\u0631\u0646\u06AF", e164: "\u0627\u06CC 164 \u0646\u0645\u0628\u0631", credit_card: "\u06A9\u0631\u06CC\u0688\u0679 \u06A9\u0627\u0631\u0688 \u0646\u0645\u0628\u0631", + currency_code: "\u06A9\u0631\u0646\u0633\u06CC \u06A9\u0648\u0688", + iban: "IBAN", jwt: "\u062C\u06D2 \u0688\u0628\u0644\u06CC\u0648 \u0679\u06CC", template_literal: "\u0627\u0646 \u067E\u0679" }; @@ -18825,12 +19307,12 @@ var error55 = () => { }; function ur_default() { return { - localeError: error55() + localeError: error56() }; } // node_modules/zod/v4/locales/uz.js -var error56 = () => { +var error57 = () => { const Sizable = { string: { unit: "belgi", verb: "bo\u2018lishi kerak" }, file: { unit: "bayt", verb: "bo\u2018lishi kerak" }, @@ -18870,6 +19352,8 @@ var error56 = () => { json_string: "JSON satr", e164: "E.164 raqam", credit_card: "kredit karta raqami", + currency_code: "valyuta kodi", + iban: "IBAN", jwt: "JWT", template_literal: "kirish" }; @@ -18937,12 +19421,12 @@ var error56 = () => { }; function uz_default() { return { - localeError: error56() + localeError: error57() }; } // node_modules/zod/v4/locales/vi.js -var error57 = () => { +var error58 = () => { const Sizable = { string: { unit: "k\xFD t\u1EF1", verb: "c\xF3" }, file: { unit: "byte", verb: "c\xF3" }, @@ -18982,6 +19466,8 @@ var error57 = () => { json_string: "chu\u1ED7i JSON", e164: "s\u1ED1 E.164", credit_card: "s\u1ED1 th\u1EBB t\xEDn d\u1EE5ng", + currency_code: "m\xE3 ti\u1EC1n t\u1EC7", + iban: "IBAN", jwt: "JWT", template_literal: "\u0111\u1EA7u v\xE0o" }; @@ -19049,12 +19535,12 @@ var error57 = () => { }; function vi_default() { return { - localeError: error57() + localeError: error58() }; } // node_modules/zod/v4/locales/zh-CN.js -var error58 = () => { +var error59 = () => { const Sizable = { string: { unit: "\u5B57\u7B26", verb: "\u5305\u542B" }, file: { unit: "\u5B57\u8282", verb: "\u5305\u542B" }, @@ -19094,6 +19580,8 @@ var error58 = () => { json_string: "JSON\u5B57\u7B26\u4E32", e164: "E.164\u53F7\u7801", credit_card: "\u4FE1\u7528\u5361\u53F7", + currency_code: "\u8D27\u5E01\u4EE3\u7801", + iban: "IBAN", jwt: "JWT", template_literal: "\u8F93\u5165" }; @@ -19162,12 +19650,12 @@ var error58 = () => { }; function zh_CN_default() { return { - localeError: error58() + localeError: error59() }; } // node_modules/zod/v4/locales/zh-TW.js -var error59 = () => { +var error60 = () => { const Sizable = { string: { unit: "\u5B57\u5143", verb: "\u64C1\u6709" }, file: { unit: "\u4F4D\u5143\u7D44", verb: "\u64C1\u6709" }, @@ -19207,6 +19695,8 @@ var error59 = () => { json_string: "JSON \u5B57\u4E32", e164: "E.164 \u6578\u503C", credit_card: "\u4FE1\u7528\u5361\u865F", + currency_code: "\u8CA8\u5E63\u4EE3\u78BC", + iban: "IBAN", jwt: "JWT", template_literal: "\u8F38\u5165" }; @@ -19273,12 +19763,12 @@ var error59 = () => { }; function zh_TW_default() { return { - localeError: error59() + localeError: error60() }; } // node_modules/zod/v4/locales/yo.js -var error60 = () => { +var error61 = () => { const Sizable = { string: { unit: "\xE0mi", verb: "n\xED" }, file: { unit: "bytes", verb: "n\xED" }, @@ -19318,6 +19808,8 @@ var error60 = () => { json_string: "\u1ECD\u0300r\u1ECD\u0300 JSON", e164: "n\u1ECD\u0301mb\xE0 E.164", credit_card: "n\u1ECDmba kaadi gbese", + currency_code: "koodu ow\xF3", + iban: "IBAN", jwt: "JWT", template_literal: "\u1EB9\u0300r\u1ECD \xECb\xE1w\u1ECDl\xE9" }; @@ -19384,7 +19876,7 @@ var error60 = () => { }; function yo_default() { return { - localeError: error60() + localeError: error61() }; } @@ -19464,31 +19956,8 @@ function compileValidator(schema, parser) { function compile(schema, options) { try { const parser = compileFn(schema); - const clone2 = clone(schema); - const liveRun = schema._zod.run; - const originalRun = liveRun.__originalRun ?? liveRun; - const wrapped = (payload, ctx) => { - if (ctx?.async || ctx?.direction === "backward" || ctx?.skipChecks || ctx?.[FALLBACK_FLAG]) { - return originalRun(payload, ctx); - } - if (ctx && isBackEdge(ctx, payload.value)) { - return originalRun(payload, ctx); - } - const out = parser(payload.value); - if (out !== INVALID) { - payload.value = out; - return payload; - } - if (ctx) - ctx[FALLBACK_FLAG] = true; - return originalRun(payload, ctx); - }; - wrapped.__originalRun = originalRun; - clone2._zod.bag.fallbackRun = originalRun; + const clone2 = withParser(schema, parser); clone2._zod.bag.validator = compileValidator(schema, parser); - clone2._zod.run = wrapped; - if (!liveRun.__originalRun) - installCompiledUserMethods(clone2, schema, parser); return clone2; } catch (err) { if (options?.strict) @@ -19496,6 +19965,37 @@ function compile(schema, options) { return schema; } } +function withParser(schema, parser) { + if (isRecursiveSchema(schema)) { + throw new ZodCompileUnsupportedError("a schema whose subtree contains a reference cycle"); + } + const clone2 = clone(schema); + const liveRun = schema._zod.run; + const originalRun = liveRun.__originalRun ?? liveRun; + const wrapped = (payload, ctx) => { + if (ctx?.async || ctx?.direction === "backward" || ctx?.skipChecks || ctx?.[FALLBACK_FLAG]) { + return originalRun(payload, ctx); + } + if (ctx && isBackEdge(ctx, payload.value)) { + return originalRun(payload, ctx); + } + const out = parser(payload.value); + if (out !== INVALID) { + payload.value = out; + return payload; + } + if (ctx) + ctx[FALLBACK_FLAG] = true; + return originalRun(payload, ctx); + }; + wrapped.__originalRun = originalRun; + clone2._zod.bag.fallbackRun = originalRun; + clone2._zod.bag.validator = parser; + clone2._zod.run = wrapped; + if (!liveRun.__originalRun) + installCompiledUserMethods(clone2, schema, parser); + return clone2; +} function installCompiledUserMethods(target, source, parser) { const targetAny = target; const sourceAny = source; @@ -19532,7 +20032,8 @@ function compileFn(schema, options) { const ctx = { constants: /* @__PURE__ */ new Map(), constantCounter: 0, - varCounter: 0 + varCounter: 0, + definite: true }; const doc = new Doc(["input"]); const outputAccessor = generateCheck(doc, ctx, schema, "input", !options?.assertOnly); @@ -19556,6 +20057,7 @@ ${code} if (options?.debug) { fn.code = fullCode; } + fn.definite = ctx.definite; return fn; } function addConstant(ctx, value) { @@ -19567,6 +20069,10 @@ function addConstant(ctx, value) { ctx.constants.set(name, value); return name; } +function addUserConstant(ctx, fn) { + ctx.definite = false; + return addConstant(ctx, fn); +} function newVar(ctx) { return `v${ctx.varCounter++}`; } @@ -19599,6 +20105,7 @@ function compileChild(doc, ctx, schema, accessor, needsValue = true) { } } function emitRuntimeIsland(doc, ctx, schema, accessor) { + ctx.definite = false; const schemaConst = addConstant(ctx, schema); const runConst = addConstant(ctx, runtimeRun); const outVar = newVar(ctx); @@ -19679,6 +20186,9 @@ function generateChecks(doc, ctx, schema, accessor) { case "property": generatePropertyCheck(doc, ctx, def, currentAccessor); break; + case "properties": + generatePropertiesChecks(doc, ctx, def, currentAccessor); + break; case "overwrite": { const newAccessor = newVar(ctx); generateOverwriteCheck(doc, ctx, check2, currentAccessor, newAccessor); @@ -19787,6 +20297,19 @@ function generateMimeTypeCheck(doc, ctx, def, accessor) { doc.write(`if (!${mimeSet}.has(${accessor}.type)) return INVALID;`); } } +function generatePropertiesChecks(doc, ctx, def, accessor) { + if (def.when) { + throw new ZodCompileUnsupportedError(`check with a custom "when" condition`); + } + doc.write(`if (${accessor} == null) return INVALID;`); + const shape = def.shape; + for (const key of Reflect.ownKeys(shape)) { + const keyExpr = typeof key === "symbol" ? addConstant(ctx, key) : esc(key); + const inputVar = newVar(ctx); + doc.write(`const ${inputVar} = ${accessor}[${keyExpr}];`); + compileChild(doc, ctx, shape[key], inputVar, false); + } +} function generatePropertyCheck(doc, ctx, def, accessor) { const propAccessor = `${accessor}[${JSON.stringify(def.property)}]`; generateCheck(doc, ctx, def.schema, propAccessor); @@ -19814,7 +20337,7 @@ function generateCustomRefineCheck(doc, ctx, check2, accessor) { if (isAsyncFunction(def.fn)) { throw new ZodCompileAsyncError("z.compile: async .refine() predicates are not supported"); } - const fnConst = addConstant(ctx, def.fn); + const fnConst = addUserConstant(ctx, def.fn); const throwAsyncConst = addConstant(ctx, throwAsync); const resVar = newVar(ctx); doc.write(`const ${resVar} = ${fnConst}(${accessor});`); @@ -19834,7 +20357,7 @@ function generateCustomRefineCheck(doc, ctx, check2, accessor) { throwAsync(); return fakePayload.issues.length === 0 ? fakePayload.value : INVALID; }; - const helperConst = addConstant(ctx, helperFn); + const helperConst = addUserConstant(ctx, helperFn); const outVar = newVar(ctx); doc.write(`const ${outVar} = ${helperConst}(${accessor});`); doc.write(`if (${outVar} === INVALID) return INVALID;`); @@ -19868,7 +20391,7 @@ var PATTERN_IS_COMPLETE = /* @__PURE__ */ new Set([ "uuid", "xid" ]); -function generateStringFormatCheck(doc, ctx, def, accessor) { +function generateStringFormatCheck(doc, ctx, def, accessor, needsValue = true) { const fmt = def.format; if (fmt === "base64") { const validator = addConstant(ctx, isValidBase64); @@ -19901,9 +20424,14 @@ function generateStringFormatCheck(doc, ctx, def, accessor) { doc.write(`if (!${validator}(${accessor})) return INVALID;`); return accessor; } + if (fmt === "iban") { + const validator = addConstant(ctx, isValidIBAN); + doc.write(`if (!${validator}(${accessor})) return INVALID;`); + return accessor; + } const formatDef = def; if (fmt === "url" || fmt === "httpurl" || formatDef.normalize || formatDef.hostname !== void 0 || formatDef.protocol !== void 0) { - const parseConst = addConstant(ctx, parseURLObject); + const parseConst = addConstant(ctx, validateURL); const defConst = addConstant(ctx, def); const trimVar = newVar(ctx); const urlVar = newVar(ctx); @@ -19918,6 +20446,8 @@ function generateStringFormatCheck(doc, ctx, def, accessor) { const protocolConst = addConstant(ctx, urlProtocolOk); doc.write(`if (!${protocolConst}(${urlVar}, ${defConst}.protocol)) return INVALID;`); } + if (!needsValue) + return null; const outputVar = newVar(ctx); const outputExpr = formatDef.normalize ? `${urlVar}.href` : `${addConstant(ctx, stripTabAndNewline)}(${trimVar})`; doc.write(`const ${outputVar} = ${outputExpr};`); @@ -19977,7 +20507,7 @@ function generateCheck(doc, ctx, schema, accessor, needsValue = true) { let typeAccessor; switch (type) { case "string": - typeAccessor = generateStringCheck(doc, ctx, schema, accessor); + typeAccessor = generateStringCheck(doc, ctx, schema, accessor, buildsValue); break; case "number": typeAccessor = generateNumberCheck(doc, schema, accessor); @@ -20098,12 +20628,12 @@ function generateCheck(doc, ctx, schema, accessor, needsValue = true) { return null; return generateChecks(doc, ctx, schema, typeAccessor); } -function generateStringCheck(doc, ctx, schema, accessor) { +function generateStringCheck(doc, ctx, schema, accessor, needsValue = true) { doc.write(`if (typeof ${accessor} !== "string") return INVALID;`); const def = schema._zod.def; if (def.format === void 0) return accessor; - return generateStringFormatCheck(doc, ctx, def, accessor); + return generateStringFormatCheck(doc, ctx, def, accessor, needsValue); } function generateNumberCheck(doc, schema, accessor) { doc.write(`if (typeof ${accessor} !== "number" || !Number.isFinite(${accessor})) return INVALID;`); @@ -20160,13 +20690,13 @@ function generateObjectCheck(doc, ctx, schema, accessor, buildsValue = true) { const def = schema._zod.def; doc.write(`if (typeof ${accessor} !== "object" || ${accessor} === null || Array.isArray(${accessor})) return INVALID;`); const shape = def.shape; - const keys = Object.keys(shape); + const keys2 = Object.keys(shape); const symbolKeys = Object.getOwnPropertySymbols(shape); - const allKeys = symbolKeys.length ? [...keys, ...symbolKeys] : keys; + const allKeys = symbolKeys.length ? [...keys2, ...symbolKeys] : keys2; const keyExpr = (k2) => typeof k2 === "symbol" ? addConstant(ctx, k2) : esc(k2); const propKey = (k2) => typeof k2 === "symbol" ? `[${keyExpr(k2)}]` : esc(k2); const propShape = shape; - if (keys.includes("__proto__")) { + if (keys2.includes("__proto__")) { throw new ZodCompileUnsupportedError('object shape key "__proto__"'); } const propOutputs = /* @__PURE__ */ new Map(); @@ -20208,7 +20738,7 @@ function generateObjectCheck(doc, ctx, schema, accessor, buildsValue = true) { if (catchall) { const catchallType = catchall._zod.def.type; if (catchallType === "never") { - const condition = keys.map((k2) => `k !== ${esc(k2)}`).join(" && ") || "true"; + const condition = keys2.map((k2) => `k !== ${esc(k2)}`).join(" && ") || "true"; doc.write(`for (const k in ${accessor}) {`); doc.indented((d2) => { d2.write(`if (${condition}) return INVALID;`); @@ -20221,10 +20751,10 @@ function generateObjectCheck(doc, ctx, schema, accessor, buildsValue = true) { } } const outputVar = newVar(ctx); - const hasConditionalKeys = allKeys.some((k2) => mayOutputUndefined(propShape[k2]) || dropsWhenAbsent(propShape[k2])); + const hasConditionalKeys = allKeys.some((k2) => mayOmitUndefined(propShape[k2]) || dropsWhenAbsent(propShape[k2])); if (!buildsValue) { if (unknownKeysMode === "schema") { - const knownSet = keys.length > 0 ? addConstant(ctx, new Set(keys)) : null; + const knownSet = keys2.length > 0 ? addConstant(ctx, new Set(keys2)) : null; doc.write(`for (const k in ${accessor}) {`); doc.indented((d2) => { d2.write(`if (k === "__proto__") continue;`); @@ -20248,7 +20778,7 @@ function generateObjectCheck(doc, ctx, schema, accessor, buildsValue = true) { const out = propOutputs.get(k2); if (dropsWhenAbsent(propShape[k2])) { doc.write(`if (${kx} in ${accessor}) ${outputVar}[${kx}] = ${out};`); - } else if (mayOutputUndefined(propShape[k2])) { + } else if (mayOmitUndefined(propShape[k2])) { doc.write(`if (${out} !== undefined || ${kx} in ${accessor}) ${outputVar}[${kx}] = ${out};`); } else { doc.write(`${outputVar}[${kx}] = ${out};`); @@ -20256,7 +20786,7 @@ function generateObjectCheck(doc, ctx, schema, accessor, buildsValue = true) { } } if (unknownKeysMode !== "none") { - const knownSet = keys.length > 0 ? addConstant(ctx, new Set(keys)) : null; + const knownSet = keys2.length > 0 ? addConstant(ctx, new Set(keys2)) : null; doc.write(`for (const k in ${accessor}) {`); doc.indented((d2) => { d2.write(`if (k === "__proto__") continue;`); @@ -20381,6 +20911,9 @@ function fastPathAcceptsAbsence(schema) { function dropsWhenAbsent(schema) { return schema._zod.optin === "optional" && schema._zod.optout === "optional"; } +function mayOmitUndefined(schema) { + return (schema._zod.optin !== "defaulted" || schema._zod.optout === "optional") && mayOutputUndefined(schema); +} function mayOutputUndefined(schema) { const def = schema._zod.def; switch (def.type) { @@ -20437,9 +20970,15 @@ function generateNullableCheck(doc, ctx, schema, accessor, buildsValue = true) { doc.write(`}`); return outputVar; } +function generateSizeGate(doc, schema, size) { + for (const [max, check2] of sizeCaps(schema._zod.def.checks)) { + doc.write(`if (${size} > ${numericOperand(max, check2._zod.def.check)}) return INVALID;`); + } +} function generateArrayCheck(doc, ctx, schema, accessor, buildsValue = true) { const def = schema._zod.def; doc.write(`if (!Array.isArray(${accessor})) return INVALID;`); + generateSizeGate(doc, schema, `${accessor}.length`); const outputVar = buildsValue ? newVar(ctx) : null; const iVar = newVar(ctx); const elemVar = newVar(ctx); @@ -20723,6 +21262,7 @@ function literalEquality(ctx, accessor, value) { } function generateIntersectionCheck(doc, ctx, schema, accessor) { const def = schema._zod.def; + ctx.definite = false; const leftOutput = compileChild(doc, ctx, def.left, accessor); const rightOutput = compileChild(doc, ctx, def.right, accessor); const mergeConst = addConstant(ctx, mergeValues); @@ -20776,14 +21316,13 @@ function generateRecordCheck(doc, ctx, schema, accessor) { const keyIsBareString = keyDef.type === "string" && keyDef.format === void 0 && !keyDef.coerce && (keyDef.checks?.length ?? 0) === 0; if (!keyIsBareString) { const isLoose = def.mode === "loose"; - const keyFast = addConstant(ctx, compileFn(def.keyType)); + const keyFn = compileFn(def.keyType); + if (keyFn.definite === false) + ctx.definite = false; + const keyFast = addConstant(ctx, keyFn); const numericConst = addConstant(ctx, number); - const propIsEnumerableConst = addConstant(ctx, Object.prototype.propertyIsEnumerable); const outKeyVar = newVar(ctx); - doc.write(`for (const ${kVar} of Reflect.ownKeys(${accessor})) {`); - doc.indented((d2) => { - d2.write(`if (${kVar} === "__proto__") continue;`); - d2.write(`if (!${propIsEnumerableConst}.call(${accessor}, ${kVar})) continue;`); + emitOwnKeys(doc, ctx, accessor, kVar, (d2) => { d2.write(`let ${outKeyVar} = ${keyFast}(${kVar});`); d2.write(`if (${outKeyVar} === INVALID && typeof ${kVar} === "string" && ${numericConst}.test(${kVar})) ${outKeyVar} = ${keyFast}(Number(${kVar}));`); if (isLoose) { @@ -20797,21 +21336,39 @@ function generateRecordCheck(doc, ctx, schema, accessor) { const valOutput = compileChild(d2, ctx, def.valueType, valueVar); d2.write(`${outputVar}[${outKeyVar}] = ${valOutput};`); }); - doc.write(`}`); return outputVar; } - const propIsEnumerable = addConstant(ctx, Object.prototype.propertyIsEnumerable); - doc.write(`for (const ${kVar} of Reflect.ownKeys(${accessor})) {`); - doc.indented((d2) => { - d2.write(`if (${kVar} === "__proto__") continue;`); - d2.write(`if (!${propIsEnumerable}.call(${accessor}, ${kVar})) continue;`); - d2.write(`if (typeof ${kVar} !== "string") return INVALID;`); + emitOwnKeys(doc, ctx, accessor, kVar, (d2) => { d2.write(`const ${valVar} = ${accessor}[${kVar}];`); const valOutput = compileChild(d2, ctx, def.valueType, valVar); d2.write(`${outputVar}[${kVar}] = ${valOutput};`); + }, `return INVALID;`); + return outputVar; +} +function emitOwnKeys(doc, ctx, accessor, kVar, body, onSymbol) { + const propIsEnumerableConst = addConstant(ctx, Object.prototype.propertyIsEnumerable); + const symsVar = newVar(ctx); + const keysVar = newVar(ctx); + const iVar = newVar(ctx); + doc.write(`const ${symsVar} = Object.getOwnPropertySymbols(${accessor});`); + doc.write(`const ${keysVar} = Object.getOwnPropertyNames(${accessor});`); + doc.write(`for (let ${iVar} = 0; ${iVar} < ${keysVar}.length; ${iVar}++) {`); + doc.indented((d2) => { + d2.write(`const ${kVar} = ${keysVar}[${iVar}];`); + d2.write(`if (${kVar} === "__proto__" || !${propIsEnumerableConst}.call(${accessor}, ${kVar})) continue;`); + body(d2); + }); + doc.write(`}`); + doc.write(`for (let ${iVar} = 0; ${iVar} < ${symsVar}.length; ${iVar}++) {`); + doc.indented((d2) => { + d2.write(`const ${kVar} = ${symsVar}[${iVar}];`); + d2.write(`if (!${propIsEnumerableConst}.call(${accessor}, ${kVar})) continue;`); + if (onSymbol) + d2.write(onSymbol); + else + body(d2); }); doc.write(`}`); - return outputVar; } function literalPropertyKey(ctx, key) { if (typeof key === "string") @@ -20821,6 +21378,7 @@ function literalPropertyKey(ctx, key) { function generateMapCheck(doc, ctx, schema, accessor) { const def = schema._zod.def; doc.write(`if (!(${accessor} instanceof Map)) return INVALID;`); + generateSizeGate(doc, schema, `${accessor}.size`); const outputVar = newVar(ctx); const kVar = newVar(ctx); const valVar = newVar(ctx); @@ -20837,6 +21395,7 @@ function generateMapCheck(doc, ctx, schema, accessor) { function generateSetCheck(doc, ctx, schema, accessor) { const def = schema._zod.def; doc.write(`if (!(${accessor} instanceof Set)) return INVALID;`); + generateSizeGate(doc, schema, `${accessor}.size`); const outputVar = newVar(ctx); const valVar = newVar(ctx); doc.write(`const ${outputVar} = new Set();`); @@ -20864,7 +21423,7 @@ function generateTemplateLiteralCheck(doc, ctx, schema, accessor) { } function generateLazyCheck(doc, ctx, schema, accessor) { const def = schema._zod.def; - const getterConst = addConstant(ctx, def.getter); + const getterConst = addUserConstant(ctx, def.getter); const cacheConst = addConstant(ctx, { parser: null }); doc.write(`if (!${cacheConst}.parser) {`); doc.indented((d2) => { @@ -20897,7 +21456,7 @@ function generatePipeCheck(doc, ctx, schema, accessor) { return INVALID; return fakePayload.issues.length === 0 ? result : INVALID; }; - const helperConst = addConstant(ctx, helperFn); + const helperConst = addUserConstant(ctx, helperFn); const transformedVar = newVar(ctx); doc.write(`const ${transformedVar} = ${helperConst}(${inputOutput});`); doc.write(`if (${transformedVar} === INVALID) return INVALID;`); @@ -20915,7 +21474,7 @@ function generateCustomCheck(doc, ctx, schema, accessor) { if (isAsyncFunction(def.fn)) { throw new ZodCompileAsyncError("z.compile: async custom predicates are not supported"); } - const fnConst = addConstant(ctx, def.fn); + const fnConst = addUserConstant(ctx, def.fn); const throwAsyncConst = addConstant(ctx, throwAsync); const resVar = newVar(ctx); doc.write(`const ${resVar} = ${fnConst}(${accessor});`); @@ -20948,7 +21507,7 @@ function generateCatchCheck(doc, ctx, schema, accessor) { }); doc.write(`})();`); const innerConst = addConstant(ctx, def.innerType); - const catchConst = addConstant(ctx, def.catchValue); + const catchConst = addUserConstant(ctx, def.catchValue); const catchHelperConst = addConstant(ctx, runtimeCatch); doc.write(`if (${outputVar} === INVALID) {`); doc.indented((d2) => { @@ -20972,7 +21531,7 @@ function generateTransformCheck(doc, ctx, schema, accessor) { return INVALID; return fakePayload.issues.length === 0 ? result : INVALID; }; - const helperConst = addConstant(ctx, helperFn); + const helperConst = addUserConstant(ctx, helperFn); const outputVar = newVar(ctx); doc.write(`const ${outputVar} = ${helperConst}(${accessor});`); doc.write(`if (${outputVar} === INVALID) return INVALID;`); @@ -20982,20 +21541,18 @@ function generateTransformCheck(doc, ctx, schema, accessor) { } // node_modules/zod/v4/core/api.js +function snapshotChecks(def) { + if (def.checks) + def.checks = [...def.checks]; + return def; +} // @__NO_SIDE_EFFECTS__ function _string(Class2, params) { - return new Class2({ - type: "string", - ...normalizeParams(params) - }); + return new Class2(snapshotChecks({ type: "string", ...normalizeParams(params) })); } // @__NO_SIDE_EFFECTS__ function _coercedString(Class2, params) { - return new Class2({ - type: "string", - coerce: true, - ...normalizeParams(params) - }); + return new Class2(snapshotChecks({ type: "string", coerce: true, ...normalizeParams(params) })); } // @__NO_SIDE_EFFECTS__ function _email(Class2, params) { @@ -21231,6 +21788,16 @@ function _creditCard(Class2, params) { }); } // @__NO_SIDE_EFFECTS__ +function _iban(Class2, params) { + return new Class2({ + type: "string", + format: "iban", + check: "string_format", + abort: false, + ...normalizeParams(params) + }); +} +// @__NO_SIDE_EFFECTS__ function _jwt(Class2, params) { return new Class2({ type: "string", @@ -21289,20 +21856,11 @@ function _isoDuration(Class2, params) { } // @__NO_SIDE_EFFECTS__ function _number(Class2, params) { - return new Class2({ - type: "number", - checks: [], - ...normalizeParams(params) - }); + return new Class2(snapshotChecks({ type: "number", checks: [], ...normalizeParams(params) })); } // @__NO_SIDE_EFFECTS__ function _coercedNumber(Class2, params) { - return new Class2({ - type: "number", - coerce: true, - checks: [], - ...normalizeParams(params) - }); + return new Class2(snapshotChecks({ type: "number", coerce: true, checks: [], ...normalizeParams(params) })); } // @__NO_SIDE_EFFECTS__ function _int(Class2, params) { @@ -21644,8 +22202,12 @@ function _property(property, schema, params) { }); } // @__NO_SIDE_EFFECTS__ -function _properties(shape) { - return Object.entries(shape).map(([property, schema]) => new $ZodCheckProperty({ check: "property", property, schema })); +function _properties(shape, params) { + return new $ZodCheckProperties({ + check: "properties", + shape, + ...normalizeParams(params) + }); } // @__NO_SIDE_EFFECTS__ function _mime(types, params) { @@ -22082,7 +22644,7 @@ function handleUnrepresentable(schema, ctx, json2, params, message) { Object.assign(json2, result); return true; } -function process2(schema, ctx, _params = { path: [], schemaPath: [] }) { +function processSchema(schema, ctx, _params = { path: [], schemaPath: [] }) { var _a3; const def = schema._zod.def; const seen = ctx.seen.get(schema); @@ -22121,7 +22683,7 @@ function process2(schema, ctx, _params = { path: [], schemaPath: [] }) { if (parent) { if (!result.ref) result.ref = parent; - process2(parent, ctx, params); + processSchema(parent, ctx, params); ctx.seen.get(parent).isParent = true; } } @@ -22228,7 +22790,6 @@ Set the \`cycles\` parameter to \`"ref"\` to resolve cyclical schemas with defs. if (seen.count > 1) { if (ctx.reused === "ref") { extractToDef(entry); - continue; } } } @@ -22244,8 +22805,8 @@ function compactTypeUnion(schema) { if (!option || typeof option !== "object") return; compactTypeUnion(option); - const keys = Object.keys(option); - if (keys.length !== 1 || keys[0] !== "type") + const keys2 = Object.keys(option); + if (keys2.length !== 1 || keys2[0] !== "type") return; const type = option.type; for (const member of Array.isArray(type) ? type : [type]) { @@ -22279,8 +22840,8 @@ function foldObjects(members2) { } const properties = {}; const required2 = /* @__PURE__ */ new Set(); - for (const object3 of objects) { - for (const key in object3.properties) { + for (const object4 of objects) { + for (const key in object4.properties) { if (Object.prototype.hasOwnProperty.call(properties, key)) continue; const parts = []; @@ -22294,18 +22855,18 @@ function foldObjects(members2) { const merged = parts.length === 1 ? parts[0] : foldObjects(parts) ?? { allOf: parts }; assignProp(properties, key, merged); } - for (const key of object3.required ?? []) + for (const key of object4.required ?? []) required2.add(key); } const folded = { type: "object", properties }; if (required2.size) folded.required = [...required2]; - if (objects.every((object3) => object3.additionalProperties === false)) { + if (objects.every((object4) => object4.additionalProperties === false)) { folded.additionalProperties = false; } else { const constraints = []; - for (const object3 of objects) { - const constraint = undeclaredConstraint(object3); + for (const object4 of objects) { + const constraint = undeclaredConstraint(object4); if (constraint && !constraints.some((seen) => JSON.stringify(seen) === JSON.stringify(constraint))) constraints.push(constraint); } @@ -22556,19 +23117,107 @@ function isTransforming(_schema, _ctx) { } var createToJSONSchemaMethod = (schema, processors = {}) => (params) => { const ctx = initializeContext({ ...params, processors }); - process2(schema, ctx); + processSchema(schema, ctx); extractDefs(ctx, schema); return finalize(ctx, schema); }; var createStandardJSONSchemaMethod = (schema, io, processors = {}) => (params) => { const { libraryOptions, target } = params ?? {}; const ctx = initializeContext({ ...libraryOptions ?? {}, target, io, processors }); - process2(schema, ctx); + processSchema(schema, ctx); extractDefs(ctx, schema); return finalize(ctx, schema); }; // node_modules/zod/v4/core/json-schema-processors.js +var narrowMin = (agg, key, value) => { + if (agg[key] === void 0 || value > agg[key]) + agg[key] = value; +}; +var narrowMax = (agg, key, value) => { + if (agg[key] === void 0 || value < agg[key]) + agg[key] = value; +}; +var narrowBoth = (agg, value) => { + narrowMin(agg, "minimum", value); + narrowMax(agg, "maximum", value); +}; +var addDivisor = (agg, value) => { + agg.multipleOf ?? (agg.multipleOf = []); + if (!agg.multipleOf.includes(value)) + agg.multipleOf.push(value); +}; +var addPattern = (agg, pattern) => { + agg.patterns ?? (agg.patterns = /* @__PURE__ */ new Set()); + agg.patterns.add(pattern); +}; +var intersectMime = (agg, mime) => { + agg.mime = agg.mime ? agg.mime.filter((m2) => mime.includes(m2)) : [...mime]; +}; +var setFormat = (agg, format) => { + agg.format = format; + if (format.includes("int")) + agg.isInt = true; +}; +var minContributor = (agg, def) => narrowMin(agg, "minimum", def.minimum); +var maxContributor = (agg, def) => narrowMax(agg, "maximum", def.maximum); +var formatContributor = (ranges) => (agg, def) => { + setFormat(agg, def.format); + const [minimum, maximum] = ranges[def.format]; + narrowMin(agg, "minimum", minimum); + narrowMax(agg, "maximum", maximum); +}; +var contributors = { + greater_than: (agg, def) => narrowMin(agg, def.inclusive ? "minimum" : "exclusiveMinimum", def.value), + less_than: (agg, def) => narrowMax(agg, def.inclusive ? "maximum" : "exclusiveMaximum", def.value), + multiple_of: (agg, def) => addDivisor(agg, def.value), + number_format: formatContributor(NUMBER_FORMAT_RANGES), + bigint_format: formatContributor(BIGINT_FORMAT_RANGES), + min_length: minContributor, + max_length: maxContributor, + length_equals: (agg, def) => narrowBoth(agg, def.length), + min_size: minContributor, + max_size: maxContributor, + size_equals: (agg, def) => narrowBoth(agg, def.size), + string_format: (agg, def) => { + setFormat(agg, def.format); + if (def.pattern) + addPattern(agg, def.pattern); + if (def.format === "base64" || def.format === "base64url") + agg.contentEncoding = def.format; + if (def.local || def.precision === -1) + agg.laxFormat = true; + }, + mime_type: (agg, def) => intersectMime(agg, def.mime) +}; +function aggregateChecks(schema) { + const agg = {}; + const def = schema._zod.def; + const list = schema._zod.traits.has("$ZodCheck") ? [schema, ...def.checks ?? []] : def.checks ?? []; + for (const ch of list) + contributors[ch._zod.def.check]?.(agg, ch._zod.def); + const bag = schema._zod.bag; + if (bag.minimum !== void 0) + narrowMin(agg, "minimum", bag.minimum); + if (bag.exclusiveMinimum !== void 0) + narrowMin(agg, "exclusiveMinimum", bag.exclusiveMinimum); + if (bag.maximum !== void 0) + narrowMax(agg, "maximum", bag.maximum); + if (bag.exclusiveMaximum !== void 0) + narrowMax(agg, "exclusiveMaximum", bag.exclusiveMaximum); + if (bag.multipleOf !== void 0) + addDivisor(agg, bag.multipleOf); + if (bag.format !== void 0) { + agg.format ?? (agg.format = bag.format); + if (bag.format.includes("int")) + agg.isInt = true; + } + if (bag.mime) + intersectMime(agg, bag.mime); + for (const pattern of bag.patterns ?? []) + addPattern(agg, pattern); + return agg; +} var formatMap = { guid: "uuid", url: "uri", @@ -22577,10 +23226,15 @@ var formatMap = { regex: "" // do not set }; +var exactPatterns = /* @__PURE__ */ new Map([ + [base64Charset, base64], + [base64urlCharset, base64url] +]); +var exactPattern = (p) => exactPatterns.get(p) ?? p; var stringProcessor = (schema, ctx, _json, _params) => { const json2 = _json; json2.type = "string"; - const { minimum, maximum, format, patterns, contentEncoding, laxFormat } = schema._zod.bag; + const { minimum, maximum, format, patterns, contentEncoding, laxFormat } = aggregateChecks(schema); if (typeof minimum === "number") json2.minLength = minimum; if (typeof maximum === "number") @@ -22596,7 +23250,7 @@ var stringProcessor = (schema, ctx, _json, _params) => { if (contentEncoding) json2.contentEncoding = contentEncoding; if (patterns && patterns.size > 0) { - const patternList = [...patterns]; + const patternList = [...patterns].map(exactPattern); if (patternList.length === 1) json2.pattern = patternList[0].source; else if (patternList.length > 1) { @@ -22611,11 +23265,8 @@ var stringProcessor = (schema, ctx, _json, _params) => { }; var numberProcessor = (schema, ctx, _json, params) => { const json2 = _json; - const { minimum, maximum, format, multipleOf, exclusiveMaximum, exclusiveMinimum } = schema._zod.bag; - if (typeof format === "string" && format.includes("int")) - json2.type = "integer"; - else - json2.type = "number"; + const { minimum, maximum, multipleOf, exclusiveMaximum, exclusiveMinimum, isInt } = aggregateChecks(schema); + json2.type = isInt ? "integer" : "number"; const exMin = typeof exclusiveMinimum === "number" && exclusiveMinimum >= (minimum ?? Number.NEGATIVE_INFINITY); const exMax = typeof exclusiveMaximum === "number" && exclusiveMaximum <= (maximum ?? Number.POSITIVE_INFINITY); const legacy = ctx.target === "draft-04" || ctx.target === "openapi-3.0"; @@ -22639,11 +23290,19 @@ var numberProcessor = (schema, ctx, _json, params) => { } else if (typeof maximum === "number") { json2.maximum = maximum; } - if (typeof multipleOf === "number") { - if (Number.isFinite(multipleOf) && multipleOf !== 0) - json2.multipleOf = Math.abs(multipleOf); - else - handleUnrepresentable(schema, ctx, json2, params, `A multipleOf divisor of ${multipleOf} cannot be represented in JSON Schema`); + if (multipleOf) { + const divisors = /* @__PURE__ */ new Set(); + for (const divisor of multipleOf) { + if (Number.isFinite(divisor) && divisor !== 0) + divisors.add(Math.abs(divisor)); + else + handleUnrepresentable(schema, ctx, json2, params, `A multipleOf divisor of ${divisor} cannot be represented in JSON Schema`); + } + const [first, ...rest] = divisors; + if (first !== void 0) + json2.multipleOf = first; + if (rest.length) + json2.allOf = [...json2.allOf ?? [], ...rest.map((m2) => ({ multipleOf: m2 }))]; } }; var booleanProcessor = (_schema, _ctx, json2, _params) => { @@ -22746,27 +23405,22 @@ var templateLiteralProcessor = (schema, _ctx, json2, _params) => { }; var fileProcessor = (schema, _ctx, json2, _params) => { const _json = json2; - const file2 = { - type: "string", - format: "binary", - contentEncoding: "binary" - }; - const { minimum, maximum, mime } = schema._zod.bag; + _json.type = "string"; + _json.format = "binary"; + _json.contentEncoding = "binary"; + const { minimum, maximum, mime } = aggregateChecks(schema); if (minimum !== void 0) - file2.minLength = minimum; + _json.minLength = minimum; if (maximum !== void 0) - file2.maxLength = maximum; - if (mime) { - if (mime.length === 1) { - file2.contentMediaType = mime[0]; - Object.assign(_json, file2); - } else { - Object.assign(_json, file2); - _json.anyOf = mime.map((m2) => ({ contentMediaType: m2 })); - } - } else { - Object.assign(_json, file2); - } + _json.maxLength = maximum; + if (!mime) + return; + if (mime.length === 0) + _json.not = {}; + else if (mime.length === 1) + _json.contentMediaType = mime[0]; + else + _json.anyOf = mime.map((m2) => ({ contentMediaType: m2 })); }; var successProcessor = (_schema, _ctx, json2, _params) => { json2.type = "boolean"; @@ -22789,13 +23443,13 @@ var setProcessor = (schema, ctx, json2, params) => { var arrayProcessor = (schema, ctx, _json, params) => { const json2 = _json; const def = schema._zod.def; - const { minimum, maximum } = schema._zod.bag; + const { minimum, maximum } = aggregateChecks(schema); if (typeof minimum === "number") json2.minItems = minimum; if (typeof maximum === "number") json2.maxItems = maximum; json2.type = "array"; - json2.items = process2(def.element, ctx, { + json2.items = processSchema(def.element, ctx, { ...params, path: [...params.path, "items"] }); @@ -22821,22 +23475,20 @@ var objectProcessor = (schema, ctx, _json, params) => { json2.type = "object"; json2.properties = {}; for (const key in shape) { - assignProp(json2.properties, key, process2(shape[key], ctx, { + assignProp(json2.properties, key, processSchema(shape[key], ctx, { ...params, path: [...params.path, "properties", key] })); } - const allKeys = new Set(Object.keys(shape)); - const requiredKeys = new Set([...allKeys].filter((key) => { + const requiredKeys = []; + for (const key of Object.keys(shape)) { const field = def.shape[key]; - if (ctx.io === "input") { - return inputOptin(field) === void 0; - } else { - return field._zod.optout === void 0; + if (ctx.io === "input" ? inputOptin(field) === void 0 : field._zod.optout === void 0) { + requiredKeys.push(key); } - })); - if (requiredKeys.size > 0) { - json2.required = Array.from(requiredKeys); + } + if (requiredKeys.length > 0) { + json2.required = requiredKeys; } if (def.catchall?._zod.def.type === "never") { json2.additionalProperties = false; @@ -22844,7 +23496,7 @@ var objectProcessor = (schema, ctx, _json, params) => { if (ctx.io === "output") json2.additionalProperties = false; } else if (def.catchall) { - json2.additionalProperties = process2(def.catchall, ctx, { + json2.additionalProperties = processSchema(def.catchall, ctx, { ...params, path: [...params.path, "additionalProperties"] }); @@ -22853,7 +23505,7 @@ var objectProcessor = (schema, ctx, _json, params) => { var unionProcessor = (schema, ctx, json2, params) => { const def = schema._zod.def; const isExclusive = def.inclusive === false; - const options = def.options.map((x2, i) => process2(x2, ctx, { + const options = def.options.map((x2, i) => processSchema(x2, ctx, { ...params, path: [...params.path, isExclusive ? "oneOf" : "anyOf", i] })); @@ -22865,11 +23517,11 @@ var unionProcessor = (schema, ctx, json2, params) => { }; var intersectionProcessor = (schema, ctx, json2, params) => { const def = schema._zod.def; - const a = process2(def.left, ctx, { + const a = processSchema(def.left, ctx, { ...params, path: [...params.path, "allOf", 0] }); - const b2 = process2(def.right, ctx, { + const b2 = processSchema(def.right, ctx, { ...params, path: [...params.path, "allOf", 1] }); @@ -22887,11 +23539,11 @@ var tupleProcessor = (schema, ctx, _json, params) => { json2.type = "array"; const prefixPath = ctx.target === "draft-2020-12" ? "prefixItems" : "items"; const restPath = ctx.target === "draft-2020-12" ? "items" : ctx.target === "openapi-3.0" ? "items" : "additionalItems"; - const prefixItems = def.items.map((x2, i) => process2(x2, ctx, { + const prefixItems = def.items.map((x2, i) => processSchema(x2, ctx, { ...params, path: [...params.path, prefixPath, i] })); - const rest = def.rest ? process2(def.rest, ctx, { + const rest = def.rest ? processSchema(def.rest, ctx, { ...params, path: [...params.path, restPath, ...ctx.target === "openapi-3.0" ? [def.items.length] : []] }) : null; @@ -22939,7 +23591,7 @@ var tupleProcessor = (schema, ctx, _json, params) => { if (isClosed) json2.maxItems = maxItems; } - const { minimum, maximum } = schema._zod.bag; + const { minimum, maximum } = aggregateChecks(schema); if (typeof minimum === "number") json2.minItems = minimum; if (typeof maximum === "number") @@ -23013,20 +23665,19 @@ var recordProcessor = (schema, ctx, _json, params) => { const def = schema._zod.def; json2.type = "object"; const keyType = def.keyType; - const keyBag = keyType._zod.bag; - const patterns = keyBag?.patterns; + const patterns = aggregateChecks(keyType).patterns; if (def.mode === "loose" && patterns && patterns.size > 0) { - const valueSchema = process2(def.valueType, ctx, { + const valueSchema = processSchema(def.valueType, ctx, { ...params, path: [...params.path, "patternProperties", "*"] }); json2.patternProperties = {}; for (const pattern of patterns) { - assignProp(json2.patternProperties, pattern.source, valueSchema); + assignProp(json2.patternProperties, exactPattern(pattern).source, valueSchema); } } else { if (ctx.target === "draft-07" || ctx.target === "draft-2020-12") { - json2.propertyNames = process2(def.keyType, ctx, { + json2.propertyNames = processSchema(def.keyType, ctx, { ...params, path: [...params.path, "propertyNames"] }); @@ -23038,7 +23689,7 @@ var recordProcessor = (schema, ctx, _json, params) => { } pending.push(schema); } - json2.additionalProperties = process2(def.valueType, ctx, { + json2.additionalProperties = processSchema(def.valueType, ctx, { ...params, path: [...params.path, "additionalProperties"] }); @@ -23054,7 +23705,7 @@ var recordProcessor = (schema, ctx, _json, params) => { }; var nullableProcessor = (schema, ctx, json2, params) => { const def = schema._zod.def; - const inner = process2(def.innerType, ctx, params); + const inner = processSchema(def.innerType, ctx, params); const seen = ctx.seen.get(schema); if (ctx.target === "openapi-3.0") { seen.ref = def.innerType; @@ -23065,7 +23716,7 @@ var nullableProcessor = (schema, ctx, json2, params) => { }; var nonoptionalProcessor = (schema, ctx, _json, params) => { const def = schema._zod.def; - process2(def.innerType, ctx, params); + processSchema(def.innerType, ctx, params); const seen = ctx.seen.get(schema); seen.ref = def.innerType; }; @@ -23085,7 +23736,7 @@ function serializeDefaultValue(value, schema, ctx, json2, params) { } var defaultProcessor = (schema, ctx, json2, params) => { const def = schema._zod.def; - process2(def.innerType, ctx, params); + processSchema(def.innerType, ctx, params); const seen = ctx.seen.get(schema); seen.ref = def.innerType; const value = serializeDefaultValue(def.defaultValue, schema, ctx, json2, params); @@ -23094,7 +23745,7 @@ var defaultProcessor = (schema, ctx, json2, params) => { }; var prefaultProcessor = (schema, ctx, json2, params) => { const def = schema._zod.def; - process2(def.innerType, ctx, params); + processSchema(def.innerType, ctx, params); const seen = ctx.seen.get(schema); seen.ref = def.innerType; if (ctx.io !== "input") @@ -23105,7 +23756,7 @@ var prefaultProcessor = (schema, ctx, json2, params) => { }; var catchProcessor = (schema, ctx, json2, params) => { const def = schema._zod.def; - process2(def.innerType, ctx, params); + processSchema(def.innerType, ctx, params); const seen = ctx.seen.get(schema); seen.ref = def.innerType; let catchValue; @@ -23121,32 +23772,32 @@ var pipeProcessor = (schema, ctx, _json, params) => { const def = schema._zod.def; const inIsTransform = def.in._zod.traits.has("$ZodTransform"); const innerType = ctx.io === "input" ? inIsTransform ? def.out : def.in : def.out; - process2(innerType, ctx, params); + processSchema(innerType, ctx, params); const seen = ctx.seen.get(schema); seen.ref = innerType; }; var readonlyProcessor = (schema, ctx, json2, params) => { const def = schema._zod.def; - process2(def.innerType, ctx, params); + processSchema(def.innerType, ctx, params); const seen = ctx.seen.get(schema); seen.ref = def.innerType; json2.readOnly = true; }; var promiseProcessor = (schema, ctx, _json, params) => { const def = schema._zod.def; - process2(def.innerType, ctx, params); + processSchema(def.innerType, ctx, params); const seen = ctx.seen.get(schema); seen.ref = def.innerType; }; var optionalProcessor = (schema, ctx, _json, params) => { const def = schema._zod.def; - process2(def.innerType, ctx, params); + processSchema(def.innerType, ctx, params); const seen = ctx.seen.get(schema); seen.ref = def.innerType; }; var lazyProcessor = (schema, ctx, _json, params) => { const innerType = schema._zod.innerType; - process2(innerType, ctx, params); + processSchema(innerType, ctx, params); const seen = ctx.seen.get(schema); seen.ref = innerType; }; @@ -23198,7 +23849,7 @@ function toJSONSchema(input2, params) { const defs = {}; for (const entry of registry2._idmap.entries()) { const [_2, schema] = entry; - process2(schema, ctx2); + processSchema(schema, ctx2); } const schemas = {}; const external = { @@ -23221,7 +23872,7 @@ function toJSONSchema(input2, params) { return { schemas }; } const ctx = initializeContext({ ...params, processors: allProcessors }); - process2(input2, ctx); + processSchema(input2, ctx); extractDefs(ctx, input2); return finalize(ctx, input2); } @@ -23280,7 +23931,7 @@ var JSONSchemaGenerator = class { * This must be called before emit(). */ process(schema, _params = { path: [], schemaPath: [] }) { - return process2(schema, this.ctx, _params); + return processSchema(schema, this.ctx, _params); } /** * Emit the final JSON Schema after processing. @@ -23337,10 +23988,10 @@ function visit(schema, fnOrHandlers) { switch (kind) { case "object": { const oldShape = def.shape; - const keys = Object.keys(oldShape); + const keys2 = Object.keys(oldShape); let changed = false; const newShape = {}; - for (const k2 of keys) { + for (const k2 of keys2) { const mapped = run(oldShape[k2]); if (mapped !== oldShape[k2]) changed = true; @@ -23479,24 +24130,24 @@ function safeParse2(schema, data) { function getObjectShape(schema) { if (!schema) return void 0; - let rawShape; + let rawShape2; if (isZ4Schema(schema)) { const v4Schema = schema; - rawShape = v4Schema._zod?.def?.shape; + rawShape2 = v4Schema._zod?.def?.shape; } else { const v3Schema = schema; - rawShape = v3Schema.shape; + rawShape2 = v3Schema.shape; } - if (!rawShape) + if (!rawShape2) return void 0; - if (typeof rawShape === "function") { + if (typeof rawShape2 === "function") { try { - return rawShape(); + return rawShape2(); } catch { return void 0; } } - return rawShape; + return rawShape2; } function getLiteralValue(schema) { if (isZ4Schema(schema)) { @@ -23531,6 +24182,7 @@ __export(external_exports, { $brand: () => $brand, $input: () => $input, $output: () => $output, + INVALID: () => INVALID, NEVER: () => NEVER, TimePrecision: () => TimePrecision, ZodAny: () => ZodAny, @@ -23564,12 +24216,14 @@ __export(external_exports, { ZodFirstPartyTypeKind: () => ZodFirstPartyTypeKind, ZodFunction: () => ZodFunction, ZodGUID: () => ZodGUID, + ZodIBAN: () => ZodIBAN, ZodIPv4: () => ZodIPv4, ZodIPv6: () => ZodIPv6, ZodISODate: () => ZodISODate, ZodISODateTime: () => ZodISODateTime, ZodISODuration: () => ZodISODuration, ZodISOTime: () => ZodISOTime, + ZodInstanceOf: () => ZodInstanceOf, ZodIntersection: () => ZodIntersection, ZodIssueCode: () => ZodIssueCode, ZodJWT: () => ZodJWT, @@ -23635,6 +24289,7 @@ __export(external_exports, { creditCard: () => creditCard2, cuid: () => cuid3, cuid2: () => cuid22, + currencyCode: () => currencyCode2, custom: () => custom, date: () => date2, decode: () => decode2, @@ -23667,6 +24322,7 @@ __export(external_exports, { hex: () => hex2, hostname: () => hostname2, httpUrl: () => httpUrl, + iban: () => iban2, includes: () => _includes, input: () => input, instanceof: () => _instanceof, @@ -23778,6 +24434,7 @@ __export(external_exports, { validate: () => validate, validateAsync: () => validateAsync, void: () => _void2, + withParser: () => withParser, xid: () => xid2, xor: () => xor }); @@ -23812,12 +24469,14 @@ __export(schemas_exports2, { ZodFile: () => ZodFile, ZodFunction: () => ZodFunction, ZodGUID: () => ZodGUID, + ZodIBAN: () => ZodIBAN, ZodIPv4: () => ZodIPv4, ZodIPv6: () => ZodIPv6, ZodISODate: () => ZodISODate, ZodISODateTime: () => ZodISODateTime, ZodISODuration: () => ZodISODuration, ZodISOTime: () => ZodISOTime, + ZodInstanceOf: () => ZodInstanceOf, ZodIntersection: () => ZodIntersection, ZodJWT: () => ZodJWT, ZodKSUID: () => ZodKSUID, @@ -23876,6 +24535,7 @@ __export(schemas_exports2, { creditCard: () => creditCard2, cuid: () => cuid3, cuid2: () => cuid22, + currencyCode: () => currencyCode2, custom: () => custom, date: () => date2, describe: () => describe2, @@ -23894,6 +24554,7 @@ __export(schemas_exports2, { hex: () => hex2, hostname: () => hostname2, httpUrl: () => httpUrl, + iban: () => iban2, instanceof: () => _instanceof, int: () => int, int32: () => int32, @@ -24196,6 +24857,12 @@ var ZodType = /* @__PURE__ */ $constructor("ZodType", (inst, def) => { set spa(value) { util_exports.own(this, "spa", value); }, + validate(data, params) { + return validate(this, data, params); + }, + validateAsync(data, params) { + return validateAsync(this, data, params); + }, encode: function _encode2(data, params) { return encode2(this, data, params, { callee: _encode2 }); }, @@ -24232,61 +24899,65 @@ var ZodType = /* @__PURE__ */ $constructor("ZodType", (inst, def) => { return this._zod.def; } }); -var _ZodString = /* @__PURE__ */ $constructor("_ZodString", (inst, def) => { - $ZodString.init(inst, def); - ZodType.init(inst, def); - inst._zod.processJSONSchema = (ctx, json2, params) => stringProcessor(inst, ctx, json2, params); - const bag = inst._zod.bag; - inst.format = bag.format ?? null; - inst.minLength = bag.minimum ?? null; - inst.maxLength = bag.maximum ?? null; -}, { - regex(...args) { - return this.check(_regex(...args)); - }, - includes(...args) { - return this.check(_includes(...args)); - }, - startsWith(...args) { - return this.check(_startsWith(...args)); - }, - endsWith(...args) { - return this.check(_endsWith(...args)); - }, - min(...args) { - return this.check(_minLength(...args)); - }, - max(...args) { - return this.check(_maxLength(...args)); - }, - length(...args) { - return this.check(_length(...args)); - }, - nonempty(...args) { - return this.check(_minLength(1, ...args)); - }, - lowercase(params) { - return this.check(_lowercase(params)); +var _ZodString = /* @__PURE__ */ $constructor( + "_ZodString", + (inst, def) => { + $ZodString.init(inst, def); + ZodType.init(inst, def); + inst._zod.processJSONSchema = (ctx, json2, params) => stringProcessor(inst, ctx, json2, params); }, - uppercase(params) { - return this.check(_uppercase(params)); - }, - trim() { - return this.check(_trim()); - }, - normalize(...args) { - return this.check(_normalize(...args)); - }, - toLowerCase() { - return this.check(_toLowerCase()); - }, - toUpperCase() { - return this.check(_toUpperCase()); - }, - slugify() { - return this.check(_slugify()); - } -}); + /* @__PURE__ */ util_exports.derived({ + format: (inst) => aggregateChecks(inst).format ?? null, + minLength: (inst) => aggregateChecks(inst).minimum ?? null, + maxLength: (inst) => aggregateChecks(inst).maximum ?? null + }, { + regex(...args) { + return this.check(_regex(...args)); + }, + includes(...args) { + return this.check(_includes(...args)); + }, + startsWith(...args) { + return this.check(_startsWith(...args)); + }, + endsWith(...args) { + return this.check(_endsWith(...args)); + }, + min(...args) { + return this.check(_minLength(...args)); + }, + max(...args) { + return this.check(_maxLength(...args)); + }, + length(...args) { + return this.check(_length(...args)); + }, + nonempty(...args) { + return this.check(_minLength(1, ...args)); + }, + lowercase(params) { + return this.check(_lowercase(params)); + }, + uppercase(params) { + return this.check(_uppercase(params)); + }, + trim() { + return this.check(_trim()); + }, + normalize(...args) { + return this.check(_normalize(...args)); + }, + toLowerCase() { + return this.check(_toLowerCase()); + }, + toUpperCase() { + return this.check(_toUpperCase()); + }, + slugify() { + return this.check(_slugify()); + } + }) +); var ZodString = /* @__PURE__ */ $constructor("ZodString", (inst, def) => { $ZodString.init(inst, def); _ZodString.init(inst, def); @@ -24432,8 +25103,8 @@ function url(params) { } function httpUrl(params) { return _url(ZodURL, { - protocol: regexes_exports.httpProtocol, - hostname: regexes_exports.domain, + protocol: httpProtocol, + hostname: domain, ...util_exports.normalizeParams(params) }); } @@ -24549,6 +25220,13 @@ var ZodCreditCard = /* @__PURE__ */ $constructor("ZodCreditCard", (inst, def) => function creditCard2(params) { return _creditCard(ZodCreditCard, params); } +var ZodIBAN = /* @__PURE__ */ $constructor("ZodIBAN", (inst, def) => { + $ZodIBAN.init(inst, def); + ZodStringFormat.init(inst, def); +}); +function iban2(params) { + return _iban(ZodIBAN, params); +} var ZodJWT = /* @__PURE__ */ $constructor("ZodJWT", (inst, def) => { $ZodJWT.init(inst, def); ZodStringFormat.init(inst, def); @@ -24564,10 +25242,13 @@ function stringFormat(format, fnOrRegex, _params = {}) { return _stringFormat(ZodCustomStringFormat, format, fnOrRegex, _params); } function hostname2(_params) { - return _stringFormat(ZodCustomStringFormat, "hostname", regexes_exports.hostname, _params); + return _stringFormat(ZodCustomStringFormat, "hostname", hostname, _params); } function hex2(_params) { - return _stringFormat(ZodCustomStringFormat, "hex", regexes_exports.hex, _params); + return _stringFormat(ZodCustomStringFormat, "hex", hex, _params); +} +function currencyCode2(_params) { + return _stringFormat(ZodCustomStringFormat, "currency_code", currencyCode, _params); } function hash(alg, params) { const enc = params?.enc ?? "hex"; @@ -24577,63 +25258,76 @@ function hash(alg, params) { throw new Error(`Unrecognized hash format: ${format}`); return _stringFormat(ZodCustomStringFormat, format, regex, params); } -var ZodNumber = /* @__PURE__ */ $constructor("ZodNumber", (inst, def) => { - $ZodNumber.init(inst, def); - ZodType.init(inst, def); - inst._zod.processJSONSchema = (ctx, json2, params) => numberProcessor(inst, ctx, json2, params); - const bag = inst._zod.bag; - inst.minValue = Math.max(bag.minimum ?? Number.NEGATIVE_INFINITY, bag.exclusiveMinimum ?? Number.NEGATIVE_INFINITY) ?? null; - inst.maxValue = Math.min(bag.maximum ?? Number.POSITIVE_INFINITY, bag.exclusiveMaximum ?? Number.POSITIVE_INFINITY) ?? null; - inst.isInt = (bag.format ?? "").includes("int") || Number.isSafeInteger(bag.multipleOf ?? 0.5); - inst.isFinite = true; - inst.format = bag.format ?? null; -}, { - gt(value, params) { - return this.check(_gt(value, params)); - }, - gte(value, params) { - return this.check(_gte(value, params)); +var ZodNumber = /* @__PURE__ */ $constructor( + "ZodNumber", + (inst, def) => { + $ZodNumber.init(inst, def); + ZodType.init(inst, def); + inst._zod.processJSONSchema = (ctx, json2, params) => numberProcessor(inst, ctx, json2, params); + inst.isFinite = true; }, - min(value, params) { - return this.check(_gte(value, params)); - }, - lt(value, params) { - return this.check(_lt(value, params)); - }, - lte(value, params) { - return this.check(_lte(value, params)); - }, - max(value, params) { - return this.check(_lte(value, params)); - }, - int(params) { - return this.check(int(params)); - }, - safe(params) { - return this.check(int(params)); - }, - positive(params) { - return this.check(_gt(0, params)); - }, - nonnegative(params) { - return this.check(_gte(0, params)); - }, - negative(params) { - return this.check(_lt(0, params)); - }, - nonpositive(params) { - return this.check(_lte(0, params)); - }, - multipleOf(value, params) { - return this.check(_multipleOf(value, params)); - }, - step(value, params) { - return this.check(_multipleOf(value, params)); - }, - finite() { - return this; - } -}); + /* @__PURE__ */ util_exports.derived({ + minValue: (inst) => { + const { minimum, exclusiveMinimum } = aggregateChecks(inst); + return Math.max(minimum ?? Number.NEGATIVE_INFINITY, exclusiveMinimum ?? Number.NEGATIVE_INFINITY); + }, + maxValue: (inst) => { + const { maximum, exclusiveMaximum } = aggregateChecks(inst); + return Math.min(maximum ?? Number.POSITIVE_INFINITY, exclusiveMaximum ?? Number.POSITIVE_INFINITY); + }, + isInt: (inst) => { + const { isInt, multipleOf } = aggregateChecks(inst); + return !!isInt || !!multipleOf?.some(Number.isSafeInteger); + }, + format: (inst) => aggregateChecks(inst).format ?? null + }, { + gt(value, params) { + return this.check(_gt(value, params)); + }, + gte(value, params) { + return this.check(_gte(value, params)); + }, + min(value, params) { + return this.check(_gte(value, params)); + }, + lt(value, params) { + return this.check(_lt(value, params)); + }, + lte(value, params) { + return this.check(_lte(value, params)); + }, + max(value, params) { + return this.check(_lte(value, params)); + }, + int(params) { + return this.check(int(params)); + }, + safe(params) { + return this.check(int(params)); + }, + positive(params) { + return this.check(_gt(0, params)); + }, + nonnegative(params) { + return this.check(_gte(0, params)); + }, + negative(params) { + return this.check(_lt(0, params)); + }, + nonpositive(params) { + return this.check(_lte(0, params)); + }, + multipleOf(value, params) { + return this.check(_multipleOf(value, params)); + }, + step(value, params) { + return this.check(_multipleOf(value, params)); + }, + finite() { + return this; + } + }) +); function number2(params) { return _number(ZodNumber, params); } @@ -24664,49 +25358,53 @@ var ZodBoolean = /* @__PURE__ */ $constructor("ZodBoolean", (inst, def) => { function boolean2(params) { return _boolean(ZodBoolean, params); } -var ZodBigInt = /* @__PURE__ */ $constructor("ZodBigInt", (inst, def) => { - $ZodBigInt.init(inst, def); - ZodType.init(inst, def); - inst._zod.processJSONSchema = (ctx, json2, params) => bigintProcessor(inst, ctx, json2, params); - const bag = inst._zod.bag; - inst.minValue = bag.minimum ?? null; - inst.maxValue = bag.maximum ?? null; - inst.format = bag.format ?? null; -}, { - gte(value, params) { - return this.check(_gte(value, params)); - }, - min(value, params) { - return this.check(_gte(value, params)); - }, - gt(value, params) { - return this.check(_gt(value, params)); - }, - lt(value, params) { - return this.check(_lt(value, params)); - }, - lte(value, params) { - return this.check(_lte(value, params)); - }, - max(value, params) { - return this.check(_lte(value, params)); +var ZodBigInt = /* @__PURE__ */ $constructor( + "ZodBigInt", + (inst, def) => { + $ZodBigInt.init(inst, def); + ZodType.init(inst, def); + inst._zod.processJSONSchema = (ctx, json2, params) => bigintProcessor(inst, ctx, json2, params); }, - positive(params) { - return this.check(_gt(BigInt(0), params)); - }, - negative(params) { - return this.check(_lt(BigInt(0), params)); - }, - nonpositive(params) { - return this.check(_lte(BigInt(0), params)); - }, - nonnegative(params) { - return this.check(_gte(BigInt(0), params)); - }, - multipleOf(value, params) { - return this.check(_multipleOf(value, params)); - } -}); + /* @__PURE__ */ util_exports.derived({ + minValue: (inst) => aggregateChecks(inst).minimum ?? null, + maxValue: (inst) => aggregateChecks(inst).maximum ?? null, + format: (inst) => aggregateChecks(inst).format ?? null + }, { + gte(value, params) { + return this.check(_gte(value, params)); + }, + min(value, params) { + return this.check(_gte(value, params)); + }, + gt(value, params) { + return this.check(_gt(value, params)); + }, + lt(value, params) { + return this.check(_lt(value, params)); + }, + lte(value, params) { + return this.check(_lte(value, params)); + }, + max(value, params) { + return this.check(_lte(value, params)); + }, + positive(params) { + return this.check(_gt(BigInt(0), params)); + }, + negative(params) { + return this.check(_lt(BigInt(0), params)); + }, + nonpositive(params) { + return this.check(_lte(BigInt(0), params)); + }, + nonnegative(params) { + return this.check(_gte(BigInt(0), params)); + }, + multipleOf(value, params) { + return this.check(_multipleOf(value, params)); + } + }) +); function bigint2(params) { return _bigint(ZodBigInt, params); } @@ -24776,16 +25474,26 @@ var ZodVoid = /* @__PURE__ */ $constructor("ZodVoid", (inst, def) => { function _void2(params) { return _void(ZodVoid, params); } -var ZodDate = /* @__PURE__ */ $constructor("ZodDate", (inst, def) => { - $ZodDate.init(inst, def); - ZodType.init(inst, def); - inst._zod.processJSONSchema = (ctx, json2, params) => dateProcessor(inst, ctx, json2, params); - inst.min = (value, params) => inst.check(_gte(value, params)); - inst.max = (value, params) => inst.check(_lte(value, params)); - const c = inst._zod.bag; - inst.minDate = c.minimum ? new Date(c.minimum) : null; - inst.maxDate = c.maximum ? new Date(c.maximum) : null; -}); +var ZodDate = /* @__PURE__ */ $constructor( + "ZodDate", + (inst, def) => { + $ZodDate.init(inst, def); + ZodType.init(inst, def); + inst._zod.processJSONSchema = (ctx, json2, params) => dateProcessor(inst, ctx, json2, params); + inst.min = (value, params) => inst.check(_gte(value, params)); + inst.max = (value, params) => inst.check(_lte(value, params)); + }, + /* @__PURE__ */ util_exports.derived({ + minDate: (inst) => { + const { minimum } = aggregateChecks(inst); + return minimum ? new Date(minimum) : null; + }, + maxDate: (inst) => { + const { maximum } = aggregateChecks(inst); + return maximum ? new Date(maximum) : null; + } + }, {}) +); function date2(params) { return _date(ZodDate, params); } @@ -24830,19 +25538,19 @@ var ZodObject = /* @__PURE__ */ $constructor("ZodObject", (inst, def) => { return _enum2(Object.keys(this._zod.def.shape)); }, catchall(catchall) { - return this.clone({ ...this._zod.def, catchall }); + return this.clone(util_exports.mergeDefs(this._zod.def, { catchall })); }, passthrough() { - return this.clone({ ...this._zod.def, catchall: unknown() }); + return this.clone(util_exports.mergeDefs(this._zod.def, { catchall: unknown() })); }, loose() { - return this.clone({ ...this._zod.def, catchall: unknown() }); + return this.clone(util_exports.mergeDefs(this._zod.def, { catchall: unknown() })); }, strict() { - return this.clone({ ...this._zod.def, catchall: never() }); + return this.clone(util_exports.mergeDefs(this._zod.def, { catchall: never() })); }, strip() { - return this.clone({ ...this._zod.def, catchall: void 0 }); + return this.clone(util_exports.mergeDefs(this._zod.def, { catchall: void 0 })); }, extend(incoming) { return util_exports.extend(this, incoming); @@ -25061,12 +25769,12 @@ var ZodEnum = /* @__PURE__ */ $constructor("ZodEnum", (inst, def) => { ZodType.init(inst, def); inst._zod.processJSONSchema = (ctx, json2, params) => enumProcessor(inst, ctx, json2, params); inst.enum = def.entries; - inst.options = Object.values(def.entries); - const keys = new Set(Object.keys(def.entries)); + inst.options = [...inst._zod.values]; + const keys2 = new Set(Object.keys(def.entries)); inst.extract = (values, params) => { const newEntries = {}; for (const value of values) { - if (keys.has(value)) { + if (keys2.has(value)) { newEntries[value] = def.entries[value]; } else throw new Error(`Key ${value} not found in enum`); @@ -25081,7 +25789,7 @@ var ZodEnum = /* @__PURE__ */ $constructor("ZodEnum", (inst, def) => { inst.exclude = (values, params) => { const newEntries = { ...def.entries }; for (const value of values) { - if (keys.has(value)) { + if (keys2.has(value)) { delete newEntries[value]; } else throw new Error(`Key ${value} not found in enum`); @@ -25424,8 +26132,15 @@ function superRefine(fn, params) { } var describe2 = describe; var meta2 = meta; +var ZodInstanceOf = /* @__PURE__ */ $constructor("ZodInstanceOf", (inst, def) => { + ZodCustom.init(inst, def); +}, { + properties(shape, params) { + return this.check(_properties(shape, params)); + } +}); function _instanceof(cls, params = {}) { - const inst = new ZodCustom({ + const inst = new ZodInstanceOf({ type: "custom", check: "custom", fn: (data) => data instanceof cls, @@ -25616,13 +26331,13 @@ function resolveRef(ref, ctx) { if (!ref.startsWith("#")) { throw new Error("External $ref is not supported, only local refs (#/...) are allowed"); } - const path3 = ref.slice(1).split("/").filter(Boolean); - if (path3.length === 0) { + const path4 = ref.slice(1).split("/").filter(Boolean); + if (path4.length === 0) { return ctx.rootSchema; } const defsKey = ctx.version === "draft-2020-12" ? "$defs" : "definitions"; - if (path3[0] === defsKey) { - const key = path3[1] === void 0 ? void 0 : decodeJSONPointerSegment(path3[1]); + if (path4[0] === defsKey) { + const key = path4[1] === void 0 ? void 0 : decodeJSONPointerSegment(path4[1]); if (!key || !ctx.defs[key]) { throw new Error(`Reference not found: ${ref}`); } @@ -25630,27 +26345,188 @@ function resolveRef(ref, ctx) { } throw new Error(`Reference not found: ${ref}`); } -function checkPropertyNames(objectSchema, keySchema) { +function checkObjectGuards(objectSchema, guards) { const guard = z.transform((value) => value).check((payload) => { const value = payload.value; if (typeof value !== "object" || value === null || Array.isArray(value)) return; - for (const key of Object.getOwnPropertyNames(value)) { - const result = keySchema.safeParse(key); - if (result.success) - continue; + const keys2 = Object.getOwnPropertyNames(value); + if (guards.minProperties !== void 0 && keys2.length < guards.minProperties) { payload.issues.push({ - code: "invalid_key", - origin: "record", - issues: result.error.issues, - input: key, - path: [key], + origin: "object", + code: "too_small", + minimum: guards.minProperties, + inclusive: true, + message: `Too small: expected object to have >=${guards.minProperties} properties`, + input: value, + inst: objectSchema, continue: true }); } + if (guards.maxProperties !== void 0 && keys2.length > guards.maxProperties) { + payload.issues.push({ + origin: "object", + code: "too_big", + maximum: guards.maxProperties, + inclusive: true, + message: `Too big: expected object to have <=${guards.maxProperties} properties`, + input: value, + inst: objectSchema, + continue: true + }); + } + if (guards.keySchema) { + for (const key of keys2) { + const result = guards.keySchema.safeParse(key); + if (result.success) + continue; + payload.issues.push({ + code: "invalid_key", + origin: "record", + issues: result.error.issues, + input: key, + path: [key], + continue: true + }); + } + } }); return guard.pipe(objectSchema); } +function canonicalKey(value, seen) { + if (value === null) + return "z"; + const type = typeof value; + if (type !== "object") { + if (type === "number" && Number.isNaN(value)) + return null; + const raw = String(value); + return `${type[0]}${raw.length}:${raw}`; + } + if (seen.has(value)) + return null; + seen.add(value); + try { + if (Array.isArray(value)) { + const parts2 = []; + for (const item of value) { + const key = canonicalKey(item, seen); + if (key === null) + return null; + parts2.push(key); + } + return `a${parts2.length}:[${parts2.join(",")}]`; + } + const keys2 = Object.keys(value).sort(); + const parts = []; + for (const k2 of keys2) { + const key = canonicalKey(value[k2], seen); + if (key === null) + return null; + parts.push(`${k2.length}:${k2}=${key}`); + } + return `o${parts.length}:{${parts.join(",")}}`; + } finally { + seen.delete(value); + } +} +var SCHEMA_KEYWORDS = /* @__PURE__ */ new Set([ + "items", + "prefixItems", + "additionalItems", + "additionalProperties", + "contains", + "propertyNames", + "not", + "if", + "then", + "else", + "allOf", + "anyOf", + "oneOf", + "unevaluatedItems", + "unevaluatedProperties", + "contentSchema" +]); +var SCHEMA_MAP_KEYWORDS = /* @__PURE__ */ new Set([ + "properties", + "patternProperties", + "dependentSchemas", + "dependencies", + "$defs", + "definitions" +]); +function containsRef(value) { + if (typeof value !== "object" || value === null) + return false; + if (Array.isArray(value)) + return value.some(containsRef); + if (typeof value.$ref === "string") + return true; + return Object.entries(value).some(([key, sub]) => { + if (SCHEMA_KEYWORDS.has(key)) + return containsRef(sub); + if (!SCHEMA_MAP_KEYWORDS.has(key) || typeof sub !== "object" || sub === null) + return false; + return Object.values(sub).some(containsRef); + }); +} +function plural(n) { + return n === 1 ? "element" : "elements"; +} +function checkArrayGuards(arraySchema, guards) { + const guard = z.transform((value) => value).check((payload) => { + const items = payload.value; + if (!Array.isArray(items)) + return; + if (guards.uniqueItems === true) { + const firstSeen = /* @__PURE__ */ new Map(); + for (let i = 0; i < items.length; i++) { + const key = canonicalKey(items[i], /* @__PURE__ */ new Set()); + if (key === null) + continue; + const first = firstSeen.get(key); + if (first === void 0) { + firstSeen.set(key, i); + continue; + } + payload.issues.push({ + code: "custom", + message: `Array items must be unique: element at index ${i} duplicates the one at index ${first}`, + input: items, + path: [i], + continue: true + }); + } + } + if (guards.containsSchema) { + const minContains = guards.minContains ?? 1; + const ceiling = guards.maxContains !== void 0 ? guards.maxContains + 1 : Number.POSITIVE_INFINITY; + let matches = 0; + for (const item of items) { + if (guards.containsSchema.safeParse(item).success && ++matches >= ceiling) + break; + } + if (matches < minContains) { + payload.issues.push({ + code: "custom", + message: `Array must contain at least ${minContains} matching ${plural(minContains)}; found ${matches}`, + input: items, + continue: true + }); + } + if (guards.maxContains !== void 0 && matches > guards.maxContains) { + payload.issues.push({ + code: "custom", + message: `Array must contain at most ${guards.maxContains} matching ${plural(guards.maxContains)}`, + input: items, + continue: true + }); + } + } + }); + return guard.pipe(arraySchema); +} function getTupleRest(restSchema, ctx) { if (restSchema === false) { return void 0; @@ -25780,6 +26656,8 @@ function convertBaseSchema(schema, ctx) { stringSchema = stringSchema.check(z.e164()); } else if (format === "credit_card") { stringSchema = stringSchema.check(z.creditCard()); + } else if (format === "iban") { + stringSchema = stringSchema.check(z.iban()); } else if (format === "jwt") { stringSchema = stringSchema.check(z.jwt()); } else if (format === "emoji") { @@ -25912,9 +26790,16 @@ function convertBaseSchema(schema, ctx) { zodSchema = objectSchema.passthrough(); } } - if (schema.propertyNames !== void 0 && schema.propertyNames !== true) { - const keyJSONSchema = typeof schema.propertyNames === "object" && schema.propertyNames.type === void 0 ? { type: "string", ...schema.propertyNames } : schema.propertyNames; - zodSchema = checkPropertyNames(zodSchema, convertSchema(keyJSONSchema, ctx)); + const hasKeyGuard = schema.propertyNames !== void 0 && schema.propertyNames !== true; + const minProperties = typeof schema.minProperties === "number" ? schema.minProperties : void 0; + const maxProperties = typeof schema.maxProperties === "number" ? schema.maxProperties : void 0; + if (hasKeyGuard || minProperties !== void 0 || maxProperties !== void 0) { + let keySchema; + if (hasKeyGuard) { + const keyJSONSchema = typeof schema.propertyNames === "object" && schema.propertyNames.type === void 0 ? { type: "string", ...schema.propertyNames } : schema.propertyNames; + keySchema = convertSchema(keyJSONSchema, ctx); + } + zodSchema = checkObjectGuards(zodSchema, { keySchema, minProperties, maxProperties }); } break; } @@ -25960,6 +26845,14 @@ function convertBaseSchema(schema, ctx) { } else { zodSchema = z.array(z.any()); } + if (schema.uniqueItems === true || schema.contains !== void 0) { + zodSchema = checkArrayGuards(zodSchema, { + uniqueItems: schema.uniqueItems === true, + containsSchema: schema.contains !== void 0 ? convertSchema(schema.contains, ctx) : void 0, + minContains: typeof schema.minContains === "number" ? schema.minContains : void 0, + maxContains: typeof schema.maxContains === "number" ? schema.maxContains : void 0 + }); + } break; } default: @@ -26017,8 +26910,23 @@ function convertSchema(schema, ctx) { extraMeta[key] = schema[key]; } } - if (schema.propertyNames !== void 0 && schema.type === "object" && schema.$ref === void 0) { - extraMeta.propertyNames = schema.propertyNames; + if (schema.type === "object" && schema.$ref === void 0) { + if (schema.propertyNames !== void 0 && !containsRef(schema.propertyNames)) { + extraMeta.propertyNames = schema.propertyNames; + } + for (const key of ["minProperties", "maxProperties"]) { + if (schema[key] !== void 0) + extraMeta[key] = schema[key]; + } + } + if (schema.type === "array" && schema.$ref === void 0) { + if (schema.contains !== void 0 && !containsRef(schema.contains)) { + extraMeta.contains = schema.contains; + } + for (const key of ["uniqueItems", "minContains", "maxContains"]) { + if (schema[key] !== void 0) + extraMeta[key] = schema[key]; + } } for (const key of Object.keys(schema)) { if (!RECOGNIZED_KEYS.has(key)) { @@ -27715,6 +28623,9 @@ var Protocol = class { this.setRequestHandler(GetTaskPayloadRequestSchema, async (request, extra) => { const handleTaskResult = async () => { const taskId = request.params.taskId; + if (!await this._taskStore.getTask(taskId, extra.sessionId)) { + throw new McpError(ErrorCode.InvalidParams, `Task not found: ${taskId}`); + } if (this._taskMessageQueue) { let queuedMessage; while (queuedMessage = await this._taskMessageQueue.dequeue(taskId, extra.sessionId)) { @@ -27728,8 +28639,8 @@ var Protocol = class { resolver(message); } else { const errorMessage = message; - const error61 = new McpError(errorMessage.error.code, errorMessage.error.message, errorMessage.error.data); - resolver(error61); + const error62 = new McpError(errorMessage.error.code, errorMessage.error.message, errorMessage.error.data); + resolver(error62); } } else { const messageType = queuedMessage.type === "response" ? "Response" : "Error"; @@ -27745,12 +28656,12 @@ var Protocol = class { throw new McpError(ErrorCode.InvalidParams, `Task not found: ${taskId}`); } if (!isTerminal(task.status)) { - await this._waitForTaskUpdate(taskId, extra.signal); + await this._waitForTaskUpdate(taskId, extra.signal, extra.sessionId); return await handleTaskResult(); } if (isTerminal(task.status)) { const result = await this._taskStore.getTaskResult(taskId, extra.sessionId); - this._clearTaskQueue(taskId); + this._clearTaskQueue(taskId, extra.sessionId); return { ...result, _meta: { @@ -27773,8 +28684,8 @@ var Protocol = class { nextCursor, _meta: {} }; - } catch (error61) { - throw new McpError(ErrorCode.InvalidParams, `Failed to list tasks: ${error61 instanceof Error ? error61.message : String(error61)}`); + } catch (error62) { + throw new McpError(ErrorCode.InvalidParams, `Failed to list tasks: ${error62 instanceof Error ? error62.message : String(error62)}`); } }); this.setRequestHandler(CancelTaskRequestSchema, async (request, extra) => { @@ -27787,7 +28698,7 @@ var Protocol = class { throw new McpError(ErrorCode.InvalidParams, `Cannot cancel task in terminal status: ${task.status}`); } await this._taskStore.updateTaskStatus(request.params.taskId, "cancelled", "Client cancelled task execution.", extra.sessionId); - this._clearTaskQueue(request.params.taskId); + this._clearTaskQueue(request.params.taskId, extra.sessionId); const cancelledTask = await this._taskStore.getTask(request.params.taskId, extra.sessionId); if (!cancelledTask) { throw new McpError(ErrorCode.InvalidParams, `Task not found after cancellation: ${request.params.taskId}`); @@ -27796,11 +28707,11 @@ var Protocol = class { _meta: {}, ...cancelledTask }; - } catch (error61) { - if (error61 instanceof McpError) { - throw error61; + } catch (error62) { + if (error62 instanceof McpError) { + throw error62; } - throw new McpError(ErrorCode.InvalidRequest, `Failed to cancel task: ${error61 instanceof Error ? error61.message : String(error61)}`); + throw new McpError(ErrorCode.InvalidRequest, `Failed to cancel task: ${error62 instanceof Error ? error62.message : String(error62)}`); } }); } @@ -27861,9 +28772,9 @@ var Protocol = class { this._onclose(); }; const _onerror = this.transport?.onerror; - this._transport.onerror = (error61) => { - _onerror?.(error61); - this._onerror(error61); + this._transport.onerror = (error62) => { + _onerror?.(error62); + this._onerror(error62); }; const _onmessage = this._transport?.onmessage; this._transport.onmessage = (message, extra) => { @@ -27894,27 +28805,40 @@ var Protocol = class { controller.abort(); } this._requestHandlerAbortControllers.clear(); - const error61 = McpError.fromError(ErrorCode.ConnectionClosed, "Connection closed"); + const error62 = McpError.fromError(ErrorCode.ConnectionClosed, "Connection closed"); this._transport = void 0; this.onclose?.(); for (const handler of responseHandlers.values()) { - handler(error61); + handler(error62); } } - _onerror(error61) { - this.onerror?.(error61); + _onerror(error62) { + this.onerror?.(error62); } _onnotification(notification) { const handler = this._notificationHandlers.get(notification.method) ?? this.fallbackNotificationHandler; if (handler === void 0) { return; } - Promise.resolve().then(() => handler(notification)).catch((error61) => this._onerror(new Error(`Uncaught error in notification handler: ${error61}`))); + Promise.resolve().then(() => handler(notification)).catch((error62) => this._onerror(new Error(`Uncaught error in notification handler: ${error62}`))); } _onrequest(request, extra) { const handler = this._requestHandlers.get(request.method) ?? this.fallbackRequestHandler; const capturedTransport = this._transport; const relatedTaskId = request.params?._meta?.[RELATED_TASK_META_KEY]?.taskId; + const sessionId = capturedTransport?.sessionId; + const store = this._taskStore; + let relatedTaskFound = true; + let relatedTaskLookup; + if (relatedTaskId && store && this._taskMessageQueue && sessionId !== void 0) { + relatedTaskFound = false; + relatedTaskLookup = (async () => { + if (!await store.getTask(relatedTaskId, sessionId)) { + throw new McpError(ErrorCode.InvalidParams, `Task not found: ${relatedTaskId}`); + } + relatedTaskFound = true; + })(); + } if (handler === void 0) { const errorResponse = { jsonrpc: "2.0", @@ -27924,14 +28848,17 @@ var Protocol = class { message: "Method not found" } }; - if (relatedTaskId && this._taskMessageQueue) { + if (relatedTaskId && relatedTaskLookup) { + const queuedError = { type: "error", message: errorResponse, timestamp: Date.now() }; + relatedTaskLookup.then(() => this._enqueueTaskMessage(relatedTaskId, queuedError, sessionId), () => capturedTransport?.send(errorResponse)).catch((error62) => this._onerror(new Error(`Failed to send an error response: ${error62}`))); + } else if (relatedTaskId && this._taskMessageQueue) { this._enqueueTaskMessage(relatedTaskId, { type: "error", message: errorResponse, timestamp: Date.now() - }, capturedTransport?.sessionId).catch((error61) => this._onerror(new Error(`Failed to enqueue error response: ${error61}`))); + }, capturedTransport?.sessionId).catch((error62) => this._onerror(new Error(`Failed to enqueue error response: ${error62}`))); } else { - capturedTransport?.send(errorResponse).catch((error61) => this._onerror(new Error(`Failed to send an error response: ${error61}`))); + capturedTransport?.send(errorResponse).catch((error62) => this._onerror(new Error(`Failed to send an error response: ${error62}`))); } return; } @@ -27975,7 +28902,10 @@ var Protocol = class { closeSSEStream: extra?.closeSSEStream, closeStandaloneSSEStream: extra?.closeStandaloneSSEStream }; - Promise.resolve().then(() => { + (relatedTaskLookup ?? Promise.resolve()).then(() => { + if (relatedTaskLookup && abortController.signal.aborted) { + throw new McpError(ErrorCode.ConnectionClosed, "Request was cancelled"); + } if (taskCreationParams) { this.assertTaskHandlerCapability(request.method); } @@ -27997,7 +28927,7 @@ var Protocol = class { } else { await capturedTransport?.send(response); } - }, async (error61) => { + }, async (error62) => { if (abortController.signal.aborted) { return; } @@ -28005,12 +28935,12 @@ var Protocol = class { jsonrpc: "2.0", id: request.id, error: { - code: Number.isSafeInteger(error61["code"]) ? error61["code"] : ErrorCode.InternalError, - message: error61.message ?? "Internal error", - ...error61["data"] !== void 0 && { data: error61["data"] } + code: Number.isSafeInteger(error62["code"]) ? error62["code"] : ErrorCode.InternalError, + message: error62.message ?? "Internal error", + ...error62["data"] !== void 0 && { data: error62["data"] } } }; - if (relatedTaskId && this._taskMessageQueue) { + if (relatedTaskId && this._taskMessageQueue && relatedTaskFound) { await this._enqueueTaskMessage(relatedTaskId, { type: "error", message: errorResponse, @@ -28019,7 +28949,7 @@ var Protocol = class { } else { await capturedTransport?.send(errorResponse); } - }).catch((error61) => this._onerror(new Error(`Failed to send response: ${error61}`))).finally(() => { + }).catch((error62) => this._onerror(new Error(`Failed to send response: ${error62}`))).finally(() => { if (this._requestHandlerAbortControllers.get(request.id) === abortController) { this._requestHandlerAbortControllers.delete(request.id); } @@ -28038,11 +28968,11 @@ var Protocol = class { if (timeoutInfo && responseHandler && timeoutInfo.resetTimeoutOnProgress) { try { this._resetTimeout(messageId); - } catch (error61) { + } catch (error62) { this._responseHandlers.delete(messageId); this._progressHandlers.delete(messageId); this._cleanupTimeout(messageId); - responseHandler(error61); + responseHandler(error62); return; } } @@ -28056,8 +28986,8 @@ var Protocol = class { if (isJSONRPCResultResponse(response)) { resolver(response); } else { - const error61 = new McpError(response.error.code, response.error.message, response.error.data); - resolver(error61); + const error62 = new McpError(response.error.code, response.error.message, response.error.data); + resolver(error62); } return; } @@ -28085,8 +29015,8 @@ var Protocol = class { if (isJSONRPCResultResponse(response)) { handler(response); } else { - const error61 = McpError.fromError(response.error.code, response.error.message, response.error.data); - handler(error61); + const error62 = McpError.fromError(response.error.code, response.error.message, response.error.data); + handler(error62); } } get transport() { @@ -28131,10 +29061,10 @@ var Protocol = class { try { const result = await this.request(request, resultSchema, options); yield { type: "result", result }; - } catch (error61) { + } catch (error62) { yield { type: "error", - error: error61 instanceof McpError ? error61 : new McpError(ErrorCode.InternalError, String(error61)) + error: error62 instanceof McpError ? error62 : new McpError(ErrorCode.InternalError, String(error62)) }; } return; @@ -28177,10 +29107,10 @@ var Protocol = class { await new Promise((resolve) => setTimeout(resolve, pollInterval)); options?.signal?.throwIfAborted(); } - } catch (error61) { + } catch (error62) { yield { type: "error", - error: error61 instanceof McpError ? error61 : new McpError(ErrorCode.InternalError, String(error61)) + error: error62 instanceof McpError ? error62 : new McpError(ErrorCode.InternalError, String(error62)) }; } } @@ -28192,8 +29122,8 @@ var Protocol = class { request(request, resultSchema, options) { const { relatedRequestId, resumptionToken, onresumptiontoken, task, relatedTask } = options ?? {}; return new Promise((resolve, reject) => { - const earlyReject = (error61) => { - reject(error61); + const earlyReject = (error62) => { + reject(error62); }; if (!this._transport) { earlyReject(new Error("Not connected")); @@ -28253,9 +29183,9 @@ var Protocol = class { requestId: messageId, reason: String(reason) } - }, { relatedRequestId, resumptionToken, onresumptiontoken }).catch((error62) => this._onerror(new Error(`Failed to send cancellation: ${error62}`))); - const error61 = reason instanceof McpError ? reason : new McpError(ErrorCode.RequestTimeout, String(reason)); - reject(error61); + }, { relatedRequestId, resumptionToken, onresumptiontoken }).catch((error63) => this._onerror(new Error(`Failed to send cancellation: ${error63}`))); + const error62 = reason instanceof McpError ? reason : new McpError(ErrorCode.RequestTimeout, String(reason)); + reject(error62); }; this._responseHandlers.set(messageId, (response) => { if (options?.signal?.aborted) { @@ -28271,8 +29201,8 @@ var Protocol = class { } else { resolve(parseResult.data); } - } catch (error61) { - reject(error61); + } catch (error62) { + reject(error62); } }); options?.signal?.addEventListener("abort", () => { @@ -28296,14 +29226,14 @@ var Protocol = class { type: "request", message: jsonrpcRequest, timestamp: Date.now() - }).catch((error61) => { + }).catch((error62) => { this._cleanupTimeout(messageId); - reject(error61); + reject(error62); }); } else { - this._transport.send(jsonrpcRequest, { relatedRequestId, resumptionToken, onresumptiontoken }).catch((error61) => { + this._transport.send(jsonrpcRequest, { relatedRequestId, resumptionToken, onresumptiontoken }).catch((error62) => { this._cleanupTimeout(messageId); - reject(error61); + reject(error62); }); } }); @@ -28396,7 +29326,7 @@ var Protocol = class { } }; } - this._transport?.send(jsonrpcNotification2, options).catch((error61) => this._onerror(error61)); + this._transport?.send(jsonrpcNotification2, options).catch((error62) => this._onerror(error62)); }); return; } @@ -28490,7 +29420,7 @@ var Protocol = class { throw new Error("Cannot enqueue task message: taskStore and taskMessageQueue are not configured"); } const maxQueueSize = this._options?.maxTaskQueueSize; - await this._taskMessageQueue.enqueue(taskId, message, sessionId, maxQueueSize); + await this._taskMessageQueue.enqueue(taskId, message, sessionId ?? this._transport?.sessionId, maxQueueSize); } /** * Clears the message queue for a task and rejects any pending request resolvers. @@ -28519,12 +29449,13 @@ var Protocol = class { * Uses polling to check for updates at the task's configured poll interval. * @param taskId The task ID to wait for * @param signal Abort signal to cancel the wait + * @param sessionId Session of the request that waits, passed to the task store * @returns Promise that resolves when an update occurs or rejects if aborted */ - async _waitForTaskUpdate(taskId, signal) { + async _waitForTaskUpdate(taskId, signal, sessionId) { let interval = this._options?.defaultTaskPollInterval ?? 1e3; try { - const task = await this._taskStore?.getTask(taskId); + const task = await this._taskStore?.getTask(taskId, sessionId); if (task?.pollInterval) { interval = task.pollInterval; } @@ -29249,11 +30180,11 @@ var Server = class extends Protocol { if (!validationResult.valid) { throw new McpError(ErrorCode.InvalidParams, `Elicitation response content does not match requested schema: ${validationResult.errorMessage}`); } - } catch (error61) { - if (error61 instanceof McpError) { - throw error61; + } catch (error62) { + if (error62 instanceof McpError) { + throw error62; } - throw new McpError(ErrorCode.InternalError, `Error validating elicitation response: ${error61 instanceof Error ? error61.message : String(error61)}`); + throw new McpError(ErrorCode.InternalError, `Error validating elicitation response: ${error62 instanceof Error ? error62.message : String(error62)}`); } } return result; @@ -29316,7 +30247,7 @@ var Server = class extends Protocol { }; // node_modules/@modelcontextprotocol/sdk/dist/esm/server/stdio.js -import process3 from "node:process"; +import process2 from "node:process"; // node_modules/@modelcontextprotocol/sdk/dist/esm/shared/stdio.js var STDIO_DEFAULT_MAX_BUFFER_SIZE = 10 * 1024 * 1024; @@ -29357,7 +30288,7 @@ function serializeMessage(message) { // node_modules/@modelcontextprotocol/sdk/dist/esm/server/stdio.js var StdioServerTransport = class { - constructor(_stdin = process3.stdin, _stdout = process3.stdout, options) { + constructor(_stdin = process2.stdin, _stdout = process2.stdout, options) { this._stdin = _stdin; this._stdout = _stdout; this._started = false; @@ -29365,14 +30296,14 @@ var StdioServerTransport = class { try { this._readBuffer.append(chunk); this.processReadBuffer(); - } catch (error61) { - this.onerror?.(error61); + } catch (error62) { + this.onerror?.(error62); this.close().catch(() => { }); } }; - this._onerror = (error61) => { - this.onerror?.(error61); + this._onerror = (error62) => { + this.onerror?.(error62); }; this._readBuffer = new ReadBuffer({ maxBufferSize: options?.maxBufferSize }); } @@ -29395,8 +30326,8 @@ var StdioServerTransport = class { break; } this.onmessage?.(message); - } catch (error61) { - this.onerror?.(error61); + } catch (error62) { + this.onerror?.(error62); } } } @@ -29424,9 +30355,16 @@ var StdioServerTransport = class { // src/db.ts import Database from "better-sqlite3"; -import path2 from "path"; -import fs2 from "fs"; -import * as sqliteVec from "sqlite-vec"; + +// src/record-admission.ts +import fs2 from "node:fs"; +import path2 from "node:path"; +import crypto from "node:crypto"; +import os2 from "node:os"; +import { spawnSync } from "node:child_process"; + +// src/constants.ts +var SUMMARIZER_CONTEXT_MARKER = "Context: This summary will be shown in a list to help users and Claude choose which conversations are relevant"; // src/paths.ts import os from "os"; @@ -29464,6 +30402,642 @@ function getDbPath() { return path.join(getIndexDir(), "db.sqlite"); } +// src/redaction-rules.ts +var SECRET_NAME = String.raw`(?:secret|password|passwd|passphrase|api[_-]?key|access[_-]?key|account[_-]?key|private[_-]?key|` + String.raw`shared[_-]?(?:access[_-]?)?key|primary[_-]?key|secondary[_-]?key|master[_-]?key|signing[_-]?key|` + String.raw`subscription[_-]?key|client[_-]?key|encryption[_-]?key|token|credentials?)\d*`; +var NOT_PLACEHOLDER = String.raw`(?!\[REDACTED:|\$\{|\$\(|\$[A-Za-z_]\w*["'<\s]|#\{|\{\{|<[^>]*>|%[A-Za-z_]\w*%|__[A-Za-z0-9_]+__|` + String.raw`(?:string|null|true|false|none|undefined|\*+)["'<])`; +var JSON_STRING_VALUE = String.raw`"${NOT_PLACEHOLDER}([^"\\]+(?:\\.[^"\\]*)*)"`; +var SAME_OBJECT = String.raw`(?:[^{}"]|"(?:[^"\\]|\\.)*"){0,400}?`; +var KEY_VAULT_ID = String.raw`"id"[ \t]*:[ \t]*"https://[^"\s]+\.vault\.(?:azure\.net|azure\.cn|usgovcloudapi\.net|microsoftazure\.de)/secrets/[^"\s]*"`; +var DEFAULT_REDACTION_CONFIG = { + rules: [ + { + id: "private-key-block", + description: "PEM/OpenSSH private key block; a truncated block (no END line) is redacted to the end of the text", + pattern: String.raw`-----BEGIN[A-Z0-9 _-]{0,40}PRIVATE KEY(?: BLOCK)?-----[\s\S]*?(?:-----END[A-Z0-9 _-]{0,40}PRIVATE KEY(?: BLOCK)?-----|$)`, + keywords: ["private key"] + }, + { + id: "connection-string-secret", + description: "Secret value inside an Azure/ADO.NET connection string; account, server and database names are kept. Keywords are case-insensitive, as in ADO.NET; a path after Pwd= is the shell PWD variable and is kept; the value needs a letter or digit, so Markdown like `Password=` is kept", + pattern: String.raw`\b(?:(?:AccountKey|SharedAccessKey|SharedAccessSignature|SharedSecret|ClientSecret|Password)=|Pwd=(?![/~]))(?![\[$<{%])(?=[^;"'\s]*[A-Za-z0-9])([^;"'\s]+)`, + flags: "i", + secretGroup: 1, + keywords: ["accountkey", "sharedaccess", "sharedsecret", "clientsecret", "password", "pwd"] + }, + { + id: "azure-sas-token", + description: "Signature of an Azure SAS token; the resource URL and other SAS parameters are kept", + pattern: String.raw`\bsig=([A-Za-z0-9%+/=_-]{16,})`, + secretGroup: 1, + keywords: ["sig="] + }, + { + id: "jwt", + description: "JSON Web Token (Entra ID / Azure access tokens, id tokens)", + pattern: String.raw`\beyJ[A-Za-z0-9_-]{8,}\.eyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}`, + keywords: ["eyj"] + }, + { + id: "anthropic-api-key", + pattern: String.raw`\bsk-ant-[a-z]{2,10}\d{2}-[A-Za-z0-9_-]{32,}`, + keywords: ["sk-ant-"] + }, + { + id: "openai-api-key", + pattern: String.raw`\bsk-(?:[a-z]+-)?[A-Za-z0-9_-]{16,}T3BlbkFJ[A-Za-z0-9_-]{16,}`, + keywords: ["t3blbkfj"] + }, + { + id: "github-token", + description: "GitHub classic, OAuth, app and fine-grained tokens", + pattern: String.raw`\b(?:gh[pousr]_[A-Za-z0-9]{36,255}|github_pat_[A-Za-z0-9_]{22,255})\b`, + keywords: ["ghp_", "gho_", "ghu_", "ghs_", "ghr_", "github_pat_"] + }, + { + id: "aws-access-key-id", + pattern: String.raw`\b(?:AKIA|ASIA|ABIA|ACCA)[A-Z2-7]{16}\b`, + keywords: ["akia", "asia", "abia", "acca"] + }, + { + id: "aws-secret-access-key", + pattern: String.raw`aws[_-]?secret[_-]?access[_-]?key["']?[ \t]*[:=][ \t]*["']?([A-Za-z0-9/+]{40})(?![A-Za-z0-9/+])`, + flags: "i", + secretGroup: 1, + keywords: ["secret"] + }, + { + id: "slack-token", + pattern: String.raw`\bxox[abposr]-[A-Za-z0-9-]{10,}`, + keywords: ["xox"] + }, + { + id: "google-api-key", + pattern: String.raw`\bAIza[0-9A-Za-z_-]{35}(?![0-9A-Za-z_-])`, + keywords: ["aiza"] + }, + { + id: "npm-token", + pattern: String.raw`\bnpm_[A-Za-z0-9]{36}\b`, + keywords: ["npm_"] + }, + { + id: "azure-client-secret", + description: 'Entra ID (Azure AD) application client secret: 3 chars, a digit, "Q~", 31-34 chars. A trailing "." may follow, so a secret ending a sentence still matches', + pattern: String.raw`(?]+:(?![\[$<{%])([^\s@/"'<>]+)@`, + secretGroup: 1, + keywords: ["://"] + }, + { + id: "bearer-token", + pattern: String.raw`\bBearer\s+([A-Za-z0-9\-._~+/]{20,}=*)`, + flags: "i", + secretGroup: 1, + keywords: ["bearer"] + }, + { + id: "basic-auth", + pattern: String.raw`\bAuthorization[ \t]*:[ \t]*Basic\s+([A-Za-z0-9+/]{8,}={0,2})`, + flags: "i", + secretGroup: 1, + keywords: ["basic"] + }, + { + id: "azure-keyvault-secret", + description: 'The "value" of a Key Vault secret bundle (az keyvault secret show/set, SDK JSON), whatever the secret is named', + pattern: KEY_VAULT_ID + String.raw`(?:[^{}]|\{[^{}]*\}){0,800}?"value"[ \t]*:[ \t]*` + JSON_STRING_VALUE + String.raw`|"value"[ \t]*:[ \t]*` + JSON_STRING_VALUE + String.raw`(?=(?:[^{}]|\{[^{}]*\}){0,800}?` + KEY_VAULT_ID + ")", + secretGroup: [1, 2], + useAllowlist: false, + keywords: [".vault."] + }, + { + id: "name-value-secret", + description: 'The "value" of a {"name"/"key": , "value": ...} object, in either order: az webapp/functionapp config appsettings list, Kubernetes env, ARM/Bicep parameters', + pattern: String.raw`"(?:name|key)"[ \t]*:[ \t]*"[^"\\]{0,80}?` + SECRET_NAME + '"' + SAME_OBJECT + String.raw`"value"[ \t]*:[ \t]*` + JSON_STRING_VALUE + String.raw`|"value"[ \t]*:[ \t]*` + JSON_STRING_VALUE + "(?=" + SAME_OBJECT + String.raw`"(?:name|key)"[ \t]*:[ \t]*"[^"\\]{0,80}?` + SECRET_NAME + '")', + flags: "i", + secretGroup: [1, 2], + useAllowlist: false, + keywords: ['"value"'] + }, + { + id: "xml-appsettings-secret", + description: 'web.config / app.config , either attribute order', + pattern: String.raw`]*?\bkey[ \t]*=[ \t]*"[^"]{0,80}?` + SECRET_NAME + String.raw`"[^>]*?\bvalue[ \t]*=[ \t]*"` + NOT_PLACEHOLDER + String.raw`([^"]+)"` + String.raw`|]*?\bvalue[ \t]*=[ \t]*"` + NOT_PLACEHOLDER + String.raw`([^"]+)"(?=[^>]*?\bkey[ \t]*=[ \t]*"[^"]{0,80}?` + SECRET_NAME + '")', + flags: "i", + secretGroup: [1, 2], + useAllowlist: false, + keywords: ["]*)?>` + NOT_PLACEHOLDER + String.raw`([^<]{1,4096})`, + flags: "i", + secretGroup: 2, + useAllowlist: false, + keywords: [", %X%, {{x}}) and code (calls, dotted member access)', + pattern: String.raw`\b[A-Za-z0-9_.-]{0,40}?(?:secret|password|passwd|passphrase|api[_-]?key|access[_-]?key|private[_-]?key|token)` + String.raw`["']?[ \t]*[:=][ \t]*["']?(?![\[$<{%])(?=[^\s"',;&\\]*\d)` + // Not a dotted identifier path (env.AUTH0_SECRET, this.config.token2): code, not a value. + String.raw`(?![A-Za-z_$][\w$]*(?:\.[A-Za-z_$][\w$]*)+(?:$|[\s"',;&\\)}\]]))` + String.raw`([^\s"',;&\\()<>{}\[\]]{8,})(?=$|[\s"',;&\\)}\]])`, + flags: "i", + secretGroup: 1, + keywords: ["secret", "passw", "passphrase", "key", "token"] + } + ], + allowlist: [ + { + id: "git-sha", + description: "Full or short git commit SHA", + pattern: String.raw`\b[0-9a-f]{7,40}\b` + }, + { + id: "guid", + description: "GUID/UUID (Entra tenant, client and object IDs, subscription IDs)", + pattern: String.raw`\b[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\b` + } + ], + entropy: { + enabled: false, + minLength: 32, + threshold: 4.5, + requireKeyword: true, + keywords: ["secret", "key", "token", "password", "passwd", "credential", "signature"], + window: 40 + }, + secretFields: { + enabled: true, + // Normalized key (lowercase letters/digits only, trailing digits dropped) + // must end with one of these. Harness keys (signature, apiKeySource, + // input_tokens, token_count) don't. + keyPattern: "secret|password|passwd|userpwd|passphrase|apikey|accesskey|accountkey|privatekey|sharedkey|primarykey|secondarykey|masterkey|signingkey|subscriptionkey|clientkey|encryptionkey|token|credentials?" + } +}; + +// src/redaction.ts +var COPY_CHUNK_BYTES = 1 << 20; + +// src/record-admission.ts +var EXCLUSION_MARKERS = [ + "DO NOT INDEX THIS CHAT", + "Only use NO_INSIGHTS_FOUND", + SUMMARIZER_CONTEXT_MARKER +]; +var MARKER_SCAN_CHUNK_BYTES = 1 << 20; +function instructionText(record2) { + if (!object3(record2)) return ""; + if (record2.type === "event_msg" && object3(record2.payload)) { + return record2.payload.type === "user_message" && typeof record2.payload.message === "string" ? record2.payload.message : ""; + } + const message = record2.type === "response_item" ? record2.payload : record2.message; + if (!object3(message) || (message.role ?? record2.role) !== "user") return ""; + const content = record2.type === "opencode_message" ? record2.parts : message.content; + if (typeof content === "string") return content; + if (!Array.isArray(content)) return ""; + return content.filter((part) => object3(part) && ["text", "input_text"].includes(String(part.type)) && typeof part.text === "string").map((part) => part.text).join("\n"); +} +function descriptorHasExclusionMarker(fd, byteLimit = Infinity, digest) { + if (!fs2.fstatSync(fd).isFile()) throw new Error("Conversation input must be a regular file"); + const buffer = Buffer.allocUnsafe(MARKER_SCAN_CHUNK_BYTES); + const decoder = new TextDecoder("utf-8", { fatal: true }); + let pending = Buffer.alloc(0); + let position = 0; + const excluded = (bytes) => { + const line = decoder.decode(bytes); + let text; + try { + text = instructionText(JSON.parse(line)); + } catch { + text = line; + } + return EXCLUSION_MARKERS.some((marker) => text.includes(marker)); + }; + let count; + while (position < byteLimit && (count = fs2.readSync(fd, buffer, 0, Math.min(buffer.length, byteLimit - position), position)) > 0) { + position += count; + digest?.update(buffer.subarray(0, count)); + pending = Buffer.concat([pending, buffer.subarray(0, count)]); + let newline; + while ((newline = pending.indexOf(10)) !== -1) { + if (newline > 64 * 1024 * 1024) throw new Error("Conversation record exceeds admission byte limit"); + if (excluded(pending.subarray(0, newline))) return true; + pending = pending.subarray(newline + 1); + } + if (pending.length > 64 * 1024 * 1024) throw new Error("Conversation record exceeds admission byte limit"); + } + if (position < byteLimit && byteLimit !== Infinity) throw new Error("Conversation changed during record admission"); + return pending.length > 0 && excluded(pending); +} +function assertUnchangedInput(file2, initial) { + const current = fs2.lstatSync(file2); + if (!current.isFile() || initial.dev !== current.dev || initial.ino !== current.ino || initial.size !== current.size || initial.mtimeMs !== current.mtimeMs || initial.ctimeMs !== current.ctimeMs) { + throw new Error("Conversation changed during record admission"); + } +} +function assertUnchangedSearchPrefix(file2, initial, expectedDigest) { + const changed = () => new Error("Conversation changed during record admission"); + let current; + try { + current = fs2.lstatSync(file2); + } catch { + throw changed(); + } + if (!current.isFile() || initial.dev !== current.dev || initial.ino !== current.ino || current.size < initial.size) { + throw changed(); + } + if (current.size === initial.size && current.mtimeMs === initial.mtimeMs && current.ctimeMs === initial.ctimeMs) return; + let fd; + try { + fd = fs2.openSync(file2, fs2.constants.O_RDONLY | fs2.constants.O_NOFOLLOW); + const opened = fs2.fstatSync(fd); + if (!opened.isFile() || opened.dev !== initial.dev || opened.ino !== initial.ino || opened.size < initial.size) throw changed(); + const digest = crypto.createHash("sha256"); + const buffer = Buffer.allocUnsafe(MARKER_SCAN_CHUNK_BYTES); + let position = 0; + while (position < initial.size) { + const count = fs2.readSync(fd, buffer, 0, Math.min(buffer.length, initial.size - position), position); + if (count <= 0) throw changed(); + digest.update(buffer.subarray(0, count)); + position += count; + } + if (digest.digest("hex") !== expectedDigest) throw changed(); + current = fs2.lstatSync(file2); + if (!current.isFile() || current.dev !== initial.dev || current.ino !== initial.ino || current.size < initial.size) throw changed(); + } catch { + throw changed(); + } finally { + if (fd !== void 0) fs2.closeSync(fd); + } +} +function assertUnchangedPolicy(expected) { + if (JSON.stringify(readRecordExclusions()) !== expected) { + throw new Error("Record exclusion policy changed during record admission"); + } +} +var POLICY_ERROR = "Invalid or unreadable record exclusion policy"; +var UUID = /[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}/ig; +function object3(value) { + return value !== null && typeof value === "object" && !Array.isArray(value); +} +function keys(value, expected) { + return Object.keys(value).length === expected.length && expected.every((key) => key in value); +} +function transcriptName(value) { + return identity(value) && !/[\\/]/.test(value) && value.endsWith(".jsonl") && value !== ".jsonl"; +} +function identity(value) { + return typeof value === "string" && value.length > 0 && value.length <= 512 && !/[\x00-\x1f]/.test(value); +} +function screeningText(value, depth = 0) { + if (depth > 24) throw new Error("Record screening nesting limit exceeded"); + if (typeof value === "string") { + let decoded; + try { + decoded = JSON.parse(value); + } catch { + return value; + } + if (typeof decoded === "string" || object3(decoded) || Array.isArray(decoded)) { + return screeningText(decoded, depth + 1); + } + return value; + } + if (Array.isArray(value)) return value.map((item) => screeningText(item, depth + 1)).join("\n"); + if (object3(value)) { + return Object.entries(value).map(([key, item]) => `${key} = ${screeningText(item, depth + 1)}`).join("\n"); + } + return value == null ? "" : String(value); +} +function createRecordScreener(policy) { + const config2 = policy?.screening; + if (!config2) return { scan() { + }, close() { + } }; + const unavailable = () => new Error("Record screening unavailable"); + let initial; + let fd; + try { + if (fs2.realpathSync(config2.executable) !== config2.executable) throw unavailable(); + fd = fs2.openSync(config2.executable, fs2.constants.O_RDONLY | fs2.constants.O_NOFOLLOW); + initial = fs2.fstatSync(fd); + if (!initial.isFile() || initial.size < 1 || initial.size > 134217728 || (initial.mode & 18) !== 0 || typeof process.getuid === "function" && initial.uid !== 0 && initial.uid !== process.getuid()) { + throw unavailable(); + } + fs2.accessSync(config2.executable, fs2.constants.X_OK); + const digest = crypto.createHash("sha256").update(fs2.readFileSync(fd)).digest("hex"); + if (digest !== config2.sha256) throw unavailable(); + } catch { + throw unavailable(); + } finally { + if (fd !== void 0) fs2.closeSync(fd); + } + let directory; + try { + directory = fs2.mkdtempSync(path2.join(os2.tmpdir(), "episodic-screen-")); + fs2.chmodSync(directory, 448); + } catch { + throw unavailable(); + } + const unchanged = () => { + try { + const current = fs2.lstatSync(config2.executable); + if (!current.isFile() || current.dev !== initial.dev || current.ino !== initial.ino || current.size !== initial.size || current.mtimeMs !== initial.mtimeMs || current.ctimeMs !== initial.ctimeMs) throw unavailable(); + } catch { + throw unavailable(); + } + }; + return { + scan(value) { + const raw = typeof value === "string" ? value : JSON.stringify(value); + if (raw === void 0) return; + if (Buffer.byteLength(raw) > config2.max_record_bytes) throw new Error("Record screening byte limit exceeded"); + const input2 = screeningText(value); + if (Buffer.byteLength(input2) > config2.max_record_bytes) throw new Error("Record screening byte limit exceeded"); + if (!input2.trim()) return; + unchanged(); + const result = spawnSync(config2.executable, [ + "stdin", + "--redact", + "--no-banner", + "--log-level", + "error", + "--ignore-gitleaks-allow", + "--exit-code", + "10", + "--report-format", + "json", + "--report-path", + "-" + ], { + input: input2, + encoding: "utf8", + cwd: directory, + shell: false, + env: { HOME: directory, USERPROFILE: directory, PATH: path2.dirname(config2.executable) }, + timeout: config2.timeout_ms, + killSignal: "SIGKILL", + maxBuffer: 1048576, + windowsHide: true + }); + unchanged(); + if (result.error || result.signal || ![0, 10].includes(result.status ?? -1)) throw unavailable(); + let report; + try { + report = JSON.parse(result.stdout); + } catch { + throw unavailable(); + } + if (!Array.isArray(report) || result.status === 0 && report.length !== 0 || result.status === 10 && (report.length === 0 || !report.every((item) => object3(item) && typeof item.RuleID === "string"))) { + throw unavailable(); + } + if (result.status === 10) throw new Error("Record content rejected by screening"); + }, + close() { + fs2.rmSync(directory, { recursive: true, force: true }); + } + }; +} +function readRecordExclusions() { + const file2 = path2.join(getSuperpowersDir(), "record-exclusions.json"); + let fd; + try { + fd = fs2.openSync(file2, fs2.constants.O_RDONLY | fs2.constants.O_NOFOLLOW); + const info = fs2.fstatSync(fd); + if (!info.isFile() || info.size > 4 * 1024 * 1024) throw new Error(POLICY_ERROR); + const value = JSON.parse(fs2.readFileSync(fd, "utf8")); + if (!object3(value) || !keys(value, "screening" in value ? ["version", "exchanges", "tool_calls", "screening"] : ["version", "exchanges", "tool_calls"]) || value.version !== 1 || !Array.isArray(value.exchanges) || !Array.isArray(value.tool_calls)) throw new Error(POLICY_ERROR); + if ("screening" in value) { + const s = value.screening; + if (!object3(s) || !keys(s, ["engine", "executable", "sha256", "timeout_ms", "max_record_bytes"]) || s.engine !== "gitleaks" || typeof s.executable !== "string" || !path2.isAbsolute(s.executable) || /[\x00-\x1f]/.test(s.executable) || typeof s.sha256 !== "string" || !/^[a-f0-9]{64}$/.test(s.sha256) || !Number.isSafeInteger(s.timeout_ms) || Number(s.timeout_ms) < 50 || Number(s.timeout_ms) > 3e4 || !Number.isSafeInteger(s.max_record_bytes) || Number(s.max_record_bytes) < 1 || Number(s.max_record_bytes) > 8388608) { + throw new Error(POLICY_ERROR); + } + } + for (const row of value.exchanges) { + if (!object3(row) || !keys(row, ["session_id", "transcript", "line_start", "line_end"]) || !identity(row.session_id) || !transcriptName(row.transcript) || !Number.isSafeInteger(row.line_start) || !Number.isSafeInteger(row.line_end) || row.line_start < 1 || row.line_end < row.line_start) throw new Error(POLICY_ERROR); + } + for (const row of value.tool_calls) { + if (!object3(row) || !keys(row, ["session_id", "transcript", "call_id"]) || !identity(row.session_id) || !transcriptName(row.transcript) || !identity(row.call_id)) throw new Error(POLICY_ERROR); + } + return value; + } catch (error62) { + if (fd === void 0 && error62.code === "ENOENT") return null; + throw new Error(POLICY_ERROR); + } finally { + if (fd !== void 0) fs2.closeSync(fd); + } +} +function excludedSession(policy, sessionId) { + return !!sessionId && !!policy && (policy.exchanges.some((x2) => x2.session_id === sessionId) || policy.tool_calls.some((x2) => x2.session_id === sessionId)); +} +function sessionFromPath(file2) { + return path2.basename(file2).match(UUID)?.at(-1); +} +function sessionFromRecord(value) { + if (value?.type === "session_meta") return value.payload?.id; + if (value?.type === "session") return value.id; + if (value?.type === "opencode_session") return value.session?.id; + return value?.sessionId ?? value?.sessionID ?? value?.message?.sessionID; +} +function blockedCall(policy, sessionId, transcript, callId) { + return typeof callId === "string" && policy.tool_calls.some((x2) => x2.session_id === sessionId && x2.transcript === transcript && x2.call_id === callId); +} +function filterRecord(line, number4, policy, transcript, sessionId) { + if (!policy || !sessionId) return line; + if (policy.exchanges.some((x2) => x2.session_id === sessionId && x2.transcript === transcript && number4 >= x2.line_start && number4 <= x2.line_end)) return ""; + let value; + try { + value = JSON.parse(line); + } catch { + return line; + } + if (blockedCall(policy, sessionId, transcript, value?.payload?.call_id) || blockedCall(policy, sessionId, transcript, value?.call_id)) return ""; + const content = value?.message?.content; + if (Array.isArray(content)) { + const kept = content.filter((block) => !blockedCall(policy, sessionId, transcript, block?.id ?? block?.tool_use_id)); + if (kept.length !== content.length) { + return JSON.stringify({ ...value, message: { ...value.message, content: kept } }); + } + } + if (Array.isArray(value?.parts)) { + const kept = value.parts.filter((part) => !blockedCall(policy, sessionId, transcript, part?.callID ?? part?.id)); + if (kept.length !== value.parts.length) return JSON.stringify({ ...value, parts: kept }); + } + return line; +} +async function* admittedRecords(file2, policy, startLine = 1, endLine = Infinity) { + const policySnapshot = JSON.stringify(policy); + assertUnchangedPolicy(policySnapshot); + const input2 = fs2.openSync(file2, fs2.constants.O_RDONLY | fs2.constants.O_NOFOLLOW); + const initial = fs2.fstatSync(input2); + let stream; + let screener; + const decoder = new TextDecoder("utf-8", { fatal: true }); + let pending = Buffer.alloc(0); + let sessionId = sessionFromPath(file2); + let primaryIdentitySeen = false; + let number4 = 0; + const admit = (bytes, terminated) => { + number4++; + const crlf = terminated && bytes.at(-1) === 13; + const line = decoder.decode(crlf ? bytes.subarray(0, -1) : bytes); + let recordSession; + try { + recordSession = sessionFromRecord(JSON.parse(line)); + } catch { + } + if (recordSession && !primaryIdentitySeen) { + if (sessionId && recordSession !== sessionId && (excludedSession(policy, sessionId) || excludedSession(policy, recordSession))) { + throw new Error("Record exclusion session identity mismatch"); + } + sessionId = recordSession; + primaryIdentitySeen = true; + } + const filtered = number4 >= startLine && number4 <= endLine ? filterRecord(line, number4, policy, path2.basename(file2), sessionId) : ""; + screener?.scan(filtered); + return { line: filtered, ending: terminated ? crlf ? "\r\n" : "\n" : "" }; + }; + try { + if (descriptorHasExclusionMarker(input2)) return; + assertUnchangedInput(file2, initial); + screener = createRecordScreener(policy); + stream = fs2.createReadStream(file2, { fd: input2, autoClose: true, start: 0 }); + for await (const chunk of stream) { + assertUnchangedInput(file2, initial); + assertUnchangedPolicy(policySnapshot); + pending = Buffer.concat([pending, chunk]); + let newline; + while ((newline = pending.indexOf(10)) !== -1) { + yield admit(pending.subarray(0, newline), true); + pending = pending.subarray(newline + 1); + } + if (pending.length > 64 * 1024 * 1024) throw new Error("Conversation record exceeds admission byte limit"); + } + if (pending.length) yield admit(pending, false); + assertUnchangedInput(file2, initial); + assertUnchangedPolicy(policySnapshot); + } finally { + if (stream) stream.destroy(); + else fs2.closeSync(input2); + screener?.close(); + } +} +async function readAdmittedConversation(file2, startLine = 1, endLine = Infinity) { + if (!Number.isSafeInteger(startLine) || startLine < 1 || endLine !== Infinity && (!Number.isSafeInteger(endLine) || endLine < startLine)) { + throw new Error("Invalid conversation line range"); + } + const lines = []; + for await (const record2 of admittedRecords(file2, readRecordExclusions(), startLine, endLine)) lines.push(record2.line); + return lines.join("\n"); +} +function admitExchange(exchange, policy = readRecordExclusions()) { + if (!policy) return exchange; + const sessionIds = [exchange.sessionId, sessionFromPath(exchange.archivePath)].filter((id) => !!id); + const transcript = path2.basename(exchange.archivePath); + if (policy.exchanges.some((x2) => sessionIds.includes(x2.session_id) && x2.transcript === transcript && exchange.lineStart <= x2.line_end && exchange.lineEnd >= x2.line_start)) return null; + const kept = exchange.toolCalls?.filter((call) => !sessionIds.some((id) => blockedCall(policy, id, transcript, call.id))); + const admitted = kept && kept.length !== exchange.toolCalls?.length ? { ...exchange, toolCalls: kept } : exchange; + const screener = createRecordScreener(policy); + try { + screener.scan(admitted.userMessage); + screener.scan(admitted.assistantMessage); + for (const call of admitted.toolCalls ?? []) screener.scan(call); + const { userMessage, assistantMessage, toolCalls, ...metadata } = admitted; + screener.scan(metadata); + return admitted; + } finally { + screener.close(); + } +} +function createSearchAdmission() { + const policy = readRecordExclusions(); + const policySnapshot = JSON.stringify(policy); + const files = /* @__PURE__ */ new Map(); + const checkFile = (file2) => { + assertUnchangedPolicy(policySnapshot); + const cached2 = files.get(file2); + if (cached2) return !cached2.excluded; + let fd; + try { + let pathInfo; + try { + pathInfo = fs2.lstatSync(file2); + } catch (error62) { + if (error62.code !== "ENOENT") throw error62; + files.set(file2, { excluded: true }); + return false; + } + if (!pathInfo.isFile()) throw new Error("Conversation changed during record admission"); + fd = fs2.openSync(file2, fs2.constants.O_RDONLY | fs2.constants.O_NOFOLLOW); + const initial = fs2.fstatSync(fd); + if (!initial.isFile() || initial.dev !== pathInfo.dev || initial.ino !== pathInfo.ino || !Number.isSafeInteger(initial.size)) throw new Error("Conversation changed during record admission"); + const hash2 = crypto.createHash("sha256"); + const excluded = descriptorHasExclusionMarker(fd, initial.size, hash2); + files.set(file2, { initial, excluded, digest: excluded ? void 0 : hash2.digest("hex") }); + return !excluded; + } catch { + throw new Error("Conversation changed during record admission"); + } finally { + if (fd !== void 0) fs2.closeSync(fd); + } + }; + return { + admit(exchange) { + if (!checkFile(exchange.archivePath)) return null; + const admitted = admitExchange(exchange, policy); + assertUnchangedPolicy(policySnapshot); + return admitted; + }, + summary(exchange, text) { + const ids = [exchange.sessionId, sessionFromPath(exchange.archivePath)]; + const transcript = path2.basename(exchange.archivePath); + if (policy && [...policy.exchanges, ...policy.tool_calls].some((row) => row.transcript === transcript && ids.includes(row.session_id))) return void 0; + return admitExchange({ ...exchange, userMessage: text, assistantMessage: "", toolCalls: [] }, policy) ? text : void 0; + }, + verify() { + assertUnchangedPolicy(policySnapshot); + for (const [file2, { initial, digest }] of files) { + if (initial !== void 0 && digest !== void 0) assertUnchangedSearchPrefix(file2, initial, digest); + } + assertUnchangedPolicy(policySnapshot); + } + }; +} + +// src/db.ts +import path3 from "path"; +import fs3 from "fs"; +import * as sqliteVec from "sqlite-vec"; + // src/file-lock.ts import * as lockfile from "proper-lockfile"; var DEFAULT_STALE_MS = 10 * 60 * 1e3; @@ -29545,9 +31119,9 @@ function migrateToolCallsCascade(db) { } function initDatabase() { const dbPath = getDbPath(); - const dbDir = path2.dirname(dbPath); - if (!fs2.existsSync(dbDir)) { - fs2.mkdirSync(dbDir, { recursive: true }); + const dbDir = path3.dirname(dbPath); + if (!fs3.existsSync(dbDir)) { + fs3.mkdirSync(dbDir, { recursive: true }); } const db = new Database(dbPath); sqliteVec.load(db); @@ -29658,10 +31232,10 @@ async function initEmbeddings() { transformers.env.allowLocalModels = true; transformers.env.useBrowserCache = false; pipeline = transformers.pipeline; - } catch (error61) { + } catch (error62) { throw new EmbeddingsUnavailableError( - `Failed to load the embedding backend (@huggingface/transformers). This usually means the native "sharp" binding could not load libvips; semantic search and indexing are unavailable until it is fixed. Underlying error: ${error61 instanceof Error ? error61.message : String(error61)}`, - { cause: error61 } + `Failed to load the embedding backend (@huggingface/transformers). This usually means the native "sharp" binding could not load libvips; semantic search and indexing are unavailable until it is fixed. Underlying error: ${error62 instanceof Error ? error62.message : String(error62)}`, + { cause: error62 } ); } console.error("Loading embedding model (first run may take time)..."); @@ -29708,7 +31282,7 @@ function isErroredSentinel(content) { } // src/search.ts -import fs3 from "fs"; +import fs4 from "fs"; import readline from "readline"; var SIDECHAIN_DISTANCE_PENALTY = 0.05; function buildSearchFilters(options) { @@ -29826,84 +31400,88 @@ async function searchConversations(query, options = {}) { const sidechainClause = includeSidechains ? "" : "AND e.is_sidechain = 0"; if (after) validateISODate(after, "--after"); if (before) validateISODate(before, "--before"); + const admission = createSearchAdmission(); const db = initDatabase(); + const admitted = (row) => admission.admit(exchangeFromRow(row)) !== null; let results = []; const { sql: filterClause, params: filterParams } = buildSearchFilters(options); - if (mode === "vector" || mode === "both") { - await initEmbeddings(); - const queryEmbedding = await generateQueryEmbedding(query); - const k2 = limit * 3; - const stmt = db.prepare(` - SELECT - ${EXCHANGE_SELECT_COLUMNS}, - vec.distance - FROM vec_exchanges AS vec - JOIN exchanges AS e ON vec.id = e.id - WHERE vec.embedding MATCH ? - AND k = ? - ${sidechainClause} - ${filterClause} - ORDER BY (vec.distance + e.is_sidechain * ?) ASC - `); - results = stmt.all( - Buffer.from(new Float32Array(queryEmbedding).buffer), - k2, - ...filterParams, - SIDECHAIN_DISTANCE_PENALTY - ); - if (results.length > limit) { - results = results.slice(0, limit); - } - } - if (mode === "text" || mode === "both") { - const { sql: textMatchSql, params: textMatchParams } = buildTextMatchClause(query); - const textStmt = db.prepare(` - SELECT - ${EXCHANGE_SELECT_COLUMNS}, - 0 as distance - FROM exchanges AS e - WHERE ${textMatchSql} - ${sidechainClause} - ${filterClause} - ORDER BY e.is_sidechain ASC, e.timestamp DESC - LIMIT ? - `); - const textResults = textStmt.all(...textMatchParams, ...filterParams, limit); - if (mode === "both") { - const seenIds = new Set(results.map((r) => r.id)); - for (const textResult of textResults) { - if (!seenIds.has(textResult.id)) { - results.push(textResult); + try { + if (mode === "vector" || mode === "both") { + await initEmbeddings(); + const queryEmbedding = await generateQueryEmbedding(query); + const stmt = db.prepare(` + SELECT ${EXCHANGE_SELECT_COLUMNS}, + vec_distance_l2(vec.embedding, ?) AS retrieval_distance + FROM vec_exchanges AS vec + JOIN exchanges AS e ON vec.id = e.id + WHERE 1 = 1 ${sidechainClause} ${filterClause} + ORDER BY (retrieval_distance + e.is_sidechain * ?) ASC + `); + for (const row of stmt.iterate( + Buffer.from(new Float32Array(queryEmbedding).buffer), + ...filterParams, + SIDECHAIN_DISTANCE_PENALTY + )) { + if (admitted(row)) results.push(row); + if (results.length === limit) break; + } + } + if (mode === "text" || mode === "both") { + const { sql: textMatchSql, params: textMatchParams } = buildTextMatchClause(query); + const textStmt = db.prepare(` + SELECT + ${EXCHANGE_SELECT_COLUMNS}, + 0 as retrieval_distance + FROM exchanges AS e + WHERE ${textMatchSql} + ${sidechainClause} + ${filterClause} + ORDER BY e.is_sidechain ASC, e.timestamp DESC + `); + const textResults = []; + for (const row of textStmt.iterate(...textMatchParams, ...filterParams)) { + if (admitted(row)) textResults.push(row); + if (textResults.length === limit) break; + } + if (mode === "both") { + const seenIds = new Set(results.map((r) => r.id)); + for (const textResult of textResults) { + if (!seenIds.has(textResult.id)) { + results.push(textResult); + } } + } else { + results = textResults; } - } else { - results = textResults; } + } finally { + db.close(); } - db.close(); - return results.map((row) => { + const output2 = results.map((row) => { const exchange = exchangeFromRow(row); const summaryPath = row.archive_path.replace(".jsonl", "-summary.txt"); let summary; - if (fs3.existsSync(summaryPath)) { - const raw = fs3.readFileSync(summaryPath, "utf-8"); + if (fs4.existsSync(summaryPath)) { + const raw = fs4.readFileSync(summaryPath, "utf-8"); if (!isErroredSentinel(raw)) { - summary = raw.trim(); + summary = admission.summary(exchange, raw.trim()); } } const snippetText = exchange.userMessage.substring(0, 200).replace(/\s+/g, " ").trim(); const snippet = snippetText + (exchange.userMessage.length > 200 ? "..." : ""); return { exchange, - similarity: mode === "text" ? void 0 : l2DistanceToCosineSimilarity(row.distance), + similarity: mode === "text" ? void 0 : l2DistanceToCosineSimilarity(row.retrieval_distance), snippet, summary }; }); + admission.verify(); + return output2; } async function countLines(filePath) { try { - const fileStream = fs3.createReadStream(filePath); + const fileStream = fs4.createReadStream(filePath); const rl = readline.createInterface({ input: fileStream, crlfDelay: Infinity @@ -29913,15 +31491,15 @@ async function countLines(filePath) { if (line.trim()) count++; } return count; - } catch (error61) { + } catch (error62) { return 0; } } function getFileSizeInKB(filePath) { try { - const stats = fs3.statSync(filePath); + const stats = fs4.statSync(filePath); return Math.round(stats.size / 1024 * 10) / 10; - } catch (error61) { + } catch (error62) { return 0; } } @@ -30046,140 +31624,147 @@ function L() { return { async: false, breaks: false, extensions: null, gfm: true, hooks: null, pedantic: false, renderer: null, silent: false, tokenizer: null, walkTokens: null }; } var T = L(); -function G(u3) { - T = u3; +function G(l3) { + T = l3; } -var I = { exec: () => null }; -function d(u3, e = "") { - let t = typeof u3 == "string" ? u3 : u3.source, n = { replace: (r, i) => { +var E = { exec: () => null }; +function d(l3, e = "") { + let t = typeof l3 == "string" ? l3 : l3.source, n = { replace: (r, i) => { let s = typeof i == "string" ? i : i.source; return s = s.replace(m.caret, "$1"), t = t.replace(r, s), n; }, getRegex: () => new RegExp(t, e) }; return n; } -var m = { codeRemoveIndent: /^(?: {1,4}| {0,3}\t)/gm, outputLinkReplace: /\\([\[\]])/g, indentCodeCompensation: /^(\s+)(?:```)/, beginningSpace: /^\s+/, endingHash: /#$/, startingSpaceChar: /^ /, endingSpaceChar: / $/, nonSpaceChar: /[^ ]/, newLineCharGlobal: /\n/g, tabCharGlobal: /\t/g, multipleSpaceGlobal: /\s+/g, blankLine: /^[ \t]*$/, doubleBlankLine: /\n[ \t]*\n[ \t]*$/, blockquoteStart: /^ {0,3}>/, blockquoteSetextReplace: /\n {0,3}((?:=+|-+) *)(?=\n|$)/g, blockquoteSetextReplace2: /^ {0,3}>[ \t]?/gm, listReplaceTabs: /^\t+/, listReplaceNesting: /^ {1,4}(?=( {4})*[^ ])/g, listIsTask: /^\[[ xX]\] /, listReplaceTask: /^\[[ xX]\] +/, anyLine: /\n.*\n/, hrefBrackets: /^<(.*)>$/, tableDelimiter: /[:|]/, tableAlignChars: /^\||\| *$/g, tableRowBlankLine: /\n[ \t]*$/, tableAlignRight: /^ *-+: *$/, tableAlignCenter: /^ *:-+: *$/, tableAlignLeft: /^ *:-+ *$/, startATag: /^/i, startPreScriptTag: /^<(pre|code|kbd|script)(\s|>)/i, endPreScriptTag: /^<\/(pre|code|kbd|script)(\s|>)/i, startAngleBracket: /^$/, pedanticHrefTitle: /^([^'"]*[^\s])\s+(['"])(.*)\2/, unicodeAlphaNumeric: /[\p{L}\p{N}]/u, escapeTest: /[&<>"']/, escapeReplace: /[&<>"']/g, escapeTestNoEncode: /[<>"']|&(?!(#\d{1,7}|#[Xx][a-fA-F0-9]{1,6}|\w+);)/, escapeReplaceNoEncode: /[<>"']|&(?!(#\d{1,7}|#[Xx][a-fA-F0-9]{1,6}|\w+);)/g, unescapeTest: /&(#(?:\d+)|(?:#x[0-9A-Fa-f]+)|(?:\w+));?/ig, caret: /(^|[^\[])\^/g, percentDecode: /%25/g, findPipe: /\|/g, splitPipe: / \|/, slashPipe: /\\\|/g, carriageReturn: /\r\n|\r/g, spaceLine: /^ +$/gm, notSpaceStart: /^\S*/, endingNewline: /\n$/, listItemRegex: (u3) => new RegExp(`^( {0,3}${u3})((?:[ ][^\\n]*)?(?:\\n|$))`), nextBulletRegex: (u3) => new RegExp(`^ {0,${Math.min(3, u3 - 1)}}(?:[*+-]|\\d{1,9}[.)])((?:[ ][^\\n]*)?(?:\\n|$))`), hrRegex: (u3) => new RegExp(`^ {0,${Math.min(3, u3 - 1)}}((?:- *){3,}|(?:_ *){3,}|(?:\\* *){3,})(?:\\n+|$)`), fencesBeginRegex: (u3) => new RegExp(`^ {0,${Math.min(3, u3 - 1)}}(?:\`\`\`|~~~)`), headingBeginRegex: (u3) => new RegExp(`^ {0,${Math.min(3, u3 - 1)}}#`), htmlBeginRegex: (u3) => new RegExp(`^ {0,${Math.min(3, u3 - 1)}}<(?:[a-z].*>|!--)`, "i") }; -var be = /^(?:[ \t]*(?:\n|$))+/; -var Re = /^((?: {4}| {0,3}\t)[^\n]+(?:\n(?:[ \t]*(?:\n|$))*)?)+/; -var Te = /^ {0,3}(`{3,}(?=[^`\n]*(?:\n|$))|~{3,})([^\n]*)(?:\n|$)(?:|([\s\S]*?)(?:\n|$))(?: {0,3}\1[~`]* *(?=\n|$)|$)/; -var E = /^ {0,3}((?:-[\t ]*){3,}|(?:_[ \t]*){3,}|(?:\*[ \t]*){3,})(?:\n+|$)/; -var Oe = /^ {0,3}(#{1,6})(?=\s|$)(.*)(?:\n+|$)/; +var be = (() => { + try { + return !!new RegExp("(?<=1)(?/, blockquoteSetextReplace: /\n {0,3}((?:=+|-+) *)(?=\n|$)/g, blockquoteSetextReplace2: /^ {0,3}>[ \t]?/gm, listReplaceTabs: /^\t+/, listReplaceNesting: /^ {1,4}(?=( {4})*[^ ])/g, listIsTask: /^\[[ xX]\] /, listReplaceTask: /^\[[ xX]\] +/, anyLine: /\n.*\n/, hrefBrackets: /^<(.*)>$/, tableDelimiter: /[:|]/, tableAlignChars: /^\||\| *$/g, tableRowBlankLine: /\n[ \t]*$/, tableAlignRight: /^ *-+: *$/, tableAlignCenter: /^ *:-+: *$/, tableAlignLeft: /^ *:-+ *$/, startATag: /^/i, startPreScriptTag: /^<(pre|code|kbd|script)(\s|>)/i, endPreScriptTag: /^<\/(pre|code|kbd|script)(\s|>)/i, startAngleBracket: /^$/, pedanticHrefTitle: /^([^'"]*[^\s])\s+(['"])(.*)\2/, unicodeAlphaNumeric: /[\p{L}\p{N}]/u, escapeTest: /[&<>"']/, escapeReplace: /[&<>"']/g, escapeTestNoEncode: /[<>"']|&(?!(#\d{1,7}|#[Xx][a-fA-F0-9]{1,6}|\w+);)/, escapeReplaceNoEncode: /[<>"']|&(?!(#\d{1,7}|#[Xx][a-fA-F0-9]{1,6}|\w+);)/g, unescapeTest: /&(#(?:\d+)|(?:#x[0-9A-Fa-f]+)|(?:\w+));?/ig, caret: /(^|[^\[])\^/g, percentDecode: /%25/g, findPipe: /\|/g, splitPipe: / \|/, slashPipe: /\\\|/g, carriageReturn: /\r\n|\r/g, spaceLine: /^ +$/gm, notSpaceStart: /^\S*/, endingNewline: /\n$/, listItemRegex: (l3) => new RegExp(`^( {0,3}${l3})((?:[ ][^\\n]*)?(?:\\n|$))`), nextBulletRegex: (l3) => new RegExp(`^ {0,${Math.min(3, l3 - 1)}}(?:[*+-]|\\d{1,9}[.)])((?:[ ][^\\n]*)?(?:\\n|$))`), hrRegex: (l3) => new RegExp(`^ {0,${Math.min(3, l3 - 1)}}((?:- *){3,}|(?:_ *){3,}|(?:\\* *){3,})(?:\\n+|$)`), fencesBeginRegex: (l3) => new RegExp(`^ {0,${Math.min(3, l3 - 1)}}(?:\`\`\`|~~~)`), headingBeginRegex: (l3) => new RegExp(`^ {0,${Math.min(3, l3 - 1)}}#`), htmlBeginRegex: (l3) => new RegExp(`^ {0,${Math.min(3, l3 - 1)}}<(?:[a-z].*>|!--)`, "i") }; +var Re = /^(?:[ \t]*(?:\n|$))+/; +var Te = /^((?: {4}| {0,3}\t)[^\n]+(?:\n(?:[ \t]*(?:\n|$))*)?)+/; +var Oe = /^ {0,3}(`{3,}(?=[^`\n]*(?:\n|$))|~{3,})([^\n]*)(?:\n|$)(?:|([\s\S]*?)(?:\n|$))(?: {0,3}\1[~`]* *(?=\n|$)|$)/; +var I = /^ {0,3}((?:-[\t ]*){3,}|(?:_[ \t]*){3,}|(?:\*[ \t]*){3,})(?:\n+|$)/; +var we = /^ {0,3}(#{1,6})(?=\s|$)(.*)(?:\n+|$)/; var F = /(?:[*+-]|\d{1,9}[.)])/; var ie = /^(?!bull |blockCode|fences|blockquote|heading|html|table)((?:.|\n(?!\s*?\n|bull |blockCode|fences|blockquote|heading|html|table))+?)\n {0,3}(=+|-+) *(?:\n+|$)/; var oe = d(ie).replace(/bull/g, F).replace(/blockCode/g, /(?: {4}| {0,3}\t)/).replace(/fences/g, / {0,3}(?:`{3,}|~{3,})/).replace(/blockquote/g, / {0,3}>/).replace(/heading/g, / {0,3}#{1,6}/).replace(/html/g, / {0,3}<[^\n>]+>\n/).replace(/\|table/g, "").getRegex(); -var we = d(ie).replace(/bull/g, F).replace(/blockCode/g, /(?: {4}| {0,3}\t)/).replace(/fences/g, / {0,3}(?:`{3,}|~{3,})/).replace(/blockquote/g, / {0,3}>/).replace(/heading/g, / {0,3}#{1,6}/).replace(/html/g, / {0,3}<[^\n>]+>\n/).replace(/table/g, / {0,3}\|?(?:[:\- ]*\|)+[\:\- ]*\n/).getRegex(); +var ye = d(ie).replace(/bull/g, F).replace(/blockCode/g, /(?: {4}| {0,3}\t)/).replace(/fences/g, / {0,3}(?:`{3,}|~{3,})/).replace(/blockquote/g, / {0,3}>/).replace(/heading/g, / {0,3}#{1,6}/).replace(/html/g, / {0,3}<[^\n>]+>\n/).replace(/table/g, / {0,3}\|?(?:[:\- ]*\|)+[\:\- ]*\n/).getRegex(); var j = /^([^\n]+(?:\n(?!hr|heading|lheading|blockquote|fences|list|html|table| +\n)[^\n]+)*)/; -var ye = /^[^\n]+/; +var Pe = /^[^\n]+/; var Q = /(?!\s*\])(?:\\[\s\S]|[^\[\]\\])+/; -var Pe = d(/^ {0,3}\[(label)\]: *(?:\n[ \t]*)?([^<\s][^\s]*|<.*?>)(?:(?: +(?:\n[ \t]*)?| *\n[ \t]*)(title))? *(?:\n+|$)/).replace("label", Q).replace("title", /(?:"(?:\\"?|[^"\\])*"|'[^'\n]*(?:\n[^'\n]+)*\n?'|\([^()]*\))/).getRegex(); -var Se = d(/^( {0,3}bull)([ \t][^\n]+?)?(?:\n|$)/).replace(/bull/g, F).getRegex(); +var Se = d(/^ {0,3}\[(label)\]: *(?:\n[ \t]*)?([^<\s][^\s]*|<.*?>)(?:(?: +(?:\n[ \t]*)?| *\n[ \t]*)(title))? *(?:\n+|$)/).replace("label", Q).replace("title", /(?:"(?:\\"?|[^"\\])*"|'[^'\n]*(?:\n[^'\n]+)*\n?'|\([^()]*\))/).getRegex(); +var $e = d(/^( {0,3}bull)([ \t][^\n]+?)?(?:\n|$)/).replace(/bull/g, F).getRegex(); var v = "address|article|aside|base|basefont|blockquote|body|caption|center|col|colgroup|dd|details|dialog|dir|div|dl|dt|fieldset|figcaption|figure|footer|form|frame|frameset|h[1-6]|head|header|hr|html|iframe|legend|li|link|main|menu|menuitem|meta|nav|noframes|ol|optgroup|option|p|param|search|section|summary|table|tbody|td|tfoot|th|thead|title|tr|track|ul"; var U = /|$))/; -var $e = d("^ {0,3}(?:<(script|pre|style|textarea)[\\s>][\\s\\S]*?(?:[^\\n]*\\n+|$)|comment[^\\n]*(\\n+|$)|<\\?[\\s\\S]*?(?:\\?>\\n*|$)|\\n*|$)|\\n*|$)|)[\\s\\S]*?(?:(?:\\n[ ]*)+\\n|$)|<(?!script|pre|style|textarea)([a-z][\\w-]*)(?:attribute)*? */?>(?=[ \\t]*(?:\\n|$))[\\s\\S]*?(?:(?:\\n[ ]*)+\\n|$)|(?=[ \\t]*(?:\\n|$))[\\s\\S]*?(?:(?:\\n[ ]*)+\\n|$))", "i").replace("comment", U).replace("tag", v).replace("attribute", / +[a-zA-Z:_][\w.:-]*(?: *= *"[^"\n]*"| *= *'[^'\n]*'| *= *[^\s"'=<>`]+)?/).getRegex(); -var ae = d(j).replace("hr", E).replace("heading", " {0,3}#{1,6}(?:\\s|$)").replace("|lheading", "").replace("|table", "").replace("blockquote", " {0,3}>").replace("fences", " {0,3}(?:`{3,}(?=[^`\\n]*\\n)|~{3,})[^\\n]*\\n").replace("list", " {0,3}(?:[*+-]|1[.)]) ").replace("html", ")|<(?:script|pre|style|textarea|!--)").replace("tag", v).getRegex(); -var _e = d(/^( {0,3}> ?(paragraph|[^\n]*)(?:\n|$))+/).replace("paragraph", ae).getRegex(); -var K = { blockquote: _e, code: Re, def: Pe, fences: Te, heading: Oe, hr: E, html: $e, lheading: oe, list: Se, newline: be, paragraph: ae, table: I, text: ye }; -var re = d("^ *([^\\n ].*)\\n {0,3}((?:\\| *)?:?-+:? *(?:\\| *:?-+:? *)*(?:\\| *)?)(?:\\n((?:(?! *\\n|hr|heading|blockquote|code|fences|list|html).*(?:\\n|$))*)\\n*|$)").replace("hr", E).replace("heading", " {0,3}#{1,6}(?:\\s|$)").replace("blockquote", " {0,3}>").replace("code", "(?: {4}| {0,3} )[^\\n]").replace("fences", " {0,3}(?:`{3,}(?=[^`\\n]*\\n)|~{3,})[^\\n]*\\n").replace("list", " {0,3}(?:[*+-]|1[.)]) ").replace("html", ")|<(?:script|pre|style|textarea|!--)").replace("tag", v).getRegex(); -var Le = { ...K, lheading: we, table: re, paragraph: d(j).replace("hr", E).replace("heading", " {0,3}#{1,6}(?:\\s|$)").replace("|lheading", "").replace("table", re).replace("blockquote", " {0,3}>").replace("fences", " {0,3}(?:`{3,}(?=[^`\\n]*\\n)|~{3,})[^\\n]*\\n").replace("list", " {0,3}(?:[*+-]|1[.)]) ").replace("html", ")|<(?:script|pre|style|textarea|!--)").replace("tag", v).getRegex() }; -var Me = { ...K, html: d(`^ *(?:comment *(?:\\n|\\s*$)|<(tag)[\\s\\S]+? *(?:\\n{2,}|\\s*$)|\\s]*)*?/?> *(?:\\n{2,}|\\s*$))`).replace("comment", U).replace(/tag/g, "(?!(?:a|em|strong|small|s|cite|q|dfn|abbr|data|time|code|var|samp|kbd|sub|sup|i|b|u|mark|ruby|rt|rp|bdi|bdo|span|br|wbr|ins|del|img)\\b)\\w+(?!:|[^\\w\\s@]*@)\\b").getRegex(), def: /^ *\[([^\]]+)\]: *]+)>?(?: +(["(][^\n]+[")]))? *(?:\n+|$)/, heading: /^(#{1,6})(.*)(?:\n+|$)/, fences: I, lheading: /^(.+?)\n {0,3}(=+|-+) *(?:\n+|$)/, paragraph: d(j).replace("hr", E).replace("heading", ` *#{1,6} *[^ +var _e = d("^ {0,3}(?:<(script|pre|style|textarea)[\\s>][\\s\\S]*?(?:[^\\n]*\\n+|$)|comment[^\\n]*(\\n+|$)|<\\?[\\s\\S]*?(?:\\?>\\n*|$)|\\n*|$)|\\n*|$)|)[\\s\\S]*?(?:(?:\\n[ ]*)+\\n|$)|<(?!script|pre|style|textarea)([a-z][\\w-]*)(?:attribute)*? */?>(?=[ \\t]*(?:\\n|$))[\\s\\S]*?(?:(?:\\n[ ]*)+\\n|$)|(?=[ \\t]*(?:\\n|$))[\\s\\S]*?(?:(?:\\n[ ]*)+\\n|$))", "i").replace("comment", U).replace("tag", v).replace("attribute", / +[a-zA-Z:_][\w.:-]*(?: *= *"[^"\n]*"| *= *'[^'\n]*'| *= *[^\s"'=<>`]+)?/).getRegex(); +var ae = d(j).replace("hr", I).replace("heading", " {0,3}#{1,6}(?:\\s|$)").replace("|lheading", "").replace("|table", "").replace("blockquote", " {0,3}>").replace("fences", " {0,3}(?:`{3,}(?=[^`\\n]*\\n)|~{3,})[^\\n]*\\n").replace("list", " {0,3}(?:[*+-]|1[.)]) ").replace("html", ")|<(?:script|pre|style|textarea|!--)").replace("tag", v).getRegex(); +var Le = d(/^( {0,3}> ?(paragraph|[^\n]*)(?:\n|$))+/).replace("paragraph", ae).getRegex(); +var K = { blockquote: Le, code: Te, def: Se, fences: Oe, heading: we, hr: I, html: _e, lheading: oe, list: $e, newline: Re, paragraph: ae, table: E, text: Pe }; +var re = d("^ *([^\\n ].*)\\n {0,3}((?:\\| *)?:?-+:? *(?:\\| *:?-+:? *)*(?:\\| *)?)(?:\\n((?:(?! *\\n|hr|heading|blockquote|code|fences|list|html).*(?:\\n|$))*)\\n*|$)").replace("hr", I).replace("heading", " {0,3}#{1,6}(?:\\s|$)").replace("blockquote", " {0,3}>").replace("code", "(?: {4}| {0,3} )[^\\n]").replace("fences", " {0,3}(?:`{3,}(?=[^`\\n]*\\n)|~{3,})[^\\n]*\\n").replace("list", " {0,3}(?:[*+-]|1[.)]) ").replace("html", ")|<(?:script|pre|style|textarea|!--)").replace("tag", v).getRegex(); +var Me = { ...K, lheading: ye, table: re, paragraph: d(j).replace("hr", I).replace("heading", " {0,3}#{1,6}(?:\\s|$)").replace("|lheading", "").replace("table", re).replace("blockquote", " {0,3}>").replace("fences", " {0,3}(?:`{3,}(?=[^`\\n]*\\n)|~{3,})[^\\n]*\\n").replace("list", " {0,3}(?:[*+-]|1[.)]) ").replace("html", ")|<(?:script|pre|style|textarea|!--)").replace("tag", v).getRegex() }; +var ze = { ...K, html: d(`^ *(?:comment *(?:\\n|\\s*$)|<(tag)[\\s\\S]+? *(?:\\n{2,}|\\s*$)|\\s]*)*?/?> *(?:\\n{2,}|\\s*$))`).replace("comment", U).replace(/tag/g, "(?!(?:a|em|strong|small|s|cite|q|dfn|abbr|data|time|code|var|samp|kbd|sub|sup|i|b|u|mark|ruby|rt|rp|bdi|bdo|span|br|wbr|ins|del|img)\\b)\\w+(?!:|[^\\w\\s@]*@)\\b").getRegex(), def: /^ *\[([^\]]+)\]: *]+)>?(?: +(["(][^\n]+[")]))? *(?:\n+|$)/, heading: /^(#{1,6})(.*)(?:\n+|$)/, fences: E, lheading: /^(.+?)\n {0,3}(=+|-+) *(?:\n+|$)/, paragraph: d(j).replace("hr", I).replace("heading", ` *#{1,6} *[^ ]`).replace("lheading", oe).replace("|table", "").replace("blockquote", " {0,3}>").replace("|fences", "").replace("|list", "").replace("|html", "").replace("|tag", "").getRegex() }; -var ze = /^\\([!"#$%&'()*+,\-./:;<=>?@\[\]\\^_`{|}~])/; -var Ae = /^(`+)([^`]|[^`][\s\S]*?[^`])\1(?!`)/; +var Ae = /^\\([!"#$%&'()*+,\-./:;<=>?@\[\]\\^_`{|}~])/; +var Ee = /^(`+)([^`]|[^`][\s\S]*?[^`])\1(?!`)/; var le = /^( {2,}|\\)\n(?!\s*$)/; var Ie = /^(`+|[^`])(?:(?= {2,}\n)|[\s\S]*?(?:(?=[\\]*?>/g; +var Be = /(?!~)[\s\p{P}\p{S}]/u; +var qe = /(?:[^\s\p{P}\p{S}]|~)/u; +var ve = d(/link|precode-code|html/, "g").replace("link", /\[(?:[^\[\]`]|(?`+)[^`]+\k(?!`))*?\]\((?:\\[\s\S]|[^\\\(\)]|\((?:\\[\s\S]|[^\\\(\)])*\))*\)/).replace("precode-", be ? "(?`+)[^`]+\k(?!`)/).replace("html", /<(?! )[^<>]*?>/).getRegex(); var ce = /^(?:\*+(?:((?!\*)punct)|[^\s*]))|^_+(?:((?!_)punct)|([^\s_]))/; -var ve = d(ce, "u").replace(/punct/g, D).getRegex(); -var De = d(ce, "u").replace(/punct/g, pe).getRegex(); +var De = d(ce, "u").replace(/punct/g, D).getRegex(); +var He = d(ce, "u").replace(/punct/g, pe).getRegex(); var he = "^[^_*]*?__[^_*]*?\\*[^_*]*?(?=__)|[^*]+(?=[^*])|(?!\\*)punct(\\*+)(?=[\\s]|$)|notPunctSpace(\\*+)(?!\\*)(?=punctSpace|$)|(?!\\*)punctSpace(\\*+)(?=notPunctSpace)|[\\s](\\*+)(?!\\*)(?=punct)|(?!\\*)punct(\\*+)(?!\\*)(?=punct)|notPunctSpace(\\*+)(?=notPunctSpace)"; -var He = d(he, "gu").replace(/notPunctSpace/g, ue).replace(/punctSpace/g, W).replace(/punct/g, D).getRegex(); -var Ze = d(he, "gu").replace(/notPunctSpace/g, Be).replace(/punctSpace/g, Ce).replace(/punct/g, pe).getRegex(); -var Ge = d("^[^_*]*?\\*\\*[^_*]*?_[^_*]*?(?=\\*\\*)|[^_]+(?=[^_])|(?!_)punct(_+)(?=[\\s]|$)|notPunctSpace(_+)(?!_)(?=punctSpace|$)|(?!_)punctSpace(_+)(?=notPunctSpace)|[\\s](_+)(?!_)(?=punct)|(?!_)punct(_+)(?!_)(?=punct)", "gu").replace(/notPunctSpace/g, ue).replace(/punctSpace/g, W).replace(/punct/g, D).getRegex(); -var Ne = d(/\\(punct)/, "gu").replace(/punct/g, D).getRegex(); -var Fe = d(/^<(scheme:[^\s\x00-\x1f<>]*|email)>/).replace("scheme", /[a-zA-Z][a-zA-Z0-9+.-]{1,31}/).replace("email", /[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+(@)[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)+(?![-_])/).getRegex(); -var je = d(U).replace("(?:-->|$)", "-->").getRegex(); -var Qe = d("^comment|^|^<[a-zA-Z][\\w-]*(?:attribute)*?\\s*/?>|^<\\?[\\s\\S]*?\\?>|^|^").replace("comment", je).replace("attribute", /\s+[a-zA-Z:_][\w.:-]*(?:\s*=\s*"[^"]*"|\s*=\s*'[^']*'|\s*=\s*[^\s"'=<>`]+)?/).getRegex(); +var Ze = d(he, "gu").replace(/notPunctSpace/g, ue).replace(/punctSpace/g, W).replace(/punct/g, D).getRegex(); +var Ge = d(he, "gu").replace(/notPunctSpace/g, qe).replace(/punctSpace/g, Be).replace(/punct/g, pe).getRegex(); +var Ne = d("^[^_*]*?\\*\\*[^_*]*?_[^_*]*?(?=\\*\\*)|[^_]+(?=[^_])|(?!_)punct(_+)(?=[\\s]|$)|notPunctSpace(_+)(?!_)(?=punctSpace|$)|(?!_)punctSpace(_+)(?=notPunctSpace)|[\\s](_+)(?!_)(?=punct)|(?!_)punct(_+)(?!_)(?=punct)", "gu").replace(/notPunctSpace/g, ue).replace(/punctSpace/g, W).replace(/punct/g, D).getRegex(); +var Fe = d(/\\(punct)/, "gu").replace(/punct/g, D).getRegex(); +var je = d(/^<(scheme:[^\s\x00-\x1f<>]*|email)>/).replace("scheme", /[a-zA-Z][a-zA-Z0-9+.-]{1,31}/).replace("email", /[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+(@)[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)+(?![-_])/).getRegex(); +var Qe = d(U).replace("(?:-->|$)", "-->").getRegex(); +var Ue = d("^comment|^|^<[a-zA-Z][\\w-]*(?:attribute)*?\\s*/?>|^<\\?[\\s\\S]*?\\?>|^|^").replace("comment", Qe).replace("attribute", /\s+[a-zA-Z:_][\w.:-]*(?:\s*=\s*"[^"]*"|\s*=\s*'[^']*'|\s*=\s*[^\s"'=<>`]+)?/).getRegex(); var q = /(?:\[(?:\\[\s\S]|[^\[\]\\])*\]|\\[\s\S]|`+[^`]*?`+(?!`)|[^\[\]\\`])*?/; -var Ue = d(/^!?\[(label)\]\(\s*(href)(?:(?:[ \t]*(?:\n[ \t]*)?)(title))?\s*\)/).replace("label", q).replace("href", /<(?:\\.|[^\n<>\\])+>|[^ \t\n\x00-\x1f]*/).replace("title", /"(?:\\"?|[^"\\])*"|'(?:\\'?|[^'\\])*'|\((?:\\\)?|[^)\\])*\)/).getRegex(); +var Ke = d(/^!?\[(label)\]\(\s*(href)(?:(?:[ \t]*(?:\n[ \t]*)?)(title))?\s*\)/).replace("label", q).replace("href", /<(?:\\.|[^\n<>\\])+>|[^ \t\n\x00-\x1f]*/).replace("title", /"(?:\\"?|[^"\\])*"|'(?:\\'?|[^'\\])*'|\((?:\\\)?|[^)\\])*\)/).getRegex(); var de = d(/^!?\[(label)\]\[(ref)\]/).replace("label", q).replace("ref", Q).getRegex(); var ke = d(/^!?\[(ref)\](?:\[\])?/).replace("ref", Q).getRegex(); -var Ke = d("reflink|nolink(?!\\()", "g").replace("reflink", de).replace("nolink", ke).getRegex(); +var We = d("reflink|nolink(?!\\()", "g").replace("reflink", de).replace("nolink", ke).getRegex(); var se = /[hH][tT][tT][pP][sS]?|[fF][tT][pP]/; -var X = { _backpedal: I, anyPunctuation: Ne, autolink: Fe, blockSkip: qe, br: le, code: Ae, del: I, emStrongLDelim: ve, emStrongRDelimAst: He, emStrongRDelimUnd: Ge, escape: ze, link: Ue, nolink: ke, punctuation: Ee, reflink: de, reflinkSearch: Ke, tag: Qe, text: Ie, url: I }; -var We = { ...X, link: d(/^!?\[(label)\]\((.*?)\)/).replace("label", q).getRegex(), reflink: d(/^!?\[(label)\]\s*\[([^\]]*)\]/).replace("label", q).getRegex() }; -var N = { ...X, emStrongRDelimAst: Ze, emStrongLDelim: De, url: d(/^((?:protocol):\/\/|www\.)(?:[a-zA-Z0-9\-]+\.?)+[^\s<]*|^email/).replace("protocol", se).replace("email", /[A-Za-z0-9._+-]+(@)[a-zA-Z0-9-_]+(?:\.[a-zA-Z0-9-_]*[a-zA-Z0-9])+(?![-_])/).getRegex(), _backpedal: /(?:[^?!.,:;*_'"~()&]+|\([^)]*\)|&(?![a-zA-Z0-9]+;$)|[?!.,:;*_'"~)]+(?!$))+/, del: /^(~~?)(?=[^\s~])((?:\\[\s\S]|[^\\])*?(?:\\[\s\S]|[^\s~\\]))\1(?=[^~]|$)/, text: d(/^([`~]+|[^`~])(?:(?= {2,}\n)|(?=[a-zA-Z0-9.!#$%&'*+\/=?_`{\|}~-]+@)|[\s\S]*?(?:(?=[\\": ">", '"': """, "'": "'" }; -var ge = (u3) => Je[u3]; -function w(u3, e) { +var X = { _backpedal: E, anyPunctuation: Fe, autolink: je, blockSkip: ve, br: le, code: Ee, del: E, emStrongLDelim: De, emStrongRDelimAst: Ze, emStrongRDelimUnd: Ne, escape: Ae, link: Ke, nolink: ke, punctuation: Ce, reflink: de, reflinkSearch: We, tag: Ue, text: Ie, url: E }; +var Xe = { ...X, link: d(/^!?\[(label)\]\((.*?)\)/).replace("label", q).getRegex(), reflink: d(/^!?\[(label)\]\s*\[([^\]]*)\]/).replace("label", q).getRegex() }; +var N = { ...X, emStrongRDelimAst: Ge, emStrongLDelim: He, url: d(/^((?:protocol):\/\/|www\.)(?:[a-zA-Z0-9\-]+\.?)+[^\s<]*|^email/).replace("protocol", se).replace("email", /[A-Za-z0-9._+-]+(@)[a-zA-Z0-9-_]+(?:\.[a-zA-Z0-9-_]*[a-zA-Z0-9])+(?![-_])/).getRegex(), _backpedal: /(?:[^?!.,:;*_'"~()&]+|\([^)]*\)|&(?![a-zA-Z0-9]+;$)|[?!.,:;*_'"~)]+(?!$))+/, del: /^(~~?)(?=[^\s~])((?:\\[\s\S]|[^\\])*?(?:\\[\s\S]|[^\s~\\]))\1(?=[^~]|$)/, text: d(/^([`~]+|[^`~])(?:(?= {2,}\n)|(?=[a-zA-Z0-9.!#$%&'*+\/=?_`{\|}~-]+@)|[\s\S]*?(?:(?=[\\": ">", '"': """, "'": "'" }; +var ge = (l3) => Ve[l3]; +function w(l3, e) { if (e) { - if (m.escapeTest.test(u3)) return u3.replace(m.escapeReplace, ge); - } else if (m.escapeTestNoEncode.test(u3)) return u3.replace(m.escapeReplaceNoEncode, ge); - return u3; + if (m.escapeTest.test(l3)) return l3.replace(m.escapeReplace, ge); + } else if (m.escapeTestNoEncode.test(l3)) return l3.replace(m.escapeReplaceNoEncode, ge); + return l3; } -function J(u3) { +function J(l3) { try { - u3 = encodeURI(u3).replace(m.percentDecode, "%"); + l3 = encodeURI(l3).replace(m.percentDecode, "%"); } catch { return null; } - return u3; + return l3; } -function V(u3, e) { - let t = u3.replace(m.findPipe, (i, s, o) => { - let a = false, l = s; - for (; --l >= 0 && o[l] === "\\"; ) a = !a; - return a ? "|" : " |"; +function V(l3, e) { + let t = l3.replace(m.findPipe, (i, s, a) => { + let o = false, p = s; + for (; --p >= 0 && a[p] === "\\"; ) o = !o; + return o ? "|" : " |"; }), n = t.split(m.splitPipe), r = 0; if (n[0].trim() || n.shift(), n.length > 0 && !n.at(-1)?.trim() && n.pop(), e) if (n.length > e) n.splice(e); else for (; n.length < e; ) n.push(""); for (; r < n.length; r++) n[r] = n[r].trim().replace(m.slashPipe, "|"); return n; } -function z2(u3, e, t) { - let n = u3.length; +function z2(l3, e, t) { + let n = l3.length; if (n === 0) return ""; let r = 0; for (; r < n; ) { - let i = u3.charAt(n - r - 1); + let i = l3.charAt(n - r - 1); if (i === e && !t) r++; else if (i !== e && t) r++; else break; } - return u3.slice(0, n - r); + return l3.slice(0, n - r); } -function fe(u3, e) { - if (u3.indexOf(e[1]) === -1) return -1; +function fe(l3, e) { + if (l3.indexOf(e[1]) === -1) return -1; let t = 0; - for (let n = 0; n < u3.length; n++) if (u3[n] === "\\") n++; - else if (u3[n] === e[0]) t++; - else if (u3[n] === e[1] && (t--, t < 0)) return n; + for (let n = 0; n < l3.length; n++) if (l3[n] === "\\") n++; + else if (l3[n] === e[0]) t++; + else if (l3[n] === e[1] && (t--, t < 0)) return n; return t > 0 ? -2 : -1; } -function me(u3, e, t, n, r) { - let i = e.href, s = e.title || null, o = u3[1].replace(r.other.outputLinkReplace, "$1"); +function me(l3, e, t, n, r) { + let i = e.href, s = e.title || null, a = l3[1].replace(r.other.outputLinkReplace, "$1"); n.state.inLink = true; - let a = { type: u3[0].charAt(0) === "!" ? "image" : "link", raw: t, href: i, title: s, text: o, tokens: n.inlineTokens(o) }; - return n.state.inLink = false, a; + let o = { type: l3[0].charAt(0) === "!" ? "image" : "link", raw: t, href: i, title: s, text: a, tokens: n.inlineTokens(a) }; + return n.state.inLink = false, o; } -function Ve(u3, e, t) { - let n = u3.match(t.other.indentCodeCompensation); +function Ye(l3, e, t) { + let n = l3.match(t.other.indentCodeCompensation); if (n === null) return e; let r = n[1]; return e.split(` `).map((i) => { let s = i.match(t.other.beginningSpace); if (s === null) return i; - let [o] = s; - return o.length >= r.length ? i.slice(r.length) : i; + let [a] = s; + return a.length >= r.length ? i.slice(r.length) : i; }).join(` `); } @@ -30205,7 +31790,7 @@ var y = class { fences(e) { let t = this.rules.block.fences.exec(e); if (t) { - let n = t[0], r = Ve(n, t[3] || "", this.rules); + let n = t[0], r = Ye(n, t[3] || "", this.rules); return { type: "code", raw: n, lang: t[2] ? t[2].trim().replace(this.rules.inline.anyPunctuation, "$1") : t[2], text: r }; } } @@ -30232,19 +31817,19 @@ var y = class { `).split(` `), r = "", i = "", s = []; for (; n.length > 0; ) { - let o = false, a = [], l; - for (l = 0; l < n.length; l++) if (this.rules.other.blockquoteStart.test(n[l])) a.push(n[l]), o = true; - else if (!o) a.push(n[l]); + let a = false, o = [], p; + for (p = 0; p < n.length; p++) if (this.rules.other.blockquoteStart.test(n[p])) o.push(n[p]), a = true; + else if (!a) o.push(n[p]); else break; - n = n.slice(l); - let c = a.join(` -`), p = c.replace(this.rules.other.blockquoteSetextReplace, ` + n = n.slice(p); + let u = o.join(` +`), c = u.replace(this.rules.other.blockquoteSetextReplace, ` $1`).replace(this.rules.other.blockquoteSetextReplace2, ""); r = r ? `${r} -${c}` : c, i = i ? `${i} -${p}` : p; +${u}` : u, i = i ? `${i} +${c}` : c; let g = this.lexer.state.top; - if (this.lexer.state.top = true, this.lexer.blockTokens(p, s, true), this.lexer.state.top = g, n.length === 0) break; + if (this.lexer.state.top = true, this.lexer.blockTokens(c, s, true), this.lexer.state.top = g, n.length === 0) break; let h = s.at(-1); if (h?.type === "code") break; if (h?.type === "blockquote") { @@ -30270,45 +31855,45 @@ ${p}` : p; if (t) { let n = t[1].trim(), r = n.length > 1, i = { type: "list", raw: "", ordered: r, start: r ? +n.slice(0, -1) : "", loose: false, items: [] }; n = r ? `\\d{1,9}\\${n.slice(-1)}` : `\\${n}`, this.options.pedantic && (n = r ? n : "[*+-]"); - let s = this.rules.other.listItemRegex(n), o = false; + let s = this.rules.other.listItemRegex(n), a = false; for (; e; ) { - let l = false, c = "", p = ""; + let p = false, u = "", c = ""; if (!(t = s.exec(e)) || this.rules.block.hr.test(e)) break; - c = t[0], e = e.substring(c.length); + u = t[0], e = e.substring(u.length); let g = t[2].split(` `, 1)[0].replace(this.rules.other.listReplaceTabs, (H) => " ".repeat(3 * H.length)), h = e.split(` `, 1)[0], R = !g.trim(), f = 0; - if (this.options.pedantic ? (f = 2, p = g.trimStart()) : R ? f = t[1].length + 1 : (f = t[2].search(this.rules.other.nonSpaceChar), f = f > 4 ? 1 : f, p = g.slice(f), f += t[1].length), R && this.rules.other.blankLine.test(h) && (c += h + ` -`, e = e.substring(h.length + 1), l = true), !l) { + if (this.options.pedantic ? (f = 2, c = g.trimStart()) : R ? f = t[1].length + 1 : (f = t[2].search(this.rules.other.nonSpaceChar), f = f > 4 ? 1 : f, c = g.slice(f), f += t[1].length), R && this.rules.other.blankLine.test(h) && (u += h + ` +`, e = e.substring(h.length + 1), p = true), !p) { let H = this.rules.other.nextBulletRegex(f), ee = this.rules.other.hrRegex(f), te = this.rules.other.fencesBeginRegex(f), ne = this.rules.other.headingBeginRegex(f), xe = this.rules.other.htmlBeginRegex(f); for (; e; ) { let Z = e.split(` `, 1)[0], A; if (h = Z, this.options.pedantic ? (h = h.replace(this.rules.other.listReplaceNesting, " "), A = h) : A = h.replace(this.rules.other.tabCharGlobal, " "), te.test(h) || ne.test(h) || xe.test(h) || H.test(h) || ee.test(h)) break; - if (A.search(this.rules.other.nonSpaceChar) >= f || !h.trim()) p += ` + if (A.search(this.rules.other.nonSpaceChar) >= f || !h.trim()) c += ` ` + A.slice(f); else { if (R || g.replace(this.rules.other.tabCharGlobal, " ").search(this.rules.other.nonSpaceChar) >= 4 || te.test(g) || ne.test(g) || ee.test(g)) break; - p += ` + c += ` ` + h; } - !R && !h.trim() && (R = true), c += Z + ` + !R && !h.trim() && (R = true), u += Z + ` `, e = e.substring(Z.length + 1), g = A.slice(f); } } - i.loose || (o ? i.loose = true : this.rules.other.doubleBlankLine.test(c) && (o = true)); + i.loose || (a ? i.loose = true : this.rules.other.doubleBlankLine.test(u) && (a = true)); let O = null, Y; - this.options.gfm && (O = this.rules.other.listIsTask.exec(p), O && (Y = O[0] !== "[ ] ", p = p.replace(this.rules.other.listReplaceTask, ""))), i.items.push({ type: "list_item", raw: c, task: !!O, checked: Y, loose: false, text: p, tokens: [] }), i.raw += c; + this.options.gfm && (O = this.rules.other.listIsTask.exec(c), O && (Y = O[0] !== "[ ] ", c = c.replace(this.rules.other.listReplaceTask, ""))), i.items.push({ type: "list_item", raw: u, task: !!O, checked: Y, loose: false, text: c, tokens: [] }), i.raw += u; } - let a = i.items.at(-1); - if (a) a.raw = a.raw.trimEnd(), a.text = a.text.trimEnd(); + let o = i.items.at(-1); + if (o) o.raw = o.raw.trimEnd(), o.text = o.text.trimEnd(); else return; i.raw = i.raw.trimEnd(); - for (let l = 0; l < i.items.length; l++) if (this.lexer.state.top = false, i.items[l].tokens = this.lexer.blockTokens(i.items[l].text, []), !i.loose) { - let c = i.items[l].tokens.filter((g) => g.type === "space"), p = c.length > 0 && c.some((g) => this.rules.other.anyLine.test(g.raw)); - i.loose = p; + for (let p = 0; p < i.items.length; p++) if (this.lexer.state.top = false, i.items[p].tokens = this.lexer.blockTokens(i.items[p].text, []), !i.loose) { + let u = i.items[p].tokens.filter((g) => g.type === "space"), c = u.length > 0 && u.some((g) => this.rules.other.anyLine.test(g.raw)); + i.loose = c; } - if (i.loose) for (let l = 0; l < i.items.length; l++) i.items[l].loose = true; + if (i.loose) for (let p = 0; p < i.items.length; p++) i.items[p].loose = true; return i; } } @@ -30329,9 +31914,9 @@ ${p}` : p; let n = V(t[1]), r = t[2].replace(this.rules.other.tableAlignChars, "").split("|"), i = t[3]?.trim() ? t[3].replace(this.rules.other.tableRowBlankLine, "").split(` `) : [], s = { type: "table", raw: t[0], header: [], align: [], rows: [] }; if (n.length === r.length) { - for (let o of r) this.rules.other.tableAlignRight.test(o) ? s.align.push("right") : this.rules.other.tableAlignCenter.test(o) ? s.align.push("center") : this.rules.other.tableAlignLeft.test(o) ? s.align.push("left") : s.align.push(null); - for (let o = 0; o < n.length; o++) s.header.push({ text: n[o], tokens: this.lexer.inline(n[o]), header: true, align: s.align[o] }); - for (let o of i) s.rows.push(V(o, s.header.length).map((a, l) => ({ text: a, tokens: this.lexer.inline(a), header: false, align: s.align[l] }))); + for (let a of r) this.rules.other.tableAlignRight.test(a) ? s.align.push("right") : this.rules.other.tableAlignCenter.test(a) ? s.align.push("center") : this.rules.other.tableAlignLeft.test(a) ? s.align.push("left") : s.align.push(null); + for (let a = 0; a < n.length; a++) s.header.push({ text: n[a], tokens: this.lexer.inline(n[a]), header: true, align: s.align[a] }); + for (let a of i) s.rows.push(V(a, s.header.length).map((o, p) => ({ text: o, tokens: this.lexer.inline(o), header: false, align: s.align[p] }))); return s; } } @@ -30371,8 +31956,8 @@ ${p}` : p; let s = fe(t[2], "()"); if (s === -2) return; if (s > -1) { - let a = (t[0].indexOf("!") === 0 ? 5 : 4) + t[1].length + s; - t[2] = t[2].substring(0, s), t[0] = t[0].substring(0, a).trim(), t[3] = ""; + let o = (t[0].indexOf("!") === 0 ? 5 : 4) + t[1].length + s; + t[2] = t[2].substring(0, s), t[0] = t[0].substring(0, o).trim(), t[3] = ""; } } let r = t[2], i = ""; @@ -30398,20 +31983,20 @@ ${p}` : p; let r = this.rules.inline.emStrongLDelim.exec(e); if (!r || r[3] && n.match(this.rules.other.unicodeAlphaNumeric)) return; if (!(r[1] || r[2] || "") || !n || this.rules.inline.punctuation.exec(n)) { - let s = [...r[0]].length - 1, o, a, l = s, c = 0, p = r[0][0] === "*" ? this.rules.inline.emStrongRDelimAst : this.rules.inline.emStrongRDelimUnd; - for (p.lastIndex = 0, t = t.slice(-1 * e.length + s); (r = p.exec(t)) != null; ) { - if (o = r[1] || r[2] || r[3] || r[4] || r[5] || r[6], !o) continue; - if (a = [...o].length, r[3] || r[4]) { - l += a; + let s = [...r[0]].length - 1, a, o, p = s, u = 0, c = r[0][0] === "*" ? this.rules.inline.emStrongRDelimAst : this.rules.inline.emStrongRDelimUnd; + for (c.lastIndex = 0, t = t.slice(-1 * e.length + s); (r = c.exec(t)) != null; ) { + if (a = r[1] || r[2] || r[3] || r[4] || r[5] || r[6], !a) continue; + if (o = [...a].length, r[3] || r[4]) { + p += o; continue; - } else if ((r[5] || r[6]) && s % 3 && !((s + a) % 3)) { - c += a; + } else if ((r[5] || r[6]) && s % 3 && !((s + o) % 3)) { + u += o; continue; } - if (l -= a, l > 0) continue; - a = Math.min(a, a + l + c); - let g = [...r[0]][0].length, h = e.slice(0, s + r.index + g + a); - if (Math.min(s, a) % 2) { + if (p -= o, p > 0) continue; + o = Math.min(o, o + p + u); + let g = [...r[0]][0].length, h = e.slice(0, s + r.index + g + o); + if (Math.min(s, o) % 2) { let f = h.slice(1, -1); return { type: "em", raw: h, text: f, tokens: this.lexer.inlineTokens(f) }; } @@ -30465,7 +32050,7 @@ ${p}` : p; } } }; -var x = class u { +var x = class l { tokens; options; state; @@ -30480,10 +32065,10 @@ var x = class u { return { block: C, inline: M }; } static lex(e, t) { - return new u(t).lex(e); + return new l(t).lex(e); } static lexInline(e, t) { - return new u(t).inlineTokens(e); + return new l(t).inlineTokens(e); } lex(e) { e = e.replace(m.carriageReturn, ` @@ -30557,9 +32142,9 @@ var x = class u { } let i = e; if (this.options.extensions?.startBlock) { - let s = 1 / 0, o = e.slice(1), a; - this.options.extensions.startBlock.forEach((l) => { - a = l.call({ lexer: this }, o), typeof a == "number" && a >= 0 && (s = Math.min(s, a)); + let s = 1 / 0, a = e.slice(1), o; + this.options.extensions.startBlock.forEach((p) => { + o = p.call({ lexer: this }, a), typeof o == "number" && o >= 0 && (s = Math.min(s, o)); }), s < 1 / 0 && s >= 0 && (i = e.substring(0, s + 1)); } if (this.state.top && (r = this.tokenizer.paragraph(i))) { @@ -30599,13 +32184,14 @@ var x = class u { if (o.length > 0) for (; (r = this.tokenizer.rules.inline.reflinkSearch.exec(n)) != null; ) o.includes(r[0].slice(r[0].lastIndexOf("[") + 1, -1)) && (n = n.slice(0, r.index) + "[" + "a".repeat(r[0].length - 2) + "]" + n.slice(this.tokenizer.rules.inline.reflinkSearch.lastIndex)); } for (; (r = this.tokenizer.rules.inline.anyPunctuation.exec(n)) != null; ) n = n.slice(0, r.index) + "++" + n.slice(this.tokenizer.rules.inline.anyPunctuation.lastIndex); - for (; (r = this.tokenizer.rules.inline.blockSkip.exec(n)) != null; ) n = n.slice(0, r.index) + "[" + "a".repeat(r[0].length - 2) + "]" + n.slice(this.tokenizer.rules.inline.blockSkip.lastIndex); + let i; + for (; (r = this.tokenizer.rules.inline.blockSkip.exec(n)) != null; ) i = r[2] ? r[2].length : 0, n = n.slice(0, r.index + i) + "[" + "a".repeat(r[0].length - i - 2) + "]" + n.slice(this.tokenizer.rules.inline.blockSkip.lastIndex); n = this.options.hooks?.emStrongMask?.call({ lexer: this }, n) ?? n; - let i = false, s = ""; + let s = false, a = ""; for (; e; ) { - i || (s = ""), i = false; + s || (a = ""), s = false; let o; - if (this.options.extensions?.inline?.some((l) => (o = l.call({ lexer: this }, e, t)) ? (e = e.substring(o.raw.length), t.push(o), true) : false)) continue; + if (this.options.extensions?.inline?.some((u) => (o = u.call({ lexer: this }, e, t)) ? (e = e.substring(o.raw.length), t.push(o), true) : false)) continue; if (o = this.tokenizer.escape(e)) { e = e.substring(o.raw.length), t.push(o); continue; @@ -30620,11 +32206,11 @@ var x = class u { } if (o = this.tokenizer.reflink(e, this.tokens.links)) { e = e.substring(o.raw.length); - let l = t.at(-1); - o.type === "text" && l?.type === "text" ? (l.raw += o.raw, l.text += o.text) : t.push(o); + let u = t.at(-1); + o.type === "text" && u?.type === "text" ? (u.raw += o.raw, u.text += o.text) : t.push(o); continue; } - if (o = this.tokenizer.emStrong(e, n, s)) { + if (o = this.tokenizer.emStrong(e, n, a)) { e = e.substring(o.raw.length), t.push(o); continue; } @@ -30648,25 +32234,25 @@ var x = class u { e = e.substring(o.raw.length), t.push(o); continue; } - let a = e; + let p = e; if (this.options.extensions?.startInline) { - let l = 1 / 0, c = e.slice(1), p; - this.options.extensions.startInline.forEach((g) => { - p = g.call({ lexer: this }, c), typeof p == "number" && p >= 0 && (l = Math.min(l, p)); - }), l < 1 / 0 && l >= 0 && (a = e.substring(0, l + 1)); - } - if (o = this.tokenizer.inlineText(a)) { - e = e.substring(o.raw.length), o.raw.slice(-1) !== "_" && (s = o.raw.slice(-1)), i = true; - let l = t.at(-1); - l?.type === "text" ? (l.raw += o.raw, l.text += o.text) : t.push(o); + let u = 1 / 0, c = e.slice(1), g; + this.options.extensions.startInline.forEach((h) => { + g = h.call({ lexer: this }, c), typeof g == "number" && g >= 0 && (u = Math.min(u, g)); + }), u < 1 / 0 && u >= 0 && (p = e.substring(0, u + 1)); + } + if (o = this.tokenizer.inlineText(p)) { + e = e.substring(o.raw.length), o.raw.slice(-1) !== "_" && (a = o.raw.slice(-1)), s = true; + let u = t.at(-1); + u?.type === "text" ? (u.raw += o.raw, u.text += o.text) : t.push(o); continue; } if (e) { - let l = "Infinite loop on byte: " + e.charCodeAt(0); + let u = "Infinite loop on byte: " + e.charCodeAt(0); if (this.options.silent) { - console.error(l); + console.error(u); break; - } else throw new Error(l); + } else throw new Error(u); } } return t; @@ -30709,9 +32295,9 @@ ${this.parser.parse(e)} } list(e) { let t = e.ordered, n = e.start, r = ""; - for (let o = 0; o < e.items.length; o++) { - let a = e.items[o]; - r += this.listitem(a); + for (let a = 0; a < e.items.length; a++) { + let o = e.items[a]; + r += this.listitem(o); } let i = t ? "ol" : "ul", s = t && n !== 1 ? ' start="' + n + '"' : ""; return "<" + i + s + `> @@ -30742,7 +32328,7 @@ ${this.parser.parse(e)} for (let i = 0; i < e.rows.length; i++) { let s = e.rows[i]; n = ""; - for (let o = 0; o < s.length; o++) n += this.tablecell(s[o]); + for (let a = 0; a < s.length; a++) n += this.tablecell(s[a]); r += this.tablerow({ text: n }); } return r && (r = `${r}`), ` @@ -30824,7 +32410,7 @@ var $ = class { return ""; } }; -var b = class u2 { +var b = class l2 { options; renderer; textRenderer; @@ -30832,19 +32418,19 @@ var b = class u2 { this.options = e || T, this.options.renderer = this.options.renderer || new P(), this.renderer = this.options.renderer, this.renderer.options = this.options, this.renderer.parser = this, this.textRenderer = new $(); } static parse(e, t) { - return new u2(t).parse(e); + return new l2(t).parse(e); } static parseInline(e, t) { - return new u2(t).parseInline(e); + return new l2(t).parseInline(e); } parse(e, t = true) { let n = ""; for (let r = 0; r < e.length; r++) { let i = e[r]; if (this.options.extensions?.renderers?.[i.type]) { - let o = i, a = this.options.extensions.renderers[o.type].call({ parser: this }, o); - if (a !== false || !["space", "hr", "heading", "code", "table", "blockquote", "list", "html", "def", "paragraph", "text"].includes(o.type)) { - n += a || ""; + let a = i, o = this.options.extensions.renderers[a.type].call({ parser: this }, a); + if (o !== false || !["space", "hr", "heading", "code", "table", "blockquote", "list", "html", "def", "paragraph", "text"].includes(a.type)) { + n += o || ""; continue; } } @@ -30891,16 +32477,16 @@ var b = class u2 { continue; } case "text": { - let o = s, a = this.renderer.text(o); - for (; r + 1 < e.length && e[r + 1].type === "text"; ) o = e[++r], a += ` -` + this.renderer.text(o); - t ? n += this.renderer.paragraph({ type: "paragraph", raw: a, text: a, tokens: [{ type: "text", raw: a, text: a, escaped: true }] }) : n += a; + let a = s, o = this.renderer.text(a); + for (; r + 1 < e.length && e[r + 1].type === "text"; ) a = e[++r], o += ` +` + this.renderer.text(a); + t ? n += this.renderer.paragraph({ type: "paragraph", raw: o, text: o, tokens: [{ type: "text", raw: o, text: o, escaped: true }] }) : n += o; continue; } default: { - let o = 'Token with "' + s.type + '" type was not found.'; - if (this.options.silent) return console.error(o), ""; - throw new Error(o); + let a = 'Token with "' + s.type + '" type was not found.'; + if (this.options.silent) return console.error(a), ""; + throw new Error(a); } } } @@ -30911,9 +32497,9 @@ var b = class u2 { for (let r = 0; r < e.length; r++) { let i = e[r]; if (this.options.extensions?.renderers?.[i.type]) { - let o = this.options.extensions.renderers[i.type].call({ parser: this }, i); - if (o !== false || !["escape", "html", "link", "image", "strong", "em", "codespan", "br", "del", "text"].includes(i.type)) { - n += o || ""; + let a = this.options.extensions.renderers[i.type].call({ parser: this }, i); + if (a !== false || !["escape", "html", "link", "image", "strong", "em", "codespan", "br", "del", "text"].includes(i.type)) { + n += a || ""; continue; } } @@ -30960,9 +32546,9 @@ var b = class u2 { break; } default: { - let o = 'Token with "' + s.type + '" type was not found.'; - if (this.options.silent) return console.error(o), ""; - throw new Error(o); + let a = 'Token with "' + s.type + '" type was not found.'; + if (this.options.silent) return console.error(a), ""; + throw new Error(a); } } } @@ -31016,7 +32602,7 @@ var B = class { case "table": { let i = r; for (let s of i.header) n = n.concat(this.walkTokens(s.tokens, t)); - for (let s of i.rows) for (let o of s) n = n.concat(this.walkTokens(o.tokens, t)); + for (let s of i.rows) for (let a of s) n = n.concat(this.walkTokens(a.tokens, t)); break; } case "list": { @@ -31027,8 +32613,8 @@ var B = class { default: { let i = r; this.defaults.extensions?.childTokens?.[i.type] ? this.defaults.extensions.childTokens[i.type].forEach((s) => { - let o = i[s].flat(1 / 0); - n = n.concat(this.walkTokens(o, t)); + let a = i[s].flat(1 / 0); + n = n.concat(this.walkTokens(a, t)); }) : i.tokens && (n = n.concat(this.walkTokens(i.tokens, t))); } } @@ -31042,9 +32628,9 @@ var B = class { if (!i.name) throw new Error("extension name required"); if ("renderer" in i) { let s = t.renderers[i.name]; - s ? t.renderers[i.name] = function(...o) { - let a = i.renderer.apply(this, o); - return a === false && (a = s.apply(this, o)), a; + s ? t.renderers[i.name] = function(...a) { + let o = i.renderer.apply(this, a); + return o === false && (o = s.apply(this, a)), o; } : t.renderers[i.name] = i.renderer; } if ("tokenizer" in i) { @@ -31058,10 +32644,10 @@ var B = class { for (let s in n.renderer) { if (!(s in i)) throw new Error(`renderer '${s}' does not exist`); if (["options", "parser"].includes(s)) continue; - let o = s, a = n.renderer[o], l = i[o]; - i[o] = (...c) => { - let p = a.apply(i, c); - return p === false && (p = l.apply(i, c)), p || ""; + let a = s, o = n.renderer[a], p = i[a]; + i[a] = (...u) => { + let c = o.apply(i, u); + return c === false && (c = p.apply(i, u)), c || ""; }; } r.renderer = i; @@ -31071,10 +32657,10 @@ var B = class { for (let s in n.tokenizer) { if (!(s in i)) throw new Error(`tokenizer '${s}' does not exist`); if (["options", "rules", "lexer"].includes(s)) continue; - let o = s, a = n.tokenizer[o], l = i[o]; - i[o] = (...c) => { - let p = a.apply(i, c); - return p === false && (p = l.apply(i, c)), p; + let a = s, o = n.tokenizer[a], p = i[a]; + i[a] = (...u) => { + let c = o.apply(i, u); + return c === false && (c = p.apply(i, u)), c; }; } r.tokenizer = i; @@ -31084,30 +32670,30 @@ var B = class { for (let s in n.hooks) { if (!(s in i)) throw new Error(`hook '${s}' does not exist`); if (["options", "block"].includes(s)) continue; - let o = s, a = n.hooks[o], l = i[o]; - S.passThroughHooks.has(s) ? i[o] = (c) => { + let a = s, o = n.hooks[a], p = i[a]; + S.passThroughHooks.has(s) ? i[a] = (u) => { if (this.defaults.async && S.passThroughHooksRespectAsync.has(s)) return (async () => { - let g = await a.call(i, c); - return l.call(i, g); + let g = await o.call(i, u); + return p.call(i, g); })(); - let p = a.call(i, c); - return l.call(i, p); - } : i[o] = (...c) => { + let c = o.call(i, u); + return p.call(i, c); + } : i[a] = (...u) => { if (this.defaults.async) return (async () => { - let g = await a.apply(i, c); - return g === false && (g = await l.apply(i, c)), g; + let g = await o.apply(i, u); + return g === false && (g = await p.apply(i, u)), g; })(); - let p = a.apply(i, c); - return p === false && (p = l.apply(i, c)), p; + let c = o.apply(i, u); + return c === false && (c = p.apply(i, u)), c; }; } r.hooks = i; } if (n.walkTokens) { let i = this.defaults.walkTokens, s = n.walkTokens; - r.walkTokens = function(o) { - let a = []; - return a.push(s.call(this, o)), i && (a = a.concat(i.call(this, o))), a; + r.walkTokens = function(a) { + let o = []; + return o.push(s.call(this, a)), i && (o = o.concat(i.call(this, a))), o; }; } this.defaults = { ...this.defaults, ...r }; @@ -31124,24 +32710,24 @@ var B = class { } parseMarkdown(e) { return (n, r) => { - let i = { ...r }, s = { ...this.defaults, ...i }, o = this.onError(!!s.silent, !!s.async); - if (this.defaults.async === true && i.async === false) return o(new Error("marked(): The async option was set to true by an extension. Remove async: false from the parse options object to return a Promise.")); - if (typeof n > "u" || n === null) return o(new Error("marked(): input parameter is undefined or null")); - if (typeof n != "string") return o(new Error("marked(): input parameter is of type " + Object.prototype.toString.call(n) + ", string expected")); + let i = { ...r }, s = { ...this.defaults, ...i }, a = this.onError(!!s.silent, !!s.async); + if (this.defaults.async === true && i.async === false) return a(new Error("marked(): The async option was set to true by an extension. Remove async: false from the parse options object to return a Promise.")); + if (typeof n > "u" || n === null) return a(new Error("marked(): input parameter is undefined or null")); + if (typeof n != "string") return a(new Error("marked(): input parameter is of type " + Object.prototype.toString.call(n) + ", string expected")); if (s.hooks && (s.hooks.options = s, s.hooks.block = e), s.async) return (async () => { - let a = s.hooks ? await s.hooks.preprocess(n) : n, c = await (s.hooks ? await s.hooks.provideLexer() : e ? x.lex : x.lexInline)(a, s), p = s.hooks ? await s.hooks.processAllTokens(c) : c; - s.walkTokens && await Promise.all(this.walkTokens(p, s.walkTokens)); - let h = await (s.hooks ? await s.hooks.provideParser() : e ? b.parse : b.parseInline)(p, s); + let o = s.hooks ? await s.hooks.preprocess(n) : n, u = await (s.hooks ? await s.hooks.provideLexer() : e ? x.lex : x.lexInline)(o, s), c = s.hooks ? await s.hooks.processAllTokens(u) : u; + s.walkTokens && await Promise.all(this.walkTokens(c, s.walkTokens)); + let h = await (s.hooks ? await s.hooks.provideParser() : e ? b.parse : b.parseInline)(c, s); return s.hooks ? await s.hooks.postprocess(h) : h; - })().catch(o); + })().catch(a); try { s.hooks && (n = s.hooks.preprocess(n)); - let l = (s.hooks ? s.hooks.provideLexer() : e ? x.lex : x.lexInline)(n, s); - s.hooks && (l = s.hooks.processAllTokens(l)), s.walkTokens && this.walkTokens(l, s.walkTokens); - let p = (s.hooks ? s.hooks.provideParser() : e ? b.parse : b.parseInline)(l, s); - return s.hooks && (p = s.hooks.postprocess(p)), p; - } catch (a) { - return o(a); + let p = (s.hooks ? s.hooks.provideLexer() : e ? x.lex : x.lexInline)(n, s); + s.hooks && (p = s.hooks.processAllTokens(p)), s.walkTokens && this.walkTokens(p, s.walkTokens); + let c = (s.hooks ? s.hooks.provideParser() : e ? b.parse : b.parseInline)(p, s); + return s.hooks && (c = s.hooks.postprocess(c)), c; + } catch (o) { + return a(o); } }; } @@ -31158,19 +32744,19 @@ Please report this to https://github.com/markedjs/marked.`, e) { } }; var _ = new B(); -function k(u3, e) { - return _.parse(u3, e); +function k(l3, e) { + return _.parse(l3, e); } -k.options = k.setOptions = function(u3) { - return _.setOptions(u3), k.defaults = _.defaults, G(k.defaults), k; +k.options = k.setOptions = function(l3) { + return _.setOptions(l3), k.defaults = _.defaults, G(k.defaults), k; }; k.getDefaults = L; k.defaults = T; -k.use = function(...u3) { - return _.use(...u3), k.defaults = _.defaults, G(k.defaults), k; +k.use = function(...l3) { + return _.use(...l3), k.defaults = _.defaults, G(k.defaults), k; }; -k.walkTokens = function(u3, e) { - return _.walkTokens(u3, e); +k.walkTokens = function(l3, e) { + return _.walkTokens(l3, e); }; k.parseInline = _.parseInline; k.Parser = b; @@ -31182,21 +32768,22 @@ k.lexer = x.lex; k.Tokenizer = y; k.Hooks = S; k.parse = k; -var Ht = k.options; -var Zt = k.setOptions; -var Gt = k.use; -var Nt = k.walkTokens; -var Ft = k.parseInline; -var Qt = b.parse; -var Ut = x.lex; +var Zt = k.options; +var Gt = k.setOptions; +var Nt = k.use; +var Ft = k.walkTokens; +var jt = k.parseInline; +var Ut = b.parse; +var Kt = x.lex; // src/show.ts function formatConversationAsMarkdown(jsonl, startLine, endLine) { - const allLines = jsonl.trim().split("\n").filter((line) => line.trim()); - const lines = startLine !== void 0 || endLine !== void 0 ? allLines.slice( + const allLines = jsonl.split("\n"); + const selectedLines = startLine !== void 0 || endLine !== void 0 ? allLines.slice( startLine !== void 0 ? startLine - 1 : 0, endLine !== void 0 ? endLine : void 0 ) : allLines; + const lines = selectedLines.filter((line) => line.trim()); if (isCodexRollout(lines)) { return formatCodexConversationAsMarkdown(lines); } @@ -31861,7 +33448,7 @@ function formatCursorConversationAsMarkdown(lines) { var VERSION = "1.6.0"; // src/mcp-server.ts -import fs4 from "fs"; +import fs5 from "fs"; var SearchModeEnum = external_exports.enum(["vector", "text", "both"]); var ResponseFormatEnum = external_exports.enum(["markdown", "json"]); var SearchInputSchema = external_exports.object({ @@ -31892,11 +33479,11 @@ var ShowConversationInputSchema = external_exports.object({ startLine: external_exports.number().int().min(1).optional().describe("Starting line number (1-indexed, inclusive). Omit to start from beginning."), endLine: external_exports.number().int().min(1).optional().describe("Ending line number (1-indexed, inclusive). Omit to read to end.") }).strict(); -function handleError(error61) { - if (error61 instanceof Error) { - return `Error: ${error61.message}`; +function handleError(error62) { + if (error62 instanceof Error) { + return `Error: ${error62.message}`; } - return `Error: ${String(error61)}`; + return `Error: ${String(error62)}`; } var server = new Server( { @@ -32040,10 +33627,10 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => { } if (name === "read") { const params = ShowConversationInputSchema.parse(args); - if (!fs4.existsSync(params.path)) { + if (!fs5.existsSync(params.path)) { throw new Error(`File not found: ${params.path}`); } - const jsonlContent = fs4.readFileSync(params.path, "utf-8"); + const jsonlContent = await readAdmittedConversation(params.path, params.startLine, params.endLine); const markdownContent = formatConversationAsMarkdown( jsonlContent, params.startLine, @@ -32059,12 +33646,12 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => { }; } throw new Error(`Unknown tool: ${name}`); - } catch (error61) { + } catch (error62) { return { content: [ { type: "text", - text: handleError(error61) + text: handleError(error62) } ], isError: true @@ -32076,7 +33663,7 @@ async function main() { const transport = new StdioServerTransport(); await server.connect(transport); } -main().catch((error61) => { - console.error("Server error:", error61); +main().catch((error62) => { + console.error("Server error:", error62); process.exit(1); }); diff --git a/dist/opencode-sync.d.ts b/dist/opencode-sync.d.ts index fff32c4c..64215225 100644 --- a/dist/opencode-sync.d.ts +++ b/dist/opencode-sync.d.ts @@ -1,3 +1,4 @@ +import { type Redactor } from './redaction.js'; export interface OpencodeExportResult { exported: number; skipped: number; @@ -15,4 +16,6 @@ export declare function getOpencodeTranscriptFilePath(transcriptDir: string, inp export declare function exportOpencodeSessions(options?: { dbPath?: string; transcriptDir?: string; + /** `undefined` loads from the environment (strict mode throws on bad rules); `null` disables. */ + redactor?: Redactor | null; }): OpencodeExportResult; diff --git a/dist/opencode-sync.js b/dist/opencode-sync.js index 2f90ab75..33916055 100644 --- a/dist/opencode-sync.js +++ b/dist/opencode-sync.js @@ -2,6 +2,7 @@ import fs from 'fs'; import path from 'path'; import Database from 'better-sqlite3'; import { getOpencodeDbPath, getOpencodeTranscriptDir } from './paths.js'; +import { loadRedactor, redactJsonlLine } from './redaction.js'; function safeParseJson(value) { if (!value) return undefined; @@ -40,7 +41,7 @@ function shouldExportSession(filePath, sessionUpdatedMs) { const stat = fs.statSync(filePath); return Math.floor(stat.mtimeMs) < Math.floor(sessionUpdatedMs); } -function writeSessionTranscript(db, session, filePath) { +function writeSessionTranscript(db, session, filePath, redactor) { const messages = db.prepare(` SELECT id, session_id, time_created, time_updated, data FROM message @@ -115,14 +116,20 @@ function writeSessionTranscript(db, session, filePath) { parts, })); } + // The staging transcript is a plugin-owned plaintext copy, so redact it at + // write time like the archive (docs/REDACTION.md). + const output = redactor + ? lines.map(line => redactJsonlLine(line, redactor, { source: 'opencode', path: filePath })) + : lines; fs.mkdirSync(path.dirname(filePath), { recursive: true }); const tempPath = `${filePath}.tmp.${process.pid}`; - fs.writeFileSync(tempPath, `${lines.join('\n')}\n`, 'utf-8'); + fs.writeFileSync(tempPath, `${output.join('\n')}\n`, 'utf-8'); fs.renameSync(tempPath, filePath); const mtime = dateFromMillis(session.time_updated); fs.utimesSync(filePath, mtime, mtime); } export function exportOpencodeSessions(options = {}) { + const redactor = options.redactor === undefined ? loadRedactor() : options.redactor; const dbPath = options.dbPath || getOpencodeDbPath(); const transcriptDir = options.transcriptDir || getOpencodeTranscriptDir(); const result = { @@ -168,7 +175,7 @@ export function exportOpencodeSessions(options = {}) { result.skipped++; continue; } - writeSessionTranscript(db, session, filePath); + writeSessionTranscript(db, session, filePath, redactor); result.exported++; } catch (error) { diff --git a/dist/parser.js b/dist/parser.js index dcc1c270..10aa56a9 100644 --- a/dist/parser.js +++ b/dist/parser.js @@ -1,13 +1,9 @@ import fs from 'fs'; -import readline from 'readline'; import path from 'path'; import crypto from 'crypto'; +import { admittedConversationLines } from './record-admission.js'; async function detectConversationHarness(filePath) { - const fileStream = fs.createReadStream(filePath); - const rl = readline.createInterface({ - input: fileStream, - crlfDelay: Infinity - }); + const rl = admittedConversationLines(filePath); for await (const line of rl) { if (!line.trim()) continue; @@ -73,11 +69,7 @@ export async function parseConversation(filePath, projectName, archivePath) { } async function parseClaudeConversation(filePath, projectName, archivePath) { const exchanges = []; - const fileStream = fs.createReadStream(filePath); - const rl = readline.createInterface({ - input: fileStream, - crlfDelay: Infinity - }); + const rl = admittedConversationLines(filePath); let lineNumber = 0; let currentExchange = null; const finalizeExchange = () => { @@ -343,11 +335,7 @@ function extractOpencodeToolCalls(parts, fallbackTimestamp) { } async function parseOpencodeConversation(filePath, projectName, archivePath) { const exchanges = []; - const fileStream = fs.createReadStream(filePath); - const rl = readline.createInterface({ - input: fileStream, - crlfDelay: Infinity - }); + const rl = admittedConversationLines(filePath); let lineNumber = 0; let sessionId; let cwd; @@ -524,11 +512,7 @@ function extractOmpText(content) { */ async function parseOmpConversation(filePath, projectName, archivePath) { const exchanges = []; - const fileStream = fs.createReadStream(filePath); - const rl = readline.createInterface({ - input: fileStream, - crlfDelay: Infinity - }); + const rl = admittedConversationLines(filePath); let sessionId; let cwd; let headerTimestamp; @@ -632,11 +616,7 @@ async function parseOmpConversation(filePath, projectName, archivePath) { } async function parseCodexConversation(filePath, projectName, archivePath) { const exchanges = []; - const fileStream = fs.createReadStream(filePath); - const rl = readline.createInterface({ - input: fileStream, - crlfDelay: Infinity - }); + const rl = admittedConversationLines(filePath); let lineNumber = 0; let sessionId; let cwd; @@ -763,7 +743,8 @@ async function parseCodexConversation(filePath, projectName, archivePath) { const payload = parsed.payload; const timestamp = parsed.timestamp || new Date().toISOString(); if (parsed.type === 'session_meta' && payload) { - sessionId = payload.id || sessionId; + // Forked transcripts may retain ancestor headers after the primary one. + sessionId = sessionId || payload.id; cwd = payload.cwd || cwd; gitBranch = payload.git?.branch || gitBranch; agentVersion = payload.cli_version || agentVersion; @@ -884,13 +865,9 @@ function cursorProjectFromPath(filePath) { const UUID_PATTERN = /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/i; async function parseCursorConversation(filePath, projectName, archivePath) { const exchanges = []; - const fileStream = fs.createReadStream(filePath); - const rl = readline.createInterface({ - input: fileStream, - crlfDelay: Infinity - }); + const rl = admittedConversationLines(filePath); // Live Cursor transcripts carry no per-message timestamps; fall back to the - // file mtime (preserved from the source by sync's copyIfNewer). Legacy + // file mtime (preserved from the source by the archive writer). Legacy // exports from import-cursor-history embed real per-message timestamps. let fallbackTimestamp; try { diff --git a/dist/record-admission.d.ts b/dist/record-admission.d.ts new file mode 100644 index 00000000..39e2e71b --- /dev/null +++ b/dist/record-admission.d.ts @@ -0,0 +1,44 @@ +import type { ConversationExchange } from './types.js'; +import { type FindingsTally, type Redactor } from './redaction.js'; +/** Scan decoded instruction records without retaining the whole transcript. */ +export declare function shouldSkipConversation(filePath: string): boolean; +interface ExchangeExclusion { + session_id: string; + transcript: string; + line_start: number; + line_end: number; +} +interface ToolExclusion { + session_id: string; + transcript: string; + call_id: string; +} +export interface RecordExclusions { + version: 1; + exchanges: ExchangeExclusion[]; + tool_calls: ToolExclusion[]; + screening?: { + engine: 'gitleaks'; + executable: string; + sha256: string; + timeout_ms: number; + max_record_bytes: number; + }; +} +export declare function readRecordExclusions(): RecordExclusions | null; +export declare function excludedSession(policy: RecordExclusions | null, sessionId?: string): boolean; +/** Read original inputs without mutation, preserving their physical line numbers. */ +export declare function admittedConversationLines(file: string, policy?: RecordExclusions | null): AsyncGenerator; +/** Read the admitted display input without changing original physical coordinates. */ +export declare function readAdmittedConversation(file: string, startLine?: number, endLine?: number): Promise; +/** Enforce the same rule for direct callers that bypass transcript parsing. */ +export declare function admitExchange(exchange: ConversationExchange, policy?: RecordExclusions | null): ConversationExchange | null; +/** Publish only admitted bytes. Never write an unfiltered temporary copy. */ +export declare function archiveAdmittedConversation(source: string, destination: string, policy?: RecordExclusions | null, redactor?: Redactor | null, tally?: FindingsTally): Promise; +/** Request-local admission: retained search rows are not trusted merely because they were indexed. */ +export declare function createSearchAdmission(): { + admit(exchange: ConversationExchange): ConversationExchange | null; + summary(exchange: ConversationExchange, text: string): string | undefined; + verify(): void; +}; +export {}; diff --git a/dist/record-admission.js b/dist/record-admission.js new file mode 100644 index 00000000..9b70a996 --- /dev/null +++ b/dist/record-admission.js @@ -0,0 +1,653 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import crypto from 'node:crypto'; +import os from 'node:os'; +import { spawnSync } from 'node:child_process'; +import { SUMMARIZER_CONTEXT_MARKER } from './constants.js'; +import { getSuperpowersDir } from './paths.js'; +import { redactJsonlLine } from './redaction.js'; +const EXCLUSION_MARKERS = [ + 'DO NOT INDEX THIS CHAT', + 'Only use NO_INSIGHTS_FOUND', + SUMMARIZER_CONTEXT_MARKER, +]; +const MARKER_SCAN_CHUNK_BYTES = 1 << 20; // 1 MiB +/** Scan decoded instruction records without retaining the whole transcript. */ +export function shouldSkipConversation(filePath) { + let fd; + try { + fd = fs.openSync(filePath, fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW); + const initial = fs.fstatSync(fd); + const excluded = descriptorHasExclusionMarker(fd); + assertUnchangedInput(filePath, initial); + return excluded; + } + catch { + return true; + } + finally { + if (fd !== undefined) + fs.closeSync(fd); + } +} +function instructionText(record) { + if (!object(record)) + return ''; + // Codex records both the response item and its user-message event. + if (record.type === 'event_msg' && object(record.payload)) { + return record.payload.type === 'user_message' && typeof record.payload.message === 'string' + ? record.payload.message : ''; + } + const message = record.type === 'response_item' ? record.payload : record.message; + if (!object(message) || (message.role ?? record.role) !== 'user') + return ''; + const content = record.type === 'opencode_message' ? record.parts : message.content; + if (typeof content === 'string') + return content; + if (!Array.isArray(content)) + return ''; + // Claude tool results are wrapped in a user envelope, but are not instructions. + return content.filter(part => object(part) && ['text', 'input_text'].includes(String(part.type)) + && typeof part.text === 'string').map(part => part.text).join('\n'); +} +function descriptorHasExclusionMarker(fd, byteLimit = Infinity, digest) { + if (!fs.fstatSync(fd).isFile()) + throw new Error('Conversation input must be a regular file'); + const buffer = Buffer.allocUnsafe(MARKER_SCAN_CHUNK_BYTES); + const decoder = new TextDecoder('utf-8', { fatal: true }); + let pending = Buffer.alloc(0); + let position = 0; + const excluded = (bytes) => { + const line = decoder.decode(bytes); + let text; + try { + text = instructionText(JSON.parse(line)); + } + catch { + text = line; + } // Preserve conservative handling of legacy raw transcripts. + return EXCLUSION_MARKERS.some(marker => text.includes(marker)); + }; + let count; + while (position < byteLimit && (count = fs.readSync(fd, buffer, 0, Math.min(buffer.length, byteLimit - position), position)) > 0) { + position += count; + digest?.update(buffer.subarray(0, count)); + pending = Buffer.concat([pending, buffer.subarray(0, count)]); + let newline; + while ((newline = pending.indexOf(10)) !== -1) { + if (newline > 64 * 1024 * 1024) + throw new Error('Conversation record exceeds admission byte limit'); + if (excluded(pending.subarray(0, newline))) + return true; + pending = pending.subarray(newline + 1); + } + if (pending.length > 64 * 1024 * 1024) + throw new Error('Conversation record exceeds admission byte limit'); + } + if (position < byteLimit && byteLimit !== Infinity) + throw new Error('Conversation changed during record admission'); + return pending.length > 0 && excluded(pending); +} +function assertUnchangedInput(file, initial) { + const current = fs.lstatSync(file); + if (!current.isFile() || initial.dev !== current.dev || initial.ino !== current.ino || + initial.size !== current.size || initial.mtimeMs !== current.mtimeMs || initial.ctimeMs !== current.ctimeMs) { + throw new Error('Conversation changed during record admission'); + } +} +function assertUnchangedSearchPrefix(file, initial, expectedDigest) { + // A live transcript may grow after the search snapshot. Rehash the pinned + // prefix only when metadata changes, so appended bytes cannot hide a rewrite. + const changed = () => new Error('Conversation changed during record admission'); + let current; + try { + current = fs.lstatSync(file); + } + catch { + throw changed(); + } + if (!current.isFile() || initial.dev !== current.dev || initial.ino !== current.ino || current.size < initial.size) { + throw changed(); + } + if (current.size === initial.size && current.mtimeMs === initial.mtimeMs && current.ctimeMs === initial.ctimeMs) + return; + let fd; + try { + fd = fs.openSync(file, fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW); + const opened = fs.fstatSync(fd); + if (!opened.isFile() || opened.dev !== initial.dev || opened.ino !== initial.ino || opened.size < initial.size) + throw changed(); + const digest = crypto.createHash('sha256'); + const buffer = Buffer.allocUnsafe(MARKER_SCAN_CHUNK_BYTES); + let position = 0; + while (position < initial.size) { + const count = fs.readSync(fd, buffer, 0, Math.min(buffer.length, initial.size - position), position); + if (count <= 0) + throw changed(); + digest.update(buffer.subarray(0, count)); + position += count; + } + if (digest.digest('hex') !== expectedDigest) + throw changed(); + current = fs.lstatSync(file); + if (!current.isFile() || current.dev !== initial.dev || current.ino !== initial.ino || current.size < initial.size) + throw changed(); + } + catch { + throw changed(); + } + finally { + if (fd !== undefined) + fs.closeSync(fd); + } +} +function assertUnchangedPolicy(expected) { + if (JSON.stringify(readRecordExclusions()) !== expected) { + throw new Error('Record exclusion policy changed during record admission'); + } +} +function prepareArchiveParent(directory) { + try { + const info = fs.lstatSync(directory); + if (!info.isDirectory() || info.isSymbolicLink()) + throw new Error('Archive parent must be a non-symlink directory'); + } + catch (error) { + if (error.code !== 'ENOENT') + throw error; + const parent = path.dirname(directory); + if (parent === directory) + throw error; + prepareArchiveParent(parent); + fs.mkdirSync(directory, { mode: 0o700 }); + } + return fs.realpathSync(directory); +} +const POLICY_ERROR = 'Invalid or unreadable record exclusion policy'; +const UUID = /[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}/ig; +function object(value) { + return value !== null && typeof value === 'object' && !Array.isArray(value); +} +function keys(value, expected) { + return Object.keys(value).length === expected.length && expected.every(key => key in value); +} +function transcriptName(value) { + return identity(value) && !/[\\/]/.test(value) && value.endsWith('.jsonl') && value !== '.jsonl'; +} +function identity(value) { + return typeof value === 'string' && value.length > 0 && value.length <= 512 && !/[\x00-\x1f]/.test(value); +} +/** Keep scanner inputs local to one record, including JSON-escaped text. */ +function screeningText(value, depth = 0) { + if (depth > 24) + throw new Error('Record screening nesting limit exceeded'); + if (typeof value === 'string') { + let decoded; + try { + decoded = JSON.parse(value); + } + catch { + return value; + } + if (typeof decoded === 'string' || object(decoded) || Array.isArray(decoded)) { + return screeningText(decoded, depth + 1); + } + return value; + } + if (Array.isArray(value)) + return value.map(item => screeningText(item, depth + 1)).join('\n'); + if (object(value)) { + // Generic credential rules need an assignment delimiter, not just a nearby + // keyword. Decode nested text without destroying its key/value relation. + return Object.entries(value).map(([key, item]) => `${key} = ${screeningText(item, depth + 1)}`).join('\n'); + } + return value == null ? '' : String(value); +} +/** One pinned, short-lived local scanner context; no secret-bearing cache. */ +function createRecordScreener(policy) { + const config = policy?.screening; + if (!config) + return { scan() { }, close() { } }; + const unavailable = () => new Error('Record screening unavailable'); + let initial; + let fd; + try { + if (fs.realpathSync(config.executable) !== config.executable) + throw unavailable(); + fd = fs.openSync(config.executable, fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW); + initial = fs.fstatSync(fd); + if (!initial.isFile() || initial.size < 1 || initial.size > 134_217_728 || + (initial.mode & 0o022) !== 0 || + (typeof process.getuid === 'function' && initial.uid !== 0 && initial.uid !== process.getuid())) { + throw unavailable(); + } + fs.accessSync(config.executable, fs.constants.X_OK); + const digest = crypto.createHash('sha256').update(fs.readFileSync(fd)).digest('hex'); + if (digest !== config.sha256) + throw unavailable(); + } + catch { + throw unavailable(); + } + finally { + if (fd !== undefined) + fs.closeSync(fd); + } + let directory; + try { + directory = fs.mkdtempSync(path.join(os.tmpdir(), 'episodic-screen-')); + fs.chmodSync(directory, 0o700); + } + catch { + throw unavailable(); + } + const unchanged = () => { + try { + const current = fs.lstatSync(config.executable); + if (!current.isFile() || current.dev !== initial.dev || current.ino !== initial.ino || + current.size !== initial.size || current.mtimeMs !== initial.mtimeMs || + current.ctimeMs !== initial.ctimeMs) + throw unavailable(); + } + catch { + throw unavailable(); + } + }; + return { + scan(value) { + const raw = typeof value === 'string' ? value : JSON.stringify(value); + if (raw === undefined) + return; + if (Buffer.byteLength(raw) > config.max_record_bytes) + throw new Error('Record screening byte limit exceeded'); + const input = screeningText(value); + if (Buffer.byteLength(input) > config.max_record_bytes) + throw new Error('Record screening byte limit exceeded'); + if (!input.trim()) + return; + unchanged(); + const result = spawnSync(config.executable, [ + 'stdin', '--redact', '--no-banner', '--log-level', 'error', + '--ignore-gitleaks-allow', '--exit-code', '10', '--report-format', 'json', '--report-path', '-', + ], { + input, encoding: 'utf8', cwd: directory, shell: false, + env: { HOME: directory, USERPROFILE: directory, PATH: path.dirname(config.executable) }, + timeout: config.timeout_ms, killSignal: 'SIGKILL', maxBuffer: 1_048_576, + windowsHide: true, + }); + unchanged(); + // Exit zero alone is not a clean scan. Validate the full report and never + // forward stdout/stderr, findings, fingerprints or input into errors. + if (result.error || result.signal || ![0, 10].includes(result.status ?? -1)) + throw unavailable(); + let report; + try { + report = JSON.parse(result.stdout); + } + catch { + throw unavailable(); + } + if (!Array.isArray(report) || (result.status === 0 && report.length !== 0) || + (result.status === 10 && (report.length === 0 || !report.every(item => object(item) && typeof item.RuleID === 'string')))) { + throw unavailable(); + } + if (result.status === 10) + throw new Error('Record content rejected by screening'); + }, + close() { fs.rmSync(directory, { recursive: true, force: true }); }, + }; +} +export function readRecordExclusions() { + const file = path.join(getSuperpowersDir(), 'record-exclusions.json'); + let fd; + try { + fd = fs.openSync(file, fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW); + const info = fs.fstatSync(fd); + if (!info.isFile() || info.size > 4 * 1024 * 1024) + throw new Error(POLICY_ERROR); + const value = JSON.parse(fs.readFileSync(fd, 'utf8')); + if (!object(value) || !keys(value, 'screening' in value + ? ['version', 'exchanges', 'tool_calls', 'screening'] : ['version', 'exchanges', 'tool_calls']) || value.version !== 1 || + !Array.isArray(value.exchanges) || !Array.isArray(value.tool_calls)) + throw new Error(POLICY_ERROR); + if ('screening' in value) { + const s = value.screening; + if (!object(s) || !keys(s, ['engine', 'executable', 'sha256', 'timeout_ms', 'max_record_bytes']) || + s.engine !== 'gitleaks' || typeof s.executable !== 'string' || !path.isAbsolute(s.executable) || + /[\x00-\x1f]/.test(s.executable) || typeof s.sha256 !== 'string' || !/^[a-f0-9]{64}$/.test(s.sha256) || + !Number.isSafeInteger(s.timeout_ms) || Number(s.timeout_ms) < 50 || Number(s.timeout_ms) > 30_000 || + !Number.isSafeInteger(s.max_record_bytes) || Number(s.max_record_bytes) < 1 || Number(s.max_record_bytes) > 8_388_608) { + throw new Error(POLICY_ERROR); + } + } + for (const row of value.exchanges) { + if (!object(row) || !keys(row, ['session_id', 'transcript', 'line_start', 'line_end']) || !identity(row.session_id) || !transcriptName(row.transcript) || + !Number.isSafeInteger(row.line_start) || !Number.isSafeInteger(row.line_end) || + row.line_start < 1 || row.line_end < row.line_start) + throw new Error(POLICY_ERROR); + } + for (const row of value.tool_calls) { + if (!object(row) || !keys(row, ['session_id', 'transcript', 'call_id']) || !identity(row.session_id) || !transcriptName(row.transcript) || !identity(row.call_id)) + throw new Error(POLICY_ERROR); + } + return value; + } + catch (error) { + if (fd === undefined && error.code === 'ENOENT') + return null; + throw new Error(POLICY_ERROR); + } + finally { + if (fd !== undefined) + fs.closeSync(fd); + } +} +export function excludedSession(policy, sessionId) { + return !!sessionId && !!policy && (policy.exchanges.some(x => x.session_id === sessionId) || + policy.tool_calls.some(x => x.session_id === sessionId)); +} +function sessionFromPath(file) { + return path.basename(file).match(UUID)?.at(-1); +} +function sessionFromRecord(value) { + if (value?.type === 'session_meta') + return value.payload?.id; + if (value?.type === 'session') + return value.id; + if (value?.type === 'opencode_session') + return value.session?.id; + return value?.sessionId ?? value?.sessionID ?? value?.message?.sessionID; +} +function blockedCall(policy, sessionId, transcript, callId) { + return typeof callId === 'string' && policy.tool_calls.some(x => x.session_id === sessionId && x.transcript === transcript && x.call_id === callId); +} +function filterRecord(line, number, policy, transcript, sessionId) { + if (!policy || !sessionId) + return line; + if (policy.exchanges.some(x => x.session_id === sessionId && x.transcript === transcript && number >= x.line_start && number <= x.line_end)) + return ''; + let value; + try { + value = JSON.parse(line); + } + catch { + return line; + } + if (blockedCall(policy, sessionId, transcript, value?.payload?.call_id) || blockedCall(policy, sessionId, transcript, value?.call_id)) + return ''; + // Claude and Cursor may combine safe text and tool payloads in one physical record. + // Preserve the record and safe blocks rather than excluding the entire exchange. + const content = value?.message?.content; + if (Array.isArray(content)) { + const kept = content.filter(block => !blockedCall(policy, sessionId, transcript, block?.id ?? block?.tool_use_id)); + if (kept.length !== content.length) { + return JSON.stringify({ ...value, message: { ...value.message, content: kept } }); + } + } + if (Array.isArray(value?.parts)) { + const kept = value.parts.filter((part) => !blockedCall(policy, sessionId, transcript, part?.callID ?? part?.id)); + if (kept.length !== value.parts.length) + return JSON.stringify({ ...value, parts: kept }); + } + return line; +} +/** Keep byte delimiters and physical line coordinates stable in derived copies. */ +async function* admittedRecords(file, policy, startLine = 1, endLine = Infinity) { + const policySnapshot = JSON.stringify(policy); + assertUnchangedPolicy(policySnapshot); + const input = fs.openSync(file, fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW); + const initial = fs.fstatSync(input); + let stream; + let screener; + const decoder = new TextDecoder('utf-8', { fatal: true }); + let pending = Buffer.alloc(0); + let sessionId = sessionFromPath(file); + let primaryIdentitySeen = false; + let number = 0; + const admit = (bytes, terminated) => { + number++; + const crlf = terminated && bytes.at(-1) === 13; + const line = decoder.decode(crlf ? bytes.subarray(0, -1) : bytes); + let recordSession; + try { + recordSession = sessionFromRecord(JSON.parse(line)); + } + catch { /* Parser owns malformed records. */ } + if (recordSession && !primaryIdentitySeen) { + if (sessionId && recordSession !== sessionId && (excludedSession(policy, sessionId) || excludedSession(policy, recordSession))) { + throw new Error('Record exclusion session identity mismatch'); + } + sessionId = recordSession; + primaryIdentitySeen = true; + } + const filtered = number >= startLine && number <= endLine + ? filterRecord(line, number, policy, path.basename(file), sessionId) : ''; + screener?.scan(filtered); + return { line: filtered, ending: terminated ? (crlf ? '\r\n' : '\n') : '' }; + }; + try { + // The marker scan and record stream share one descriptor and identity. + // A path replacement cannot substitute a different unscanned transcript. + if (descriptorHasExclusionMarker(input)) + return; + assertUnchangedInput(file, initial); + screener = createRecordScreener(policy); + stream = fs.createReadStream(file, { fd: input, autoClose: true, start: 0 }); + for await (const chunk of stream) { + assertUnchangedInput(file, initial); + assertUnchangedPolicy(policySnapshot); + pending = Buffer.concat([pending, chunk]); + let newline; + while ((newline = pending.indexOf(10)) !== -1) { + yield admit(pending.subarray(0, newline), true); + pending = pending.subarray(newline + 1); + } + if (pending.length > 64 * 1024 * 1024) + throw new Error('Conversation record exceeds admission byte limit'); + } + if (pending.length) + yield admit(pending, false); + assertUnchangedInput(file, initial); + assertUnchangedPolicy(policySnapshot); + } + finally { + if (stream) + stream.destroy(); + else + fs.closeSync(input); + screener?.close(); + } +} +/** Read original inputs without mutation, preserving their physical line numbers. */ +export async function* admittedConversationLines(file, policy = readRecordExclusions()) { + if (!fs.lstatSync(file).isFile()) + throw new Error('Conversation input must be a regular non-symlink file'); + if (shouldSkipConversation(file)) + return; + for await (const record of admittedRecords(file, policy)) + yield record.line; +} +/** Read the admitted display input without changing original physical coordinates. */ +export async function readAdmittedConversation(file, startLine = 1, endLine = Infinity) { + if (!Number.isSafeInteger(startLine) || startLine < 1 || + (endLine !== Infinity && (!Number.isSafeInteger(endLine) || endLine < startLine))) { + throw new Error('Invalid conversation line range'); + } + const lines = []; + for await (const record of admittedRecords(file, readRecordExclusions(), startLine, endLine)) + lines.push(record.line); + return lines.join('\n'); +} +/** Enforce the same rule for direct callers that bypass transcript parsing. */ +export function admitExchange(exchange, policy = readRecordExclusions()) { + if (!policy) + return exchange; + // Historical Codex rows may carry a fork ancestor's ID. Neither that metadata + // nor the filename may override a denial bound to the other known identity. + const sessionIds = [exchange.sessionId, sessionFromPath(exchange.archivePath)].filter((id) => !!id); + const transcript = path.basename(exchange.archivePath); + if (policy.exchanges.some(x => sessionIds.includes(x.session_id) && x.transcript === transcript && exchange.lineStart <= x.line_end && exchange.lineEnd >= x.line_start)) + return null; + const kept = exchange.toolCalls?.filter(call => !sessionIds.some(id => blockedCall(policy, id, transcript, call.id))); + const admitted = kept && kept.length !== exchange.toolCalls?.length ? { ...exchange, toolCalls: kept } : exchange; + const screener = createRecordScreener(policy); + try { + // These are separate original records, not one keyword-prefilter fragment. + screener.scan(admitted.userMessage); + screener.scan(admitted.assistantMessage); + for (const call of admitted.toolCalls ?? []) + screener.scan(call); + const { userMessage, assistantMessage, toolCalls, ...metadata } = admitted; + screener.scan(metadata); + return admitted; + } + finally { + screener.close(); + } +} +/** Publish only admitted bytes. Never write an unfiltered temporary copy. */ +export async function archiveAdmittedConversation(source, destination, policy = readRecordExclusions(), redactor = null, tally) { + const policySnapshot = JSON.stringify(policy); + assertUnchangedPolicy(policySnapshot); + if (shouldSkipConversation(source)) + return false; + if (path.resolve(source) === path.resolve(destination)) + throw new Error('Source and derived archive must be distinct'); + const original = fs.lstatSync(source); + if (!original.isFile()) + throw new Error('Conversation source must be a regular non-symlink file'); + let target; + if (fs.existsSync(destination)) { + target = fs.lstatSync(destination); + if (!target.isFile() || target.dev === original.dev && target.ino === original.ino) { + throw new Error('Source and derived archive must be distinct regular files'); + } + } + if (!policy && !redactor && target && target.mtimeMs >= original.mtimeMs && target.size === original.size) + return false; + const initial = fs.statSync(source); + const parent = prepareArchiveParent(path.dirname(path.resolve(destination))); + const parentIdentity = fs.statSync(parent); + destination = path.join(parent, path.basename(destination)); + const temp = destination + '.tmp.' + process.pid + '.' + crypto.randomUUID(); + const fd = fs.openSync(temp, 'wx', 0o600); + const hash = crypto.createHash('sha256'); + try { + for await (const record of admittedRecords(source, policy)) { + // Exclusions use source coordinates; only admitted text reaches redaction. + const line = redactor + ? redactJsonlLine(record.line, redactor, { source: 'archive', path: source }, tally) + : record.line; + const bytes = Buffer.from(line + record.ending); + hash.update(bytes); + fs.writeFileSync(fd, bytes); + } + fs.fsyncSync(fd); + } + catch (error) { + fs.closeSync(fd); + fs.unlinkSync(temp); + throw error; + } + fs.closeSync(fd); + try { + const final = fs.statSync(source); + if (initial.dev !== final.dev || initial.ino !== final.ino || initial.size !== final.size || initial.mtimeMs !== final.mtimeMs) { + throw new Error('Conversation changed during record admission'); + } + if (fs.existsSync(destination)) { + const existing = crypto.createHash('sha256'); + for await (const chunk of fs.createReadStream(destination)) + existing.update(chunk); + if (existing.digest('hex') === hash.digest('hex')) + return false; + } + assertUnchangedInput(source, original); + assertUnchangedPolicy(policySnapshot); + const currentParent = fs.lstatSync(parent); + if (!currentParent.isDirectory() || currentParent.dev !== parentIdentity.dev || currentParent.ino !== parentIdentity.ino) { + throw new Error('Archive parent changed during record admission'); + } + fs.renameSync(temp, destination); + fs.utimesSync(destination, initial.atimeMs / 1000, Math.ceil(initial.mtimeMs) / 1000); + return true; + } + finally { + if (fs.existsSync(temp)) + fs.unlinkSync(temp); + } +} +/** Request-local admission: retained search rows are not trusted merely because they were indexed. */ +export function createSearchAdmission() { + const policy = readRecordExclusions(); + const policySnapshot = JSON.stringify(policy); + const files = new Map(); + const checkFile = (file) => { + assertUnchangedPolicy(policySnapshot); + const cached = files.get(file); + if (cached) + return !cached.excluded; + let fd; + try { + let pathInfo; + try { + pathInfo = fs.lstatSync(file); + } + catch (error) { + // A transcript removed since indexing has nothing left to verify or expose: + // withhold its retained rows instead of failing the whole search. + if (error.code !== 'ENOENT') + throw error; + files.set(file, { excluded: true }); + return false; + } + if (!pathInfo.isFile()) + throw new Error('Conversation changed during record admission'); + fd = fs.openSync(file, fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW); + const initial = fs.fstatSync(fd); + if (!initial.isFile() || initial.dev !== pathInfo.dev || initial.ino !== pathInfo.ino || + !Number.isSafeInteger(initial.size)) + throw new Error('Conversation changed during record admission'); + const hash = crypto.createHash('sha256'); + // The observed size is this request's snapshot boundary. Records appended + // later are admitted by the next request, including conversation opt-outs. + const excluded = descriptorHasExclusionMarker(fd, initial.size, hash); + files.set(file, { initial, excluded, digest: excluded ? undefined : hash.digest('hex') }); + return !excluded; + } + catch { + throw new Error('Conversation changed during record admission'); + } + finally { + if (fd !== undefined) + fs.closeSync(fd); + } + }; + return { + admit(exchange) { + if (!checkFile(exchange.archivePath)) + return null; + const admitted = admitExchange(exchange, policy); + assertUnchangedPolicy(policySnapshot); + return admitted; + }, + summary(exchange, text) { + // An old summary has no record-level provenance. A filtered source may have + // contributed any of its text, so do not expose that derived summary. + const ids = [exchange.sessionId, sessionFromPath(exchange.archivePath)]; + const transcript = path.basename(exchange.archivePath); + if (policy && [...policy.exchanges, ...policy.tool_calls].some(row => row.transcript === transcript && ids.includes(row.session_id))) + return undefined; + return admitExchange({ ...exchange, userMessage: text, assistantMessage: '', toolCalls: [] }, policy) + ? text : undefined; + }, + verify() { + assertUnchangedPolicy(policySnapshot); + for (const [file, { initial, digest }] of files) { + if (initial !== undefined && digest !== undefined) + assertUnchangedSearchPrefix(file, initial, digest); + } + assertUnchangedPolicy(policySnapshot); + }, + }; +} diff --git a/dist/redact-cli.d.ts b/dist/redact-cli.d.ts new file mode 100644 index 00000000..cb0ff5c3 --- /dev/null +++ b/dist/redact-cli.d.ts @@ -0,0 +1 @@ +export {}; diff --git a/dist/redact-cli.js b/dist/redact-cli.js new file mode 100644 index 00000000..ee142efa --- /dev/null +++ b/dist/redact-cli.js @@ -0,0 +1,134 @@ +import fs from 'fs'; +import { getArchiveDir, getCursorLegacyExportDir, getOpencodeTranscriptDir } from './paths.js'; +import { acquireFileLock, readLockHolder, releaseFileLock } from './file-lock.js'; +import { getSyncLockPath } from './logging.js'; +import { DEFAULT_REDACTION_CONFIG, FindingsTally, formatFindings, getRedactionSettings, loadRedactor, redactJsonlLine, } from './redaction.js'; +const args = process.argv.slice(2); +const HELP = ` +Usage: episodic-memory redact [--rewrite [--dry-run [--report]]] [--stdin] [--print-default-rules] + +Secret redaction for the conversation archive and index. + +OPTIONS: + --rewrite Re-run redaction over the existing archive, staging exports, + and search index (in place). Rows that change are re-embedded, + and summaries built from unredacted text are deleted (the next + sync regenerates them). Run once after upgrading, and again + after adding rules. + --dry-run With --rewrite: report what would change, write nothing. + The index is opened read-only and is not migrated. + --report With --rewrite --dry-run: list every value that would be + redacted: where it is, the rule, its shape (length, + character classes, entropy) and the redacted text around + it. Use it to spot false positives before applying. + --stdin Redact stdin to stdout, one JSONL/text line at a time, and + print rule counts to stderr. Handy for testing rules. + --print-default-rules Print the bundled rules as JSON (a starting point for + redaction-rules.json). + --help, -h Show this help + +ENVIRONMENT: + EPISODIC_MEMORY_REDACTION on (default) | off + EPISODIC_MEMORY_REDACTION_RULES custom rules file (default: /redaction-rules.json) + EPISODIC_MEMORY_REDACTION_STRICT 1 (default) fails closed on a bad rules file | 0 passes through + +Output names rule IDs, counts and value shapes only; matched values are never printed. +`; +function fail(message) { + console.error(`episodic-memory: ${message}`); + process.exit(1); +} +function requireRedactor() { + if (!getRedactionSettings().enabled) { + fail('redaction is off (EPISODIC_MEMORY_REDACTION=off); unset it to use this command.'); + } + let redactor; + try { + // Always strict here: a rewrite with no rules would be a silent no-op. + redactor = loadRedactor({ ...process.env, EPISODIC_MEMORY_REDACTION_STRICT: '1' }); + } + catch (error) { + fail(error instanceof Error ? error.message : String(error)); + } + return redactor; +} +async function runRewrite(dryRun, report) { + if (report && !dryRun) + fail('--report needs --dry-run: review the hits, then apply without it.'); + const redactor = requireRedactor(); + // Share sync's single-instance lock so a background sync can't write + // between our reads and renames. + const lockPath = getSyncLockPath(); + const lock = acquireFileLock(lockPath); + if (!lock) { + const holder = readLockHolder(lockPath); + fail(`a sync or index run is in progress (${holder !== null ? `pid ${holder}` : 'another process'}); try again when it finishes.`); + } + const release = () => releaseFileLock(lock); + process.on('exit', release); + const { rewriteArchive } = await import('./redact-rewrite.js'); + let embeddingsReady = false; + const embed = async (user, assistant, toolNames) => { + const embeddings = await import('./embeddings.js'); + if (!embeddingsReady) { + await embeddings.initEmbeddings(); + embeddingsReady = true; + } + return embeddings.generateExchangeEmbedding(user, assistant, toolNames); + }; + const archiveDir = getArchiveDir(); + const stagingDirs = [getOpencodeTranscriptDir(), getCursorLegacyExportDir()].filter(d => fs.existsSync(d)); + console.log(`Redacting${dryRun ? ' (dry run)' : ''}: ${archiveDir}`); + for (const dir of stagingDirs) + console.log(` + staging: ${dir}`); + const result = await rewriteArchive({ + archiveDir, + stagingDirs, + redactor, + embed, + dryRun, + report: report + ? hit => console.log(` ${hit.location} ${hit.ruleId} ${hit.shape} + ${hit.context}`) + : undefined, + log: message => console.log(` ${message}`), + }); + console.log(`\n${dryRun ? 'Would redact' : 'Redacted'} across archive, staging, and index: ${formatFindings(result.findings)}`); + if (dryRun && (result.filesRewritten || result.rowsUpdated || result.summariesRemoved || result.stagingFilesRewritten)) { + console.log('Run without --dry-run to apply.'); + } +} +async function runStdin() { + const redactor = requireRedactor(); + const chunks = []; + for await (const chunk of process.stdin) + chunks.push(chunk); + const input = Buffer.concat(chunks).toString('utf-8'); + const tally = new FindingsTally(); + const output = input.split('\n').map(line => redactJsonlLine(line, redactor, { source: 'stdin', path: '-' }, tally)); + process.stdout.write(output.join('\n')); + console.error(formatFindings(tally.toArray())); +} +async function main() { + if (args.length === 0 || args.includes('--help') || args.includes('-h')) { + console.log(HELP); + return; + } + if (args.includes('--print-default-rules')) { + console.log(JSON.stringify(DEFAULT_REDACTION_CONFIG, null, 2)); + return; + } + if (args.includes('--stdin')) { + await runStdin(); + return; + } + if (args.includes('--rewrite')) { + await runRewrite(args.includes('--dry-run'), args.includes('--report')); + return; + } + fail(`unknown option(s): ${args.join(' ')}. Try: episodic-memory redact --help`); +} +main().catch(error => { + console.error('Error:', error instanceof Error ? error.message : error); + process.exit(1); +}); diff --git a/dist/redact-rewrite.d.ts b/dist/redact-rewrite.d.ts new file mode 100644 index 00000000..c067931d --- /dev/null +++ b/dist/redact-rewrite.d.ts @@ -0,0 +1,56 @@ +import { type RedactionFinding, type Redactor } from './redaction.js'; +/** + * Backfill for `episodic-memory redact --rewrite`: re-run redaction over data + * written before redaction existed (or before a rule was added). + * + * 1. Archive: every .jsonl is redacted in place. Line count and mtime are + * preserved, so index line ranges stay valid and sync still sees the + * archive as current. + * 2. Staging dirs (opencode / legacy Cursor exports): same, in place. + * 3. Index: every exchanges/tool_calls row is redacted in place. Changed rows + * are re-embedded from the redacted text, so no vector is left that was + * derived from a secret. + * 4. Summaries: a `-summary.txt` for a conversation that had findings (in the + * archive or the index), or one that matches a rule itself, is deleted. + * It was generated from unredacted text, and the next sync regenerates it + * from the redacted archive. + * + * Idempotent: a second run finds nothing to change. A dry run opens the index + * read-only, so it doesn't create, migrate or otherwise touch it. + */ +export type EmbedFn = (user: string, assistant: string, toolNames?: string[]) => Promise; +export interface RewriteOptions { + archiveDir: string; + redactor: Redactor; + embed: EmbedFn; + /** Count what would change without writing anything. */ + dryRun?: boolean; + /** + * Called once per value that would be redacted, for reviewing hits before + * applying them. Never receives the value itself. + */ + report?: (hit: RedactionHit) => void; + /** Plugin-owned staging dirs to redact in place (not indexed). */ + stagingDirs?: string[]; + log?: (message: string) => void; +} +export interface RewriteResult { + filesScanned: number; + filesRewritten: number; + stagingFilesRewritten: number; + rowsUpdated: number; + summariesRemoved: number; + /** Rule IDs and counts only — never matched values. */ + findings: RedactionFinding[]; +} +/** One redacted value, described without revealing it. */ +export interface RedactionHit { + /** Archive-relative file and 1-based line, or `index:# `. */ + location: string; + ruleId: string; + /** Length, character classes and entropy (see describeShape). */ + shape: string; + /** Redacted text around the token, on one line. */ + context: string; +} +export declare function rewriteArchive(options: RewriteOptions): Promise; diff --git a/dist/redact-rewrite.js b/dist/redact-rewrite.js new file mode 100644 index 00000000..50894a78 --- /dev/null +++ b/dist/redact-rewrite.js @@ -0,0 +1,263 @@ +import fs from 'fs'; +import path from 'path'; +import { initDatabase, openDatabaseReadOnly } from './db.js'; +import { recordReembedded } from './embedding-migration.js'; +import { copyFileRedacted, describeShape, findRedactionTokens, FindingsTally, redactJsonlLine, } from './redaction.js'; +const SUMMARY_SUFFIX = '-summary.txt'; +const CONTEXT_BEFORE = 60; +const CONTEXT_AFTER = 20; +const PAGE_SIZE = 500; +function walk(dir) { + const out = []; + let entries; + try { + entries = fs.readdirSync(dir, { withFileTypes: true }); + } + catch { + return out; + } + for (const entry of entries) { + const full = path.join(dir, entry.name); + if (entry.isDirectory()) + out.push(...walk(full)); + else if (entry.isFile()) + out.push(full); + } + return out; +} +function summaryPathFor(jsonlPath) { + return jsonlPath.replace(/\.jsonl$/, SUMMARY_SUFFIX); +} +// In report mode each hit is written as its own numbered token, e.g. +// `[REDACTED:jwt--hit3]`, so it can be found in the output exactly; tokens that +// were already in the text have no number. Numbers are stripped for display. +const HIT_TOKEN = /\[REDACTED:([a-z0-9][a-z0-9-]*?)--hit(\d+)\]/g; +/** `value` with any numbered tokens in it replaced by the values they stand for. */ +function restoreHits(value, matches) { + return value.replace(HIT_TOKEN, (_token, _ruleId, n) => matches[Number(n)].value); +} +/** + * Report each hit with the redacted text around its token. A hit whose token + * was swallowed by a later one (a secret field redacted whole after a text rule + * hit part of it) is reported at the token that swallowed it. + */ +function reportHits(redacted, matches, location, report) { + let display = ''; + let pos = 0; + const spans = new Map(); + for (const t of findRedactionTokens(redacted)) { + const numbered = /^(.*)--hit(\d+)$/.exec(t.ruleId); + display += redacted.slice(pos, t.start); + const shown = numbered ? `[REDACTED:${numbered[1]}]` : redacted.slice(t.start, t.end); + if (numbered && !spans.has(Number(numbered[2]))) { + spans.set(Number(numbered[2]), [display.length, display.length + shown.length]); + } + display += shown; + pos = t.end; + } + display += redacted.slice(pos); + const spanOf = (n) => { + for (let i = n; i !== undefined; i = matches[i].absorbedBy) { + const span = spans.get(i); + if (span) + return span; + } + return undefined; + }; + const oneLine = (text) => text.replace(/\s+/g, ' '); + const hits = matches.map((m, n) => ({ m, span: spanOf(n) })); + hits.sort((a, b) => (a.span?.[0] ?? Infinity) - (b.span?.[0] ?? Infinity)); + for (const { m, span } of hits) { + const [start, end] = span ?? [0, 0]; + const before = display.slice(Math.max(0, start - CONTEXT_BEFORE), start); + report({ + location, + ruleId: m.ruleId, + shape: describeShape(m.value), + context: span + ? oneLine(`${start > CONTEXT_BEFORE ? '…' : ''}${before}${display.slice(start, end + CONTEXT_AFTER)}`) + : '', + }); + } + return display; +} +/** + * Redact with `run`, reporting each hit when `report` is set. Returns + * the redacted text with ordinary tokens either way. + */ +function redactReporting(ctx, location, report, run) { + if (!report) + return run(ctx); + const matches = []; + const out = run({ + ...ctx, + onMatch: (ruleId, value) => { + const n = matches.length; + for (const [, , k] of value.matchAll(HIT_TOKEN)) + matches[Number(k)].absorbedBy = n; + matches.push({ ruleId, value: restoreHits(value, matches) }); + return `[REDACTED:${ruleId}--hit${n}]`; + }, + }); + return matches.length > 0 ? reportHits(out, matches, location, report) : out; +} +/** Report every hit in a JSONL file, line by line. Writes nothing. */ +function reportFile(file, label, redactor, report) { + const lines = fs.readFileSync(file, 'utf-8').split('\n'); + const ctx = { source: 'rewrite', path: file }; + lines.forEach((line, i) => { + redactReporting(ctx, `${label}:${i + 1}`, report, c => redactJsonlLine(line, redactor, c)); + }); +} +/** Redact one JSONL file in place. Returns the number of values redacted. */ +function rewriteFileInPlace(file, redactor, dryRun, tally) { + const fileTally = new FindingsTally(); + const temp = `${file}.redact.${process.pid}`; + try { + copyFileRedacted(file, temp, redactor, { source: 'rewrite', path: file }, fileTally); + if (fileTally.total > 0 && !dryRun) { + const stat = fs.statSync(file); + fs.renameSync(temp, file); + // Same rounding as archiveAdmittedConversation: never leave the archive older than its source. + fs.utimesSync(file, stat.atimeMs / 1000, Math.ceil(stat.mtimeMs) / 1000); + } + } + finally { + try { + fs.unlinkSync(temp); + } + catch { } + } + tally.add(fileTally.toArray()); + return fileTally.total; +} +export async function rewriteArchive(options) { + const { archiveDir, redactor, embed } = options; + const dryRun = options.dryRun === true; + // Reporting reads files before redaction; after a real rewrite there'd be nothing to find. + if (options.report && !dryRun) + throw new Error('rewriteArchive: report requires dryRun'); + const log = options.log ?? (() => { }); + const tally = new FindingsTally(); + const result = { + filesScanned: 0, + filesRewritten: 0, + stagingFilesRewritten: 0, + rowsUpdated: 0, + summariesRemoved: 0, + findings: [], + }; + // Conversations whose summary was built from unredacted text. + const staleSummaries = new Set(); + // 1. Archive files. + const archiveFiles = walk(archiveDir); + for (const file of archiveFiles.filter(f => f.endsWith('.jsonl'))) { + result.filesScanned++; + if (rewriteFileInPlace(file, redactor, dryRun, tally) > 0) { + result.filesRewritten++; + staleSummaries.add(summaryPathFor(file)); + if (options.report) + reportFile(file, path.relative(archiveDir, file), redactor, options.report); + } + } + log(`Archive: ${result.filesRewritten} of ${result.filesScanned} file(s) ${dryRun ? 'would be ' : ''}rewritten`); + // 2. Staging dirs. + for (const dir of options.stagingDirs ?? []) { + for (const file of walk(dir).filter(f => f.endsWith('.jsonl'))) { + if (rewriteFileInPlace(file, redactor, dryRun, tally) > 0) { + result.stagingFilesRewritten++; + if (options.report) + reportFile(file, `staging:${path.relative(dir, file)}`, redactor, options.report); + } + } + } + if (options.stagingDirs?.length) { + log(`Staging exports: ${result.stagingFilesRewritten} file(s) ${dryRun ? 'would be ' : ''}rewritten`); + } + // 3. Index rows. Page by rowid so writes between pages don't disturb the scan. + // A dry run reads the index as it is; opening it normally would migrate it. + const db = dryRun ? openDatabaseReadOnly() : initDatabase(); + if (db) + try { + const page = db.prepare('SELECT rowid AS rid, id, user_message, assistant_message, archive_path FROM exchanges WHERE rowid > ? ORDER BY rowid LIMIT ?'); + const toolsFor = db.prepare('SELECT id, tool_name, tool_input, tool_result FROM tool_calls WHERE exchange_id = ? ORDER BY rowid'); + const updateExchange = db.prepare('UPDATE exchanges SET user_message = ?, assistant_message = ? WHERE id = ?'); + const updateTool = db.prepare('UPDATE tool_calls SET tool_input = ?, tool_result = ? WHERE id = ?'); + let lastRowid = 0; + for (;;) { + const rows = page.all(lastRowid, PAGE_SIZE); + if (rows.length === 0) + break; + lastRowid = rows[rows.length - 1].rid; + for (const row of rows) { + const rowTally = new FindingsTally(); + const ctx = { source: 'index', path: row.archive_path }; + const where = `index:${path.relative(archiveDir, row.archive_path)}#${row.id}`; + const redactText = (text, field) => redactReporting(ctx, `${where} ${field}`, options.report, c => { + const r = redactor.redact(text, c); + rowTally.add(r.findings); + return r.text; + }); + const user = { text: redactText(row.user_message, 'user') }; + const assistant = { text: redactText(row.assistant_message, 'assistant') }; + const tools = toolsFor.all(row.id); + const toolUpdates = []; + for (const tool of tools) { + // tool_input is JSON text; redactJsonlLine keeps it valid JSON. + const toolInput = tool.tool_input; + const input = toolInput === null ? null : redactReporting(ctx, `${where} ${tool.tool_name} input`, options.report, c => redactJsonlLine(toolInput, redactor, c, rowTally)); + const output = tool.tool_result === null ? null : redactText(tool.tool_result, `${tool.tool_name} result`); + if (input !== tool.tool_input || output !== tool.tool_result) { + toolUpdates.push({ id: tool.id, input, result: output }); + } + } + if (rowTally.total === 0) + continue; + tally.add(rowTally.toArray()); + result.rowsUpdated++; + staleSummaries.add(summaryPathFor(row.archive_path)); + if (dryRun) + continue; + const toolNames = tools.length > 0 ? tools.map(t => t.tool_name) : undefined; + const embedding = await embed(user.text, assistant.text, toolNames); + db.transaction(() => { + updateExchange.run(user.text, assistant.text, row.id); + for (const t of toolUpdates) + updateTool.run(t.input, t.result, t.id); + recordReembedded(db, row.id, embedding); + })(); + } + } + } + finally { + db.close(); + } + log(`Index: ${result.rowsUpdated} exchange(s) ${dryRun ? 'would be ' : ''}redacted and re-embedded`); + // 4. Summaries: stale ones, plus any summary that itself matches a rule. + for (const file of archiveFiles.filter(f => f.endsWith(SUMMARY_SUFFIX))) { + if (staleSummaries.has(file)) + continue; + let text; + try { + text = fs.readFileSync(file, 'utf-8'); + } + catch { + continue; + } + const r = redactor.redact(text, { source: 'summary', path: file }); + if (r.findings.length > 0) { + tally.add(r.findings); + staleSummaries.add(file); + } + } + for (const summary of staleSummaries) { + if (!fs.existsSync(summary)) + continue; + result.summariesRemoved++; + if (!dryRun) + fs.unlinkSync(summary); + } + log(`Summaries: ${result.summariesRemoved} ${dryRun ? 'would be ' : ''}removed (regenerated from redacted text on the next sync)`); + result.findings = tally.toArray(); + return result; +} diff --git a/dist/redaction-rules.d.ts b/dist/redaction-rules.d.ts new file mode 100644 index 00000000..d9c5ad24 --- /dev/null +++ b/dist/redaction-rules.d.ts @@ -0,0 +1,2 @@ +import type { RedactionConfig } from './redaction.js'; +export declare const DEFAULT_REDACTION_CONFIG: RedactionConfig; diff --git a/dist/redaction-rules.js b/dist/redaction-rules.js new file mode 100644 index 00000000..6295381c --- /dev/null +++ b/dist/redaction-rules.js @@ -0,0 +1,254 @@ +/** + * Bundled default redaction rules. + * + * Patterns are ported from gitleaks' rule set (https://github.com/gitleaks/gitleaks, + * MIT), trimmed to the credentials that realistically show up in Claude Code / + * Codex transcripts for an Azure-heavy stack, and rewritten for JavaScript + * regex (lookbehind instead of gitleaks' consuming boundary groups, so + * adjacent matches aren't swallowed). + * + * Order matters: rules run top to bottom, and a later rule never re-matches + * inside an earlier rule's `[REDACTED:...]` token. Specific shapes go first so + * the token names the most precise rule; the generic `secret-assignment` + * keyword rule goes last. + * + * `secretGroup` replaces only that capture group, so surrounding context + * (connection-string server names, SAS URL paths, usernames) stays searchable. + * + * `keywords` is a case-insensitive prefilter: a rule only runs on text that + * contains at least one keyword. It keeps the per-string cost low on large + * transcripts and bounds the generic rules' work. + * + * Users extend or override these with `redaction-rules.json`; see + * docs/REDACTION.md. `episodic-memory redact --print-default-rules` dumps + * this object as JSON. + */ +// Names that mark the value next to them as a secret, for the key-context +// rules below. A name must END in one of these (optionally plus digits), so +// tokenType, secretName, passwordPolicy, TokenEndpoint and maxTokens don't +// count. `pwd` is deliberately absent (the PWD env var); `userPWD` is handled +// by xml-secret-attribute. +const SECRET_NAME = String.raw `(?:secret|password|passwd|passphrase|api[_-]?key|access[_-]?key|account[_-]?key|private[_-]?key|` + + String.raw `shared[_-]?(?:access[_-]?)?key|primary[_-]?key|secondary[_-]?key|master[_-]?key|signing[_-]?key|` + + String.raw `subscription[_-]?key|client[_-]?key|encryption[_-]?key|token|credentials?)\d*`; +// Value is not a template/placeholder: ${X} $(X) $X #{X}# {{x}} %X% __X__, +// an existing token, a type name (Swagger "string"), or a mask (*****). +const NOT_PLACEHOLDER = String.raw `(?!\[REDACTED:|\$\{|\$\(|\$[A-Za-z_]\w*["'<\s]|#\{|\{\{|<[^>]*>|%[A-Za-z_]\w*%|__[A-Za-z0-9_]+__|` + + String.raw `(?:string|null|true|false|none|undefined|\*+)["'<])`; +// A JSON string value (escapes allowed), captured. +const JSON_STRING_VALUE = String.raw `"${NOT_PLACEHOLDER}([^"\\]+(?:\\.[^"\\]*)*)"`; +// Sibling members inside the same object: strings, or anything but braces/quotes. +const SAME_OBJECT = String.raw `(?:[^{}"]|"(?:[^"\\]|\\.)*"){0,400}?`; +const KEY_VAULT_ID = String.raw `"id"[ \t]*:[ \t]*"https://[^"\s]+\.vault\.(?:azure\.net|azure\.cn|usgovcloudapi\.net|microsoftazure\.de)/secrets/[^"\s]*"`; +export const DEFAULT_REDACTION_CONFIG = { + rules: [ + { + id: 'private-key-block', + description: 'PEM/OpenSSH private key block; a truncated block (no END line) is redacted to the end of the text', + pattern: String.raw `-----BEGIN[A-Z0-9 _-]{0,40}PRIVATE KEY(?: BLOCK)?-----[\s\S]*?(?:-----END[A-Z0-9 _-]{0,40}PRIVATE KEY(?: BLOCK)?-----|$)`, + keywords: ['private key'], + }, + { + id: 'connection-string-secret', + description: 'Secret value inside an Azure/ADO.NET connection string; account, server and database names are kept. Keywords are case-insensitive, as in ADO.NET; a path after Pwd= is the shell PWD variable and is kept; the value needs a letter or digit, so Markdown like `Password=` is kept', + pattern: String.raw `\b(?:(?:AccountKey|SharedAccessKey|SharedAccessSignature|SharedSecret|ClientSecret|Password)=|Pwd=(?![/~]))(?![\[$<{%])(?=[^;"'\s]*[A-Za-z0-9])([^;"'\s]+)`, + flags: 'i', + secretGroup: 1, + keywords: ['accountkey', 'sharedaccess', 'sharedsecret', 'clientsecret', 'password', 'pwd'], + }, + { + id: 'azure-sas-token', + description: 'Signature of an Azure SAS token; the resource URL and other SAS parameters are kept', + pattern: String.raw `\bsig=([A-Za-z0-9%+/=_-]{16,})`, + secretGroup: 1, + keywords: ['sig='], + }, + { + id: 'jwt', + description: 'JSON Web Token (Entra ID / Azure access tokens, id tokens)', + pattern: String.raw `\beyJ[A-Za-z0-9_-]{8,}\.eyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}`, + keywords: ['eyj'], + }, + { + id: 'anthropic-api-key', + pattern: String.raw `\bsk-ant-[a-z]{2,10}\d{2}-[A-Za-z0-9_-]{32,}`, + keywords: ['sk-ant-'], + }, + { + id: 'openai-api-key', + pattern: String.raw `\bsk-(?:[a-z]+-)?[A-Za-z0-9_-]{16,}T3BlbkFJ[A-Za-z0-9_-]{16,}`, + keywords: ['t3blbkfj'], + }, + { + id: 'github-token', + description: 'GitHub classic, OAuth, app and fine-grained tokens', + pattern: String.raw `\b(?:gh[pousr]_[A-Za-z0-9]{36,255}|github_pat_[A-Za-z0-9_]{22,255})\b`, + keywords: ['ghp_', 'gho_', 'ghu_', 'ghs_', 'ghr_', 'github_pat_'], + }, + { + id: 'aws-access-key-id', + pattern: String.raw `\b(?:AKIA|ASIA|ABIA|ACCA)[A-Z2-7]{16}\b`, + keywords: ['akia', 'asia', 'abia', 'acca'], + }, + { + id: 'aws-secret-access-key', + pattern: String.raw `aws[_-]?secret[_-]?access[_-]?key["']?[ \t]*[:=][ \t]*["']?([A-Za-z0-9/+]{40})(?![A-Za-z0-9/+])`, + flags: 'i', + secretGroup: 1, + keywords: ['secret'], + }, + { + id: 'slack-token', + pattern: String.raw `\bxox[abposr]-[A-Za-z0-9-]{10,}`, + keywords: ['xox'], + }, + { + id: 'google-api-key', + pattern: String.raw `\bAIza[0-9A-Za-z_-]{35}(?![0-9A-Za-z_-])`, + keywords: ['aiza'], + }, + { + id: 'npm-token', + pattern: String.raw `\bnpm_[A-Za-z0-9]{36}\b`, + keywords: ['npm_'], + }, + { + id: 'azure-client-secret', + description: 'Entra ID (Azure AD) application client secret: 3 chars, a digit, "Q~", 31-34 chars. A trailing "." may follow, so a secret ending a sentence still matches', + pattern: String.raw `(?]+:(?![\[$<{%])([^\s@/"'<>]+)@`, + secretGroup: 1, + keywords: ['://'], + }, + { + id: 'bearer-token', + pattern: String.raw `\bBearer\s+([A-Za-z0-9\-._~+/]{20,}=*)`, + flags: 'i', + secretGroup: 1, + keywords: ['bearer'], + }, + { + id: 'basic-auth', + pattern: String.raw `\bAuthorization[ \t]*:[ \t]*Basic\s+([A-Za-z0-9+/]{8,}={0,2})`, + flags: 'i', + secretGroup: 1, + keywords: ['basic'], + }, + { + id: 'azure-keyvault-secret', + description: 'The "value" of a Key Vault secret bundle (az keyvault secret show/set, SDK JSON), whatever the secret is named', + pattern: KEY_VAULT_ID + String.raw `(?:[^{}]|\{[^{}]*\}){0,800}?"value"[ \t]*:[ \t]*` + JSON_STRING_VALUE + + String.raw `|"value"[ \t]*:[ \t]*` + JSON_STRING_VALUE + String.raw `(?=(?:[^{}]|\{[^{}]*\}){0,800}?` + KEY_VAULT_ID + ')', + secretGroup: [1, 2], + useAllowlist: false, + keywords: ['.vault.'], + }, + { + id: 'name-value-secret', + description: 'The "value" of a {"name"/"key": , "value": ...} object, in either order: ' + + 'az webapp/functionapp config appsettings list, Kubernetes env, ARM/Bicep parameters', + pattern: String.raw `"(?:name|key)"[ \t]*:[ \t]*"[^"\\]{0,80}?` + SECRET_NAME + '"' + SAME_OBJECT + + String.raw `"value"[ \t]*:[ \t]*` + JSON_STRING_VALUE + + String.raw `|"value"[ \t]*:[ \t]*` + JSON_STRING_VALUE + '(?=' + SAME_OBJECT + + String.raw `"(?:name|key)"[ \t]*:[ \t]*"[^"\\]{0,80}?` + SECRET_NAME + '")', + flags: 'i', + secretGroup: [1, 2], + useAllowlist: false, + keywords: ['"value"'], + }, + { + id: 'xml-appsettings-secret', + description: 'web.config / app.config , either attribute order', + pattern: String.raw `]*?\bkey[ \t]*=[ \t]*"[^"]{0,80}?` + SECRET_NAME + String.raw `"[^>]*?\bvalue[ \t]*=[ \t]*"` + + NOT_PLACEHOLDER + String.raw `([^"]+)"` + + String.raw `|]*?\bvalue[ \t]*=[ \t]*"` + NOT_PLACEHOLDER + String.raw `([^"]+)"(?=[^>]*?\bkey[ \t]*=[ \t]*"[^"]{0,80}?` + + SECRET_NAME + '")', + flags: 'i', + secretGroup: [1, 2], + useAllowlist: false, + keywords: [']*)?>` + NOT_PLACEHOLDER + String.raw `([^<]{1,4096})`, + flags: 'i', + secretGroup: 2, + useAllowlist: false, + keywords: [', %X%, {{x}}) and code (calls, dotted member access)', + pattern: String.raw `\b[A-Za-z0-9_.-]{0,40}?(?:secret|password|passwd|passphrase|api[_-]?key|access[_-]?key|private[_-]?key|token)` + + String.raw `["']?[ \t]*[:=][ \t]*["']?(?![\[$<{%])(?=[^\s"',;&\\]*\d)` + + // Not a dotted identifier path (env.AUTH0_SECRET, this.config.token2): code, not a value. + String.raw `(?![A-Za-z_$][\w$]*(?:\.[A-Za-z_$][\w$]*)+(?:$|[\s"',;&\\)}\]]))` + + String.raw `([^\s"',;&\\()<>{}\[\]]{8,})(?=$|[\s"',;&\\)}\]])`, + flags: 'i', + secretGroup: 1, + keywords: ['secret', 'passw', 'passphrase', 'key', 'token'], + }, + ], + allowlist: [ + { + id: 'git-sha', + description: 'Full or short git commit SHA', + pattern: String.raw `\b[0-9a-f]{7,40}\b`, + }, + { + id: 'guid', + description: 'GUID/UUID (Entra tenant, client and object IDs, subscription IDs)', + pattern: String.raw `\b[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\b`, + }, + ], + entropy: { + enabled: false, + minLength: 32, + threshold: 4.5, + requireKeyword: true, + keywords: ['secret', 'key', 'token', 'password', 'passwd', 'credential', 'signature'], + window: 40, + }, + secretFields: { + enabled: true, + // Normalized key (lowercase letters/digits only, trailing digits dropped) + // must end with one of these. Harness keys (signature, apiKeySource, + // input_tokens, token_count) don't. + keyPattern: 'secret|password|passwd|userpwd|passphrase|apikey|accesskey|accountkey|privatekey|sharedkey|primarykey|' + + 'secondarykey|masterkey|signingkey|subscriptionkey|clientkey|encryptionkey|token|credentials?', + }, +}; diff --git a/dist/redaction.d.ts b/dist/redaction.d.ts new file mode 100644 index 00000000..d4af59bd --- /dev/null +++ b/dist/redaction.d.ts @@ -0,0 +1,180 @@ +import { DEFAULT_REDACTION_CONFIG } from './redaction-rules.js'; +/** + * Inline secret redaction. + * + * Secrets are replaced with typed tokens (`[REDACTED:]`) at the archive + * write, the one point every harness's transcripts pass through (see + * docs/REDACTION.md). Everything downstream (SQLite text, + * tool_calls, embeddings, summaries, show/read) reads the archive, so it only + * ever sees redacted text. + * + * Invariants: + * - Findings carry rule IDs and counts only. A matched value is never logged, + * returned, or thrown. + * - Idempotent: a `[REDACTED:...]` token never matches a rule, so redacting + * redacted text is a no-op. + * - JSONL structure is preserved: one output line per input line, valid JSON in + * gives valid JSON out, and unchanged lines are byte-identical. + */ +export { DEFAULT_REDACTION_CONFIG }; +/** Fork default. Flip to false for an opt-in upstream build. */ +export declare const REDACTION_ENABLED_BY_DEFAULT = true; +export declare const REDACTION_RULES_FILENAME = "redaction-rules.json"; +export interface RedactionRuleSpec { + id: string; + pattern: string; + /** Extra RegExp flags from [imsu]; `g` and `d` are always added. */ + flags?: string; + /** Case-insensitive prefilter: skip the rule unless the text contains one. */ + keywords?: string[]; + /** + * Replace only this capture group instead of the whole match. An array means + * "the first of these groups that participated" (for either-order patterns). + */ + secretGroup?: number | number[]; + /** Apply the allowlist to this rule's matches (default true). Key-context rules turn it off: a GUID in a password slot is a secret. */ + useAllowlist?: boolean; + description?: string; +} +export interface AllowlistSpec { + id: string; + /** Matched against the whole candidate secret; a full match keeps it. */ + pattern: string; + flags?: string; + description?: string; +} +export interface EntropySpec { + enabled: boolean; + minLength: number; + /** Shannon entropy in bits per character. */ + threshold: number; + /** Only fire when a keyword appears shortly before the candidate, on the same line. */ + requireKeyword: boolean; + keywords: string[]; + /** How many characters before the candidate to search for a keyword. */ + window: number; +} +/** + * Field-name context for parsed JSON (transcript lines, structured tool/MCP + * results, tool inputs). A string value is redacted whole when its own key, or + * the `name`/`key` of a `{name, value}` pair, matches `keyPattern`. + */ +export interface SecretFieldsSpec { + enabled: boolean; + /** + * Matched (anchored at the end) against the key lowercased with everything + * but letters and digits removed and trailing digits dropped, so + * `AzureAd:ClientSecret`, `client_secret` and `DB_PASSWORD2` all normalize to + * something ending in a keyword. + */ + keyPattern: string; +} +export interface RedactionConfig { + rules: RedactionRuleSpec[]; + allowlist: AllowlistSpec[]; + entropy: EntropySpec; + secretFields: SecretFieldsSpec; +} +/** Shape of a user `redaction-rules.json`. Every field is optional. */ +export interface RedactionRulesFile { + /** Merge with the bundled defaults (default true). */ + includeDefaults?: boolean; + /** Added after the defaults; a rule with a default's id replaces it in place. */ + rules?: RedactionRuleSpec[]; + /** Default rule ids to turn off. */ + disableRules?: string[]; + /** Added to the default allowlist (same id replaces). */ + allowlist?: AllowlistSpec[]; + entropy?: Partial; + secretFields?: Partial; +} +export interface RedactionContext { + source: string; + path: string; + /** + * Called with each value as it is redacted. For local review tooling + * (`redact --report`) only: the value is the secret itself, so never log it. + * May return the token to write instead of `[REDACTED:]`, e.g. one + * numbered per hit; it must still have the token's shape. + */ + onMatch?: (ruleId: string, value: string) => string | void; +} +export interface RedactionFinding { + ruleId: string; + count: number; +} +export interface RedactionResult { + text: string; + findings: RedactionFinding[]; +} +export interface Redactor { + redact(text: string, ctx?: RedactionContext): RedactionResult; + readonly ruleIds: string[]; + /** True when a JSON key / setting name marks its value as a secret (secretFields). */ + isSecretField(name: string): boolean; +} +export interface RedactionSettings { + enabled: boolean; + strict: boolean; + /** Explicit EPISODIC_MEMORY_REDACTION_RULES path, if set. */ + rulesPath?: string; +} +/** Rules could not be loaded or are invalid. Messages describe config only. */ +export declare class RedactionConfigError extends Error { + constructor(message: string); +} +/** + * EPISODIC_MEMORY_REDACTION on (fork default) | off + * EPISODIC_MEMORY_REDACTION_RULES path to a custom rules file + * EPISODIC_MEMORY_REDACTION_STRICT 1 (default) | 0; strict fails closed on a rules-load error + */ +export declare function getRedactionSettings(env?: NodeJS.ProcessEnv): RedactionSettings; +/** + * Merge a rules file with the bundled defaults and validate the result. + * Throws RedactionConfigError on any invalid rule. + */ +export declare function loadRedactionConfig(file: RedactionRulesFile): RedactionConfig; +/** + * Load the active redactor from the environment. + * + * Returns null when redaction is off. When the rules can't be loaded: strict + * mode (the default) throws RedactionConfigError so callers fail closed; + * non-strict mode warns and returns null, so text passes through unredacted. + */ +export declare function loadRedactor(env?: NodeJS.ProcessEnv, warn?: (message: string) => void): Redactor | null; +/** + * Describe a value without revealing it: length, character classes and + * Shannon entropy (bits per char), e.g. "len=40 aA9- H=4.9". Lets someone + * reviewing hits tell a random key from a word or a placeholder. + */ +export declare function describeShape(value: string): string; +/** Each `[REDACTED:]` token in `text`, left to right. */ +export declare function findRedactionTokens(text: string): Array<{ + ruleId: string; + start: number; + end: number; +}>; +export declare function createRedactor(config: RedactionConfig): Redactor; +/** Aggregates findings by rule id. Never holds matched values. */ +export declare class FindingsTally { + private counts; + add(findings: RedactionFinding[]): void; + get total(): number; + toArray(): RedactionFinding[]; +} +/** "7 value(s) redacted (azure-storage-key: 2, jwt: 5)" */ +export declare function formatFindings(findings: RedactionFinding[]): string; +/** + * Redact one JSONL line. Valid JSON stays valid (string values are redacted + * and the line is re-serialized only if something changed); a line that isn't + * JSON (e.g. a torn last line mid-write) is redacted as raw text. A trailing + * `\r` is preserved. + */ +export declare function redactJsonlLine(line: string, redactor: Redactor, ctx?: RedactionContext, tally?: FindingsTally): string; +/** + * Stream `src` to `dest`, redacting line by line. Writes `dest` directly; + * callers that need atomicity write to a temp path and rename. Line count and + * trailing-newline shape match the source exactly, so archive line numbers + * (exchanges.line_start/line_end, MCP read ranges) stay valid. + */ +export declare function copyFileRedacted(src: string, dest: string, redactor: Redactor, ctx?: RedactionContext, tally?: FindingsTally): void; diff --git a/dist/redaction.js b/dist/redaction.js new file mode 100644 index 00000000..e4f206d5 --- /dev/null +++ b/dist/redaction.js @@ -0,0 +1,659 @@ +import fs from 'fs'; +import path from 'path'; +import { StringDecoder } from 'string_decoder'; +import { getSuperpowersDir } from './paths.js'; +import { DEFAULT_REDACTION_CONFIG } from './redaction-rules.js'; +/** + * Inline secret redaction. + * + * Secrets are replaced with typed tokens (`[REDACTED:]`) at the archive + * write, the one point every harness's transcripts pass through (see + * docs/REDACTION.md). Everything downstream (SQLite text, + * tool_calls, embeddings, summaries, show/read) reads the archive, so it only + * ever sees redacted text. + * + * Invariants: + * - Findings carry rule IDs and counts only. A matched value is never logged, + * returned, or thrown. + * - Idempotent: a `[REDACTED:...]` token never matches a rule, so redacting + * redacted text is a no-op. + * - JSONL structure is preserved: one output line per input line, valid JSON in + * gives valid JSON out, and unchanged lines are byte-identical. + */ +export { DEFAULT_REDACTION_CONFIG }; +/** Fork default. Flip to false for an opt-in upstream build. */ +export const REDACTION_ENABLED_BY_DEFAULT = true; +export const REDACTION_RULES_FILENAME = 'redaction-rules.json'; +const RULE_ID_PATTERN = /^[a-z0-9][a-z0-9-]*$/; +const TOKEN_PATTERN = /\[REDACTED:[a-z0-9][a-z0-9-]*\]/g; +const TOKEN_PREFIX = '[REDACTED:'; +const ALLOWED_FLAGS = /^[imsu]*$/; +const ENTROPY_RULE_ID = 'high-entropy'; +const FIELD_RULE_ID = 'secret-field'; +const RESERVED_IDS = new Set([ENTROPY_RULE_ID, FIELD_RULE_ID]); +/** Values that are templates or type names, not secrets. Never redacted by field context. */ +const PLACEHOLDER_VALUE = /^(?:\$\{.*\}|\$\(.*\)|\$[A-Za-z_]\w*|#\{.*\}#?|\{\{.*\}\}|<[^>]*>|%[A-Za-z_]\w*%|__[A-Za-z0-9_]+__|string|null|true|false|none|undefined|\*+)$/i; +const KEY_VAULT_SECRET_ID = /^https:\/\/[^/\s]+\.vault\.(?:azure\.net|azure\.cn|usgovcloudapi\.net|microsoftazure\.de)\/secrets\//i; +/** Rules could not be loaded or are invalid. Messages describe config only. */ +export class RedactionConfigError extends Error { + constructor(message) { + super(message); + this.name = 'RedactionConfigError'; + } +} +// --------------------------------------------------------------------------- +// Settings and loading +// --------------------------------------------------------------------------- +const OFF_VALUES = new Set(['off', '0', 'false', 'no', 'disabled']); +const ON_VALUES = new Set(['on', '1', 'true', 'yes', 'enabled']); +/** Unknown values fall back to the default, which is the safe side for both switches. */ +function parseToggle(raw, fallback) { + const value = raw?.trim().toLowerCase(); + if (!value) + return fallback; + if (OFF_VALUES.has(value)) + return false; + if (ON_VALUES.has(value)) + return true; + return fallback; +} +/** + * EPISODIC_MEMORY_REDACTION on (fork default) | off + * EPISODIC_MEMORY_REDACTION_RULES path to a custom rules file + * EPISODIC_MEMORY_REDACTION_STRICT 1 (default) | 0; strict fails closed on a rules-load error + */ +export function getRedactionSettings(env = process.env) { + return { + enabled: parseToggle(env.EPISODIC_MEMORY_REDACTION, REDACTION_ENABLED_BY_DEFAULT), + strict: parseToggle(env.EPISODIC_MEMORY_REDACTION_STRICT, true), + rulesPath: env.EPISODIC_MEMORY_REDACTION_RULES || undefined, + }; +} +function readRulesFile(settings) { + let rulesPath = settings.rulesPath; + if (!rulesPath) { + const candidate = path.join(getSuperpowersDir(), REDACTION_RULES_FILENAME); + if (!fs.existsSync(candidate)) + return {}; + rulesPath = candidate; + } + let raw; + try { + raw = fs.readFileSync(rulesPath, 'utf-8'); + } + catch (error) { + throw new RedactionConfigError(`Redaction rules failed to load from ${rulesPath}: ${error instanceof Error ? error.message : String(error)}`); + } + try { + return JSON.parse(raw); + } + catch (error) { + throw new RedactionConfigError(`Redaction rules file ${rulesPath} is not valid JSON: ${error instanceof Error ? error.message : String(error)}`); + } +} +function replaceById(base, additions) { + const out = [...base]; + for (const item of additions) { + const at = out.findIndex(existing => existing.id === item.id); + if (at >= 0) + out[at] = item; + else + out.push(item); + } + return out; +} +/** + * Merge a rules file with the bundled defaults and validate the result. + * Throws RedactionConfigError on any invalid rule. + */ +export function loadRedactionConfig(file) { + if (!file || typeof file !== 'object' || Array.isArray(file)) { + throw new RedactionConfigError('Redaction rules file must contain a JSON object'); + } + for (const key of ['rules', 'allowlist', 'disableRules']) { + if (file[key] !== undefined && !Array.isArray(file[key])) { + throw new RedactionConfigError(`Redaction rules file: "${key}" must be an array`); + } + } + for (const key of ['entropy', 'secretFields']) { + if (file[key] !== undefined && (typeof file[key] !== 'object' || file[key] === null || Array.isArray(file[key]))) { + throw new RedactionConfigError(`Redaction rules file: "${key}" must be an object`); + } + } + const includeDefaults = file.includeDefaults !== false; + const disabled = new Set(file.disableRules ?? []); + const baseRules = includeDefaults ? DEFAULT_REDACTION_CONFIG.rules : []; + const baseAllowlist = includeDefaults ? DEFAULT_REDACTION_CONFIG.allowlist : []; + const config = { + rules: replaceById(baseRules, file.rules ?? []).filter(rule => !disabled.has(rule.id)), + allowlist: replaceById(baseAllowlist, file.allowlist ?? []), + entropy: { ...DEFAULT_REDACTION_CONFIG.entropy, ...(file.entropy ?? {}) }, + secretFields: { ...DEFAULT_REDACTION_CONFIG.secretFields, ...(file.secretFields ?? {}) }, + }; + compileConfig(config); // validate + return config; +} +/** + * Load the active redactor from the environment. + * + * Returns null when redaction is off. When the rules can't be loaded: strict + * mode (the default) throws RedactionConfigError so callers fail closed; + * non-strict mode warns and returns null, so text passes through unredacted. + */ +export function loadRedactor(env = process.env, warn = message => console.error(message)) { + const settings = getRedactionSettings(env); + if (!settings.enabled) + return null; + try { + return createRedactor(loadRedactionConfig(readRulesFile(settings))); + } + catch (error) { + const err = error instanceof RedactionConfigError + ? error + : new RedactionConfigError(`Redaction rules failed to load: ${error instanceof Error ? error.message : String(error)}`); + if (settings.strict) + throw err; + warn(`episodic-memory: ${err.message}. EPISODIC_MEMORY_REDACTION_STRICT=0, so conversations ` + + 'will be archived and indexed WITHOUT redaction this run.'); + return null; + } +} +function checkFlags(flags, what) { + const value = flags ?? ''; + if (typeof value !== 'string' || !ALLOWED_FLAGS.test(value)) { + throw new RedactionConfigError(`${what}: flags must be a combination of "imsu"`); + } + return value; +} +function compileRule(spec) { + if (!spec || typeof spec !== 'object') { + throw new RedactionConfigError('Redaction rule must be an object'); + } + if (typeof spec.id !== 'string' || !RULE_ID_PATTERN.test(spec.id)) { + throw new RedactionConfigError(`Redaction rule id ${JSON.stringify(spec.id)} is invalid: use lowercase letters, digits and dashes`); + } + if (RESERVED_IDS.has(spec.id)) { + throw new RedactionConfigError(`Redaction rule id "${spec.id}" is reserved`); + } + const what = `Redaction rule "${spec.id}"`; + if (typeof spec.pattern !== 'string' || spec.pattern.length === 0) { + throw new RedactionConfigError(`${what}: pattern must be a non-empty string`); + } + const flags = checkFlags(spec.flags, what); + let regex; + let groupCount; + try { + regex = new RegExp(spec.pattern, flags + 'gd'); + groupCount = new RegExp(`(?:${spec.pattern})|`, flags).exec('').length - 1; + } + catch (error) { + throw new RedactionConfigError(`${what}: invalid regular expression (${error instanceof Error ? error.message : String(error)})`); + } + if (new RegExp(spec.pattern, flags).test('')) { + throw new RedactionConfigError(`${what}: pattern matches the empty string`); + } + const secretGroups = Array.isArray(spec.secretGroup) ? spec.secretGroup : [spec.secretGroup ?? 0]; + if (secretGroups.length === 0) { + throw new RedactionConfigError(`${what}: secretGroup must not be an empty array`); + } + for (const group of secretGroups) { + if (!Number.isInteger(group) || group < 0 || group > groupCount) { + throw new RedactionConfigError(`${what}: secretGroup ${group} does not exist in the pattern (${groupCount} group(s))`); + } + } + if (spec.useAllowlist !== undefined && typeof spec.useAllowlist !== 'boolean') { + throw new RedactionConfigError(`${what}: useAllowlist must be a boolean`); + } + let keywords; + if (spec.keywords !== undefined) { + if (!Array.isArray(spec.keywords) || spec.keywords.some(k => typeof k !== 'string' || k.length === 0)) { + throw new RedactionConfigError(`${what}: keywords must be an array of non-empty strings`); + } + keywords = spec.keywords.length > 0 ? spec.keywords.map(k => k.toLowerCase()) : undefined; + } + return { id: spec.id, regex, keywords, secretGroups, useAllowlist: spec.useAllowlist !== false }; +} +function compileAllowlist(spec) { + const what = `Redaction allowlist entry ${JSON.stringify(spec?.id)}`; + if (!spec || typeof spec.pattern !== 'string' || spec.pattern.length === 0) { + throw new RedactionConfigError(`${what}: pattern must be a non-empty string`); + } + const flags = checkFlags(spec.flags, what); + try { + return new RegExp(`^(?:${spec.pattern})$`, flags); + } + catch (error) { + throw new RedactionConfigError(`${what}: invalid regular expression (${error instanceof Error ? error.message : String(error)})`); + } +} +function compileConfig(config) { + const seen = new Set(); + const rules = config.rules.map(spec => { + const rule = compileRule(spec); + if (seen.has(rule.id)) + throw new RedactionConfigError(`Duplicate redaction rule id "${rule.id}"`); + seen.add(rule.id); + return rule; + }); + const allowlist = config.allowlist.map(compileAllowlist); + const e = config.entropy; + if (typeof e.enabled !== 'boolean' || typeof e.requireKeyword !== 'boolean' || + !Number.isInteger(e.minLength) || e.minLength < 8 || + typeof e.threshold !== 'number' || !(e.threshold > 0) || + !Number.isInteger(e.window) || e.window < 0 || + !Array.isArray(e.keywords) || e.keywords.some(k => typeof k !== 'string' || !k)) { + throw new RedactionConfigError('Redaction entropy settings are invalid (need enabled, requireKeyword: boolean; minLength >= 8; threshold > 0; window >= 0; keywords: string[])'); + } + const f = config.secretFields; + if (!f || typeof f.enabled !== 'boolean' || typeof f.keyPattern !== 'string' || f.keyPattern.length === 0) { + throw new RedactionConfigError('Redaction secretFields settings are invalid (need enabled: boolean; keyPattern: non-empty string)'); + } + let secretField = null; + if (f.enabled) { + try { + secretField = new RegExp(`(?:${f.keyPattern})$`); + } + catch (error) { + throw new RedactionConfigError(`Redaction secretFields.keyPattern: invalid regular expression (${error instanceof Error ? error.message : String(error)})`); + } + if (secretField.test('')) + throw new RedactionConfigError('Redaction secretFields.keyPattern matches the empty string'); + } + return { rules, allowlist, entropy: { ...e, keywords: e.keywords.map(k => k.toLowerCase()) }, secretField }; +} +// --------------------------------------------------------------------------- +// Redaction engine +// --------------------------------------------------------------------------- +function tokenFor(ruleId) { + return `${TOKEN_PREFIX}${ruleId}]`; +} +const WHOLE_TOKEN = /^\[REDACTED:[a-z0-9][a-z0-9-]*\]$/; +/** The token for one redacted value: onMatch's, if it returns a valid one. */ +function tokenForMatch(ruleId, value, onMatch) { + const custom = onMatch?.(ruleId, value); + return typeof custom === 'string' && WHOLE_TOKEN.test(custom) ? custom : tokenFor(ruleId); +} +function tokenSpans(text) { + const spans = []; + for (const m of text.matchAll(TOKEN_PATTERN)) + spans.push([m.index, m.index + m[0].length]); + return spans; +} +function overlapsAny(spans, start, end) { + for (const [s, e] of spans) + if (start < e && end > s) + return true; + return false; +} +/** + * Rewrite text[start, end) keeping the existing tokens in it and replacing each + * stretch between them that has a letter or digit with `token`. Returns null + * when nothing outside the tokens needs redacting, which keeps this idempotent. + */ +function redactAroundTokens(text, spans, start, end, token) { + let out = ''; + let pos = start; + let replaced = false; + const flush = (to) => { + const piece = text.slice(pos, to); + if (/[A-Za-z0-9]/.test(piece)) { + out += token(); + replaced = true; + } + else { + out += piece; + } + }; + for (const [s, e] of spans) { + if (e <= start || s >= end) + continue; + if (s > pos) + flush(s); + out += text.slice(Math.max(s, pos), Math.min(e, end)); + pos = Math.min(e, end); + } + if (pos < end) + flush(end); + return replaced ? out : null; +} +/** + * Describe a value without revealing it: length, character classes and + * Shannon entropy (bits per char), e.g. "len=40 aA9- H=4.9". Lets someone + * reviewing hits tell a random key from a word or a placeholder. + */ +export function describeShape(value) { + const classes = (/[a-z]/.test(value) ? 'a' : '') + + (/[A-Z]/.test(value) ? 'A' : '') + + (/[0-9]/.test(value) ? '9' : '') + + (/[^A-Za-z0-9\s]/.test(value) ? '-' : '') + + (/\s/.test(value) ? '_' : ''); + return `len=${value.length} ${classes || '?'} H=${shannonEntropy(value).toFixed(1)}`; +} +/** Each `[REDACTED:]` token in `text`, left to right. */ +export function findRedactionTokens(text) { + const out = []; + for (const m of text.matchAll(TOKEN_PATTERN)) { + out.push({ ruleId: m[0].slice(TOKEN_PREFIX.length, -1), start: m.index, end: m.index + m[0].length }); + } + return out; +} +function shannonEntropy(text) { + const counts = new Map(); + for (const ch of text) + counts.set(ch, (counts.get(ch) ?? 0) + 1); + let entropy = 0; + for (const n of counts.values()) { + const p = n / text.length; + entropy -= p * Math.log2(p); + } + return entropy; +} +/** + * Replace each accepted match (or its secretGroup) with a token. A candidate + * that overlaps existing tokens keeps them, and only the text around them is + * redacted (so re-running is a no-op). A candidate is skipped when it fully + * matches an allowlist pattern. + */ +function applyRule(text, rule, isAllowed, onMatch) { + const spans = text.includes(TOKEN_PREFIX) ? tokenSpans(text) : null; + let out = ''; + let last = 0; + let count = 0; + for (const m of text.matchAll(rule.regex)) { + let range; + for (const group of rule.secretGroups) { + range = m.indices?.[group]; + if (range) + break; + } + if (!range) + continue; + const [start, end] = range; + if (end <= start || start < last) + continue; + if (rule.useAllowlist && isAllowed(text.slice(start, end))) + continue; + if (spans && overlapsAny(spans, start, end)) { + let token; + const rewritten = redactAroundTokens(text, spans, start, end, () => (token ??= tokenForMatch(rule.id, text.slice(start, end).replace(TOKEN_PATTERN, ''), onMatch))); + if (rewritten === null) + continue; + out += text.slice(last, start) + rewritten; + last = end; + count++; + continue; + } + out += text.slice(last, start) + tokenForMatch(rule.id, text.slice(start, end), onMatch); + last = end; + count++; + } + return count === 0 ? { text, count } : { text: out + text.slice(last), count }; +} +function applyEntropy(text, spec, isAllowed, onMatch) { + const candidates = new RegExp(`[A-Za-z0-9+/=_.~-]{${spec.minLength},}`, 'g'); + const spans = text.includes(TOKEN_PREFIX) ? tokenSpans(text) : null; + let out = ''; + let last = 0; + let count = 0; + for (const m of text.matchAll(candidates)) { + const start = m.index; + const end = start + m[0].length; + if (spans && overlapsAny(spans, start, end)) + continue; + if (isAllowed(m[0])) + continue; + if (shannonEntropy(m[0]) < spec.threshold) + continue; + if (spec.requireKeyword) { + let before = text.slice(Math.max(0, start - spec.window), start); + before = before.slice(before.lastIndexOf('\n') + 1).toLowerCase(); + if (!spec.keywords.some(k => before.includes(k))) + continue; + } + out += text.slice(last, start) + tokenForMatch(ENTROPY_RULE_ID, m[0], onMatch); + last = end; + count++; + } + return count === 0 ? { text, count } : { text: out + text.slice(last), count }; +} +export function createRedactor(config) { + const compiled = compileConfig(config); + const isAllowed = (secret) => compiled.allowlist.some(re => re.test(secret)); + return { + ruleIds: compiled.rules.map(r => r.id), + isSecretField(name) { + if (!compiled.secretField || typeof name !== 'string') + return false; + const normalized = name.toLowerCase().replace(/[^a-z0-9]/g, '').replace(/\d+$/, ''); + return normalized.length > 0 && compiled.secretField.test(normalized); + }, + redact(text, ctx) { + if (typeof text !== 'string' || text.length === 0) + return { text, findings: [] }; + let current = text; + let lower = null; + const counts = new Map(); + for (const rule of compiled.rules) { + if (rule.keywords) { + lower ??= current.toLowerCase(); + if (!rule.keywords.some(k => lower.includes(k))) + continue; + } + const r = applyRule(current, rule, isAllowed, ctx?.onMatch); + if (r.count > 0) { + current = r.text; + lower = null; + counts.set(rule.id, (counts.get(rule.id) ?? 0) + r.count); + } + } + if (compiled.entropy.enabled) { + const r = applyEntropy(current, compiled.entropy, isAllowed, ctx?.onMatch); + if (r.count > 0) { + current = r.text; + counts.set(ENTROPY_RULE_ID, r.count); + } + } + return { text: current, findings: [...counts].map(([ruleId, count]) => ({ ruleId, count })) }; + }, + }; +} +// --------------------------------------------------------------------------- +// Findings +// --------------------------------------------------------------------------- +/** Aggregates findings by rule id. Never holds matched values. */ +export class FindingsTally { + counts = new Map(); + add(findings) { + for (const f of findings) + this.counts.set(f.ruleId, (this.counts.get(f.ruleId) ?? 0) + f.count); + } + get total() { + let n = 0; + for (const c of this.counts.values()) + n += c; + return n; + } + toArray() { + return [...this.counts].map(([ruleId, count]) => ({ ruleId, count })); + } +} +/** "7 value(s) redacted (azure-storage-key: 2, jwt: 5)" */ +export function formatFindings(findings) { + const total = findings.reduce((n, f) => n + f.count, 0); + if (total === 0) + return 'no values redacted'; + const detail = [...findings] + .sort((a, b) => b.count - a.count || a.ruleId.localeCompare(b.ruleId)) + .map(f => `${f.ruleId}: ${f.count}`) + .join(', '); + return `${total} value(s) redacted (${detail})`; +} +// --------------------------------------------------------------------------- +// JSON / JSONL / files +// --------------------------------------------------------------------------- +/** + * Redact a parsed JSON tree in place: string values by the text rules, values + * by their field name (secretFields), and keys that are themselves secrets. + */ +function redactTree(node, redactor, ctx, tally) { + if (typeof node === 'string') { + const r = redactor.redact(node, ctx); + if (r.findings.length === 0) + return { value: node, changed: false }; + tally?.add(r.findings); + return { value: r.text, changed: r.text !== node }; + } + if (node === null || typeof node !== 'object') + return { value: node, changed: false }; + let changed = false; + if (Array.isArray(node)) { + for (let i = 0; i < node.length; i++) { + const r = redactTree(node[i], redactor, ctx, tally); + if (r.changed) { + node[i] = r.value; + changed = true; + } + } + return { value: node, changed }; + } + const obj = node; + const keys = Object.keys(obj); + for (const key of keys) { + const r = redactTree(obj[key], redactor, ctx, tally); + if (r.changed) { + setOwn(obj, key, r.value); + changed = true; + } + } + // Field-name context: the key (or a {name, value} pair's name, or a Key + // Vault secret id) says the value is a secret even when its shape matches + // no rule — e.g. a letters-only clientSecret field in a structured + // MCP result. Runs after the text rules; a value they redacted only in part + // is still replaced whole, since the field name says all of it is secret. + const redactWhole = (key) => { + const value = obj[key]; + if (!isRedactableWhole(value)) + return; + setOwn(obj, key, tokenForMatch(FIELD_RULE_ID, value, ctx?.onMatch)); + tally?.add([{ ruleId: FIELD_RULE_ID, count: 1 }]); + changed = true; + }; + for (const key of keys) { + if (redactor.isSecretField(key)) + redactWhole(key); + } + const lower = new Map(keys.map(k => [k.toLowerCase(), k])); + const valueKey = lower.get('value'); + if (valueKey !== undefined) { + const nameKey = lower.get('name') ?? lower.get('key'); + const nameValue = nameKey !== undefined ? obj[nameKey] : undefined; + const id = lower.has('id') ? obj[lower.get('id')] : undefined; + if ((typeof nameValue === 'string' && redactor.isSecretField(nameValue)) || + (typeof id === 'string' && KEY_VAULT_SECRET_ID.test(id))) { + redactWhole(valueKey); + } + } + // Keys themselves: a secret used as a key (a token-keyed cache) is renamed + // to its token. Harness keys never match a rule, so structure is unchanged. + let renamed = null; + for (let i = 0; i < keys.length; i++) { + const r = redactor.redact(keys[i], ctx); + if (r.findings.length === 0) + continue; + tally?.add(r.findings); + renamed ??= keys.map(k => [k, obj[k]]); + renamed[i][0] = r.text; + } + if (renamed) { + const rebuilt = {}; + for (const [key, value] of renamed) { + let unique = key; + for (let n = 2; Object.prototype.hasOwnProperty.call(rebuilt, unique); n++) + unique = `${key}#${n}`; + setOwn(rebuilt, unique, value); + } + return { value: rebuilt, changed: true }; + } + return { value: obj, changed }; +} +function setOwn(obj, key, value) { + // defineProperty so a "__proto__" key from JSON.parse stays an own property. + Object.defineProperty(obj, key, { value, writable: true, enumerable: true, configurable: true }); +} +// A value that is only tokens is already redacted; anything else in a secret +// field, even alongside a token, is replaced whole. +const ONLY_TOKENS = /^\s*(?:\[REDACTED:[a-z0-9][a-z0-9-]*\]\s*)+$/; +function isRedactableWhole(value) { + return typeof value === 'string' && + value.trim().length > 0 && + !ONLY_TOKENS.test(value) && + !PLACEHOLDER_VALUE.test(value.trim()); +} +/** + * Redact one JSONL line. Valid JSON stays valid (string values are redacted + * and the line is re-serialized only if something changed); a line that isn't + * JSON (e.g. a torn last line mid-write) is redacted as raw text. A trailing + * `\r` is preserved. + */ +export function redactJsonlLine(line, redactor, ctx, tally) { + const cr = line.endsWith('\r') ? '\r' : ''; + const body = cr ? line.slice(0, -1) : line; + if (body.trim().length === 0) + return line; + let parsed; + try { + parsed = JSON.parse(body); + } + catch { + const r = redactor.redact(body, ctx); + if (r.findings.length === 0) + return line; + tally?.add(r.findings); + return r.text + cr; + } + const r = redactTree(parsed, redactor, ctx, tally); + return r.changed ? JSON.stringify(r.value) + cr : line; +} +const COPY_CHUNK_BYTES = 1 << 20; // 1 MiB +/** + * Stream `src` to `dest`, redacting line by line. Writes `dest` directly; + * callers that need atomicity write to a temp path and rename. Line count and + * trailing-newline shape match the source exactly, so archive line numbers + * (exchanges.line_start/line_end, MCP read ranges) stay valid. + */ +export function copyFileRedacted(src, dest, redactor, ctx, tally) { + const fdIn = fs.openSync(src, 'r'); + let fdOut; + try { + // Create with the source's mode, as copyFileSync would, so a 0600 + // transcript doesn't become world-readable in the archive. + fdOut = fs.openSync(dest, 'w', fs.fstatSync(fdIn).mode & 0o777); + const buf = Buffer.allocUnsafe(COPY_CHUNK_BYTES); + const decoder = new StringDecoder('utf8'); + let pending = ''; + let bytesRead; + while ((bytesRead = fs.readSync(fdIn, buf, 0, buf.length, null)) > 0) { + let scanFrom = pending.length; + pending += decoder.write(buf.subarray(0, bytesRead)); + const out = []; + let start = 0; + let nl; + while ((nl = pending.indexOf('\n', scanFrom)) !== -1) { + out.push(redactJsonlLine(pending.slice(start, nl), redactor, ctx, tally), '\n'); + start = nl + 1; + scanFrom = start; + } + if (out.length > 0) + fs.writeSync(fdOut, out.join('')); + pending = pending.slice(start); + } + pending += decoder.end(); + if (pending.length > 0) + fs.writeSync(fdOut, redactJsonlLine(pending, redactor, ctx, tally)); + } + finally { + fs.closeSync(fdIn); + if (fdOut !== undefined) + fs.closeSync(fdOut); + } +} diff --git a/dist/search.js b/dist/search.js index 281b4f0f..ae5497de 100644 --- a/dist/search.js +++ b/dist/search.js @@ -1,4 +1,5 @@ import { initDatabase } from './db.js'; +import { createSearchAdmission } from './record-admission.js'; import { initEmbeddings, generateQueryEmbedding } from './embeddings.js'; import { isErroredSentinel } from './summary-sentinel.js'; import fs from 'fs'; @@ -160,67 +161,74 @@ export async function searchConversations(query, options = {}) { validateISODate(after, '--after'); if (before) validateISODate(before, '--before'); + const admission = createSearchAdmission(); const db = initDatabase(); + const admitted = (row) => admission.admit(exchangeFromRow(row)) !== null; let results = []; const { sql: filterClause, params: filterParams } = buildSearchFilters(options); - if (mode === 'vector' || mode === 'both') { - // Vector similarity search. - // vec0 applies KNN before the WHERE clause and before our sidechain - // de-rank, so we over-fetch candidates and trim after the final ordering. - await initEmbeddings(); - const queryEmbedding = await generateQueryEmbedding(query); - const k = limit * 3; - const stmt = db.prepare(` - SELECT - ${EXCHANGE_SELECT_COLUMNS}, - vec.distance - FROM vec_exchanges AS vec - JOIN exchanges AS e ON vec.id = e.id - WHERE vec.embedding MATCH ? - AND k = ? - ${sidechainClause} - ${filterClause} - ORDER BY (vec.distance + e.is_sidechain * ?) ASC - `); - results = stmt.all(Buffer.from(new Float32Array(queryEmbedding).buffer), k, ...filterParams, SIDECHAIN_DISTANCE_PENALTY); - if (results.length > limit) { - results = results.slice(0, limit); + try { + if (mode === 'vector' || mode === 'both') { + // Order the filtered candidates before admission. The vec0 KNN API caps k; + // increasing k cannot find safe results beyond a large excluded prefix. + await initEmbeddings(); + const queryEmbedding = await generateQueryEmbedding(query); + const stmt = db.prepare(` + SELECT ${EXCHANGE_SELECT_COLUMNS}, + vec_distance_l2(vec.embedding, ?) AS retrieval_distance + FROM vec_exchanges AS vec + JOIN exchanges AS e ON vec.id = e.id + WHERE 1 = 1 ${sidechainClause} ${filterClause} + ORDER BY (retrieval_distance + e.is_sidechain * ?) ASC + `); + for (const row of stmt.iterate(Buffer.from(new Float32Array(queryEmbedding).buffer), ...filterParams, SIDECHAIN_DISTANCE_PENALTY)) { + if (admitted(row)) + results.push(row); + if (results.length === limit) + break; + } } - } - if (mode === 'text' || mode === 'both') { - // Text search: AND of per-token LIKE patterns so multi-word queries match - // when every term appears somewhere in the exchange (any order, either field). - // See #127 — whole-query contiguous substring matching returned empty for - // typical multi-word searches that are not verbatim phrases. Sidechain rows - // are de-ranked (ordered after main-thread rows), not excluded (#128). - const { sql: textMatchSql, params: textMatchParams } = buildTextMatchClause(query); - const textStmt = db.prepare(` - SELECT - ${EXCHANGE_SELECT_COLUMNS}, - 0 as distance - FROM exchanges AS e - WHERE ${textMatchSql} - ${sidechainClause} - ${filterClause} - ORDER BY e.is_sidechain ASC, e.timestamp DESC - LIMIT ? - `); - const textResults = textStmt.all(...textMatchParams, ...filterParams, limit); - if (mode === 'both') { - // Merge and deduplicate by ID - const seenIds = new Set(results.map(r => r.id)); - for (const textResult of textResults) { - if (!seenIds.has(textResult.id)) { - results.push(textResult); + if (mode === 'text' || mode === 'both') { + // Text search: AND of per-token LIKE patterns so multi-word queries match + // when every term appears somewhere in the exchange (any order, either field). + // See #127 — whole-query contiguous substring matching returned empty for + // typical multi-word searches that are not verbatim phrases. Sidechain rows + // are de-ranked (ordered after main-thread rows), not excluded (#128). + const { sql: textMatchSql, params: textMatchParams } = buildTextMatchClause(query); + const textStmt = db.prepare(` + SELECT + ${EXCHANGE_SELECT_COLUMNS}, + 0 as retrieval_distance + FROM exchanges AS e + WHERE ${textMatchSql} + ${sidechainClause} + ${filterClause} + ORDER BY e.is_sidechain ASC, e.timestamp DESC + `); + const textResults = []; + for (const row of textStmt.iterate(...textMatchParams, ...filterParams)) { + if (admitted(row)) + textResults.push(row); + if (textResults.length === limit) + break; + } + if (mode === 'both') { + // Merge and deduplicate by ID + const seenIds = new Set(results.map(r => r.id)); + for (const textResult of textResults) { + if (!seenIds.has(textResult.id)) { + results.push(textResult); + } } } - } - else { - results = textResults; + else { + results = textResults; + } } } - db.close(); - return results.map((row) => { + finally { + db.close(); + } + const output = results.map((row) => { const exchange = exchangeFromRow(row); // Try to load summary if available. Skip error sentinels (#96) so failed // summarizations don't surface as the conversation's summary in results. @@ -229,7 +237,7 @@ export async function searchConversations(query, options = {}) { if (fs.existsSync(summaryPath)) { const raw = fs.readFileSync(summaryPath, 'utf-8'); if (!isErroredSentinel(raw)) { - summary = raw.trim(); + summary = admission.summary(exchange, raw.trim()); } } // Create snippet (first 200 chars, collapse newlines) @@ -237,11 +245,13 @@ export async function searchConversations(query, options = {}) { const snippet = snippetText + (exchange.userMessage.length > 200 ? '...' : ''); return { exchange, - similarity: mode === 'text' ? undefined : l2DistanceToCosineSimilarity(row.distance), + similarity: mode === 'text' ? undefined : l2DistanceToCosineSimilarity(row.retrieval_distance), snippet, summary }; }); + admission.verify(); + return output; } // Helper function to count lines in a file efficiently async function countLines(filePath) { diff --git a/dist/show-cli.js b/dist/show-cli.js index 53ca9739..c5693d30 100644 --- a/dist/show-cli.js +++ b/dist/show-cli.js @@ -1,4 +1,4 @@ -import { readFileSync } from 'fs'; +import { readAdmittedConversation } from './record-admission.js'; import { formatConversationAsMarkdown, formatConversationAsHTML } from './show.js'; const args = process.argv.slice(2); // Parse arguments @@ -42,7 +42,7 @@ if (!filePath) { process.exit(1); } try { - const jsonl = readFileSync(filePath, 'utf-8'); + const jsonl = await readAdmittedConversation(filePath); if (format === 'html') { console.log(formatConversationAsHTML(jsonl)); } diff --git a/dist/show.js b/dist/show.js index 2f297196..9d1d01f9 100644 --- a/dist/show.js +++ b/dist/show.js @@ -1,10 +1,11 @@ import { marked } from 'marked'; export function formatConversationAsMarkdown(jsonl, startLine, endLine) { - const allLines = jsonl.trim().split('\n').filter(line => line.trim()); + const allLines = jsonl.split('\n'); // Apply line range if specified (1-indexed, inclusive) - const lines = startLine !== undefined || endLine !== undefined + const selectedLines = startLine !== undefined || endLine !== undefined ? allLines.slice(startLine !== undefined ? startLine - 1 : 0, endLine !== undefined ? endLine : undefined) : allLines; + const lines = selectedLines.filter(line => line.trim()); if (isCodexRollout(lines)) { return formatCodexConversationAsMarkdown(lines); } diff --git a/dist/summarizer.d.ts b/dist/summarizer.d.ts index 4c97be52..ce808bc1 100644 --- a/dist/summarizer.d.ts +++ b/dist/summarizer.d.ts @@ -57,10 +57,11 @@ export interface CodexSummarizerCommand { command: string; args: string[]; prompt: string; - sessionId: string; + sessionId?: string; model?: string; versionArgs?: string[]; skipVersionCheck?: boolean; + isolated?: boolean; } /** * Get API environment overrides for summarization calls. @@ -150,10 +151,11 @@ export declare function buildSummarizerQueryOptions(args: { }): Record; export declare function buildCodexSummaryPrompt(): string; export declare function buildCodexSummarizerCommand(args: { - sessionId: string; + sessionId?: string; prompt: string; model?: string; codexBin?: string; + isolated?: boolean; }): CodexSummarizerCommand; export declare function runCodexCommand(command: CodexSummarizerCommand): Promise; /** @@ -166,10 +168,13 @@ export declare function runCodexCommand(command: CodexSummarizerCommand): Promis * * Default to `undefined` so `app-server` uses the current Codex config * (`~/.codex/config.toml#model`). Operators can override via - * `EPISODIC_MEMORY_CODEX_MODEL` if they need a specific model id (e.g. an - * API-key user wanting `gpt-5.5-codex`). + * `EPISODIC_MEMORY_CODEX_MODEL` if they need a supported model id for their + * selected Codex route. * * See https://github.com/obra/episodic-memory/issues/98. */ export declare function getCodexModel(_exchanges: ConversationExchange[]): string | undefined; -export declare function summarizeConversation(exchanges: ConversationExchange[], sessionId?: string): Promise; +export interface SummarizeOptions { + allowResume?: boolean; +} +export declare function summarizeConversation(exchanges: ConversationExchange[], sessionId?: string, options?: SummarizeOptions): Promise; diff --git a/dist/summarizer.js b/dist/summarizer.js index 60719809..893fc3f7 100644 --- a/dist/summarizer.js +++ b/dist/summarizer.js @@ -1,4 +1,8 @@ import fs from 'fs'; +import os from 'node:os'; +import path from 'node:path'; +import { admitExchange, readRecordExclusions } from './record-admission.js'; +import { getClaudeDir } from './paths.js'; import { query } from '@anthropic-ai/claude-agent-sdk'; import { SUMMARIZER_CONTEXT_MARKER } from './constants.js'; import { VERSION } from './version.js'; @@ -293,6 +297,25 @@ function extractSummary(text) { // Fallback if no tags found return text.trim(); } +function readClaudeSummaryAuthentication() { + let settings; + try { + settings = JSON.parse(fs.readFileSync(path.join(getClaudeDir(), 'settings.json'), 'utf8')); + } + catch (error) { + if (error.code === 'ENOENT') + return {}; + // Parser and filesystem errors can contain private setting values or paths. + throw new Error('Cannot read Claude summarizer authentication settings'); + } + const apiKeyHelper = settings?.apiKeyHelper; + const baseUrl = settings?.env?.ANTHROPIC_BASE_URL; + if ((apiKeyHelper !== undefined && typeof apiKeyHelper !== 'string') || + (baseUrl !== undefined && typeof baseUrl !== 'string')) { + throw new Error('Cannot read Claude summarizer authentication settings'); + } + return { apiKeyHelper, baseUrl }; +} /** * Build the options object passed to the Claude Agent SDK's query() for a * summarization call. @@ -307,10 +330,16 @@ function extractSummary(text) { */ export function buildSummarizerQueryOptions(args) { const { model, sessionId, cwd } = args; + const authentication = readClaudeSummaryAuthentication(); + const env = getApiEnv(); + if (!env.ANTHROPIC_BASE_URL && authentication.baseUrl) { + env.ANTHROPIC_BASE_URL = authentication.baseUrl; + } return { model, max_tokens: 4096, - env: getApiEnv(), + env, + ...(authentication.apiKeyHelper ? { settings: { apiKeyHelper: authentication.apiKeyHelper } } : {}), resume: sessionId, persistSession: false, // Summarizers never call tools — disabling them stops a resumed live @@ -325,7 +354,7 @@ export function buildSummarizerQueryOptions(args) { // strictMcpConfig only blocks ~/.claude.json MCP servers; user settings // still load every enabled plugin (each with its own MCP server) plus the // user's Stop/Notification hooks (#136) into the subprocess. Summarizers - // need no plugins, hooks, or user settings at all. + // need no behavior settings. Only authentication is passed separately. settingSources: [], // Resume looks up the session under ~/.claude/projects//, so pass the recorded cwd when it still exists on disk. ...(cwd && fs.existsSync(cwd) ? { cwd } : {}), @@ -362,12 +391,27 @@ Bad: } export function buildCodexSummarizerCommand(args) { const command = args.codexBin || process.env.EPISODIC_MEMORY_CODEX_BIN || 'codex'; + if (args.isolated && args.sessionId) + throw new Error('An isolated summary cannot resume a source session'); + const restrictions = args.isolated ? [ + ...['shell_tool', 'unified_exec', 'shell_snapshot', 'apps', 'plugins', 'remote_plugin', + 'hooks', 'multi_agent', 'multi_agent_v2', 'memories', 'goals', 'computer_use', + 'browser_use', 'browser_use_external', 'image_generation', 'workspace_dependencies', + 'skill_search', 'skill_mcp_dependency_install', 'tool_suggest', 'code_mode_host', + 'sleep_tool', 'view_image', 'default_mode_request_user_input'] + .map(feature => `features.${feature}=false`), + 'features.skip_host_skill_discovery=true', 'skills.bundled.enabled=false', + 'skills.include_instructions=false', 'project_doc_max_bytes=0', + 'tools.experimental_request_user_input.enabled=false', 'web_search="disabled"', + 'history.persistence="none"', + ].flatMap(value => ['-c', value]) : []; return { command, - args: ['app-server'], + args: ['app-server', ...restrictions], prompt: args.prompt, sessionId: args.sessionId, model: args.model, + ...(args.isolated ? { isolated: true } : {}), }; } async function callClaude(prompt, sessionId, useFallback = false, cwd) { @@ -387,6 +431,7 @@ async function callClaude(prompt, sessionId, useFallback = false, cwd) { } return result; } +const CODEX_SUMMARY_TERMINATION_GRACE_MS = 1024; function appServerTimeoutMs() { const configured = Number(process.env.EPISODIC_MEMORY_CODEX_SUMMARY_TIMEOUT_MS); return Number.isFinite(configured) && configured > 0 ? configured : 120000; @@ -421,6 +466,9 @@ async function assertSupportedCodexVersion(command) { } const output = await readCommandOutput(command.command, command.versionArgs || ['--version']); const version = parseCodexCliVersion(output); + if (command.isolated && (!version || !versionMeetsMinimum(version, '0.154.0'))) { + throw new Error('Isolated record-policy summarization requires codex-cli >= 0.154.0'); + } if (!version || !versionMeetsMinimum(version)) { throw new Error(codexVersionRequirementMessage(output)); } @@ -440,10 +488,16 @@ function requireTurnId(result, method) { return turnId; } export async function runCodexCommand(command) { + if (command.isolated && command.sessionId) + throw new Error('An isolated summary cannot resume a source session'); await assertSupportedCodexVersion(command); + const summaryCwd = command.isolated + ? fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'episodic-memory-summary-'))) + : undefined; return new Promise((resolve, reject) => { const child = spawn(command.command, command.args, { env: getApiEnv(), + ...(summaryCwd ? { cwd: summaryCwd } : {}), stdio: ['pipe', 'pipe', 'pipe'] }); let stderr = ''; @@ -451,7 +505,9 @@ export async function runCodexCommand(command) { let nextRequestId = 1; let targetTurnId; let finished = false; + let terminalOutcome; let timeout; + let terminationTimeout; const pending = new Map(); const lines = createInterface({ input: child.stdout }); child.stderr.on('data', chunk => { @@ -465,12 +521,25 @@ export async function runCodexCommand(command) { if (!child.killed) { child.kill('SIGTERM'); } + if (command.isolated && child.exitCode === null && child.signalCode === null) { + // Bound shutdown without removing the owned directory before close. + terminationTimeout = setTimeout(() => { + if (child.exitCode === null && child.signalCode === null) + child.kill('SIGKILL'); + }, CODEX_SUMMARY_TERMINATION_GRACE_MS); + } }; const finish = (error, result = '') => { if (finished) return; finished = true; cleanup(); + if (command.isolated) { + // Wait for process termination before removing its owned directory. + // Provider payloads may repeat input; never persist them in error sidecars. + terminalOutcome = { error: error ? new Error('Isolated Codex summary failed; provider details suppressed') : undefined, result }; + return; + } if (error) { reject(error); } @@ -544,6 +613,25 @@ export async function runCodexCommand(command) { finish(new Error(detail)); } }); + child.on('close', () => { + if (terminationTimeout) + clearTimeout(terminationTimeout); + if (!command.isolated) + return; + try { + if (summaryCwd) + fs.rmSync(summaryCwd, { recursive: true, force: true }); + if (terminalOutcome?.error) + reject(terminalOutcome.error); + else if (terminalOutcome) + resolve(terminalOutcome.result); + else + reject(new Error('Isolated Codex summary ended without a terminal result')); + } + catch { + reject(new Error('Isolated Codex summary cleanup failed')); + } + }); (async () => { try { await send('initialize', { @@ -557,15 +645,45 @@ export async function runCodexCommand(command) { }, }); notify('initialized'); - const fork = await send('thread/fork', { - threadId: command.sessionId, + let isolation = {}; + if (command.isolated) { + const configured = await send('config/read', { includeLayers: false }); + const servers = configured?.config?.mcp_servers; + if (!configured?.config || servers !== undefined && (!servers || typeof servers !== 'object' || Array.isArray(servers))) { + throw new Error('Cannot establish configured MCP boundary'); + } + const listed = await send('skills/list', { cwds: [summaryCwd], forceReload: true }); + if (!Array.isArray(listed?.data) || listed.data.some((entry) => !Array.isArray(entry.skills) || !Array.isArray(entry.errors) || entry.errors.length)) { + throw new Error('Cannot establish configured skill boundary'); + } + const skills = listed.data.flatMap((entry) => entry.skills).map((skill) => { + if (typeof skill.path !== 'string' || !path.isAbsolute(skill.path)) + throw new Error('Invalid configured skill identity'); + return { path: skill.path, enabled: false }; + }); + isolation = { + cwd: summaryCwd, + dynamicTools: [], environments: [], selectedCapabilityRoots: [], + allowProviderModelFallback: false, + baseInstructions: 'Summarize only the admitted transcript. Do not use tools or recover external context.', + developerInstructions: 'Output only a concise factual summary inside .', + config: { + mcp_servers: Object.fromEntries(Object.keys(servers ?? {}).map(name => [name, { enabled: false }])), + 'skills.config': skills, + }, + }; + } + const method = command.sessionId ? 'thread/fork' : 'thread/start'; + const fork = await send(method, { + ...(command.sessionId ? { threadId: command.sessionId } : {}), ephemeral: true, excludeTurns: true, sandbox: 'read-only', approvalPolicy: 'never', ...(command.model ? { model: command.model } : {}), + ...isolation, }); - const forkThreadId = requireThreadId(fork, 'thread/fork'); + const forkThreadId = requireThreadId(fork, method); const turn = await send('turn/start', { threadId: forkThreadId, input: [{ @@ -582,8 +700,8 @@ export async function runCodexCommand(command) { })(); }); } -async function callCodex(prompt, sessionId, model) { - const command = buildCodexSummarizerCommand({ sessionId, prompt, model }); +async function callCodex(prompt, sessionId, model, isolated = false) { + const command = buildCodexSummarizerCommand({ sessionId, prompt, model, isolated }); return runCodexCommand(command); } function chunkExchanges(exchanges, chunkSize) { @@ -597,7 +715,7 @@ function getCodexSessionId(exchanges, sessionId) { if (!exchanges.some(exchange => exchange.harness === 'codex')) { return undefined; } - return sessionId || exchanges.find(exchange => exchange.sessionId)?.sessionId; + return sessionId || exchanges.find(exchange => exchange.harness === 'codex' && exchange.sessionId)?.sessionId; } /** * Resolve the model to pass into Codex `thread/fork` for summarization. @@ -609,15 +727,18 @@ function getCodexSessionId(exchanges, sessionId) { * * Default to `undefined` so `app-server` uses the current Codex config * (`~/.codex/config.toml#model`). Operators can override via - * `EPISODIC_MEMORY_CODEX_MODEL` if they need a specific model id (e.g. an - * API-key user wanting `gpt-5.5-codex`). + * `EPISODIC_MEMORY_CODEX_MODEL` if they need a supported model id for their + * selected Codex route. * * See https://github.com/obra/episodic-memory/issues/98. */ export function getCodexModel(_exchanges) { return process.env.EPISODIC_MEMORY_CODEX_MODEL || undefined; } -export async function summarizeConversation(exchanges, sessionId) { +export async function summarizeConversation(exchanges, sessionId, options = {}) { + const recordPolicy = readRecordExclusions(); + exchanges = exchanges.map(exchange => admitExchange(exchange, recordPolicy)).filter((exchange) => exchange !== null); + const allowResume = options.allowResume !== false && recordPolicy === null; // Handle trivial conversations if (exchanges.length === 0) { return 'Trivial conversation with no substantive content.'; @@ -628,20 +749,20 @@ export async function summarizeConversation(exchanges, sessionId) { return 'Trivial conversation with no substantive content.'; } } - const codexSessionId = getCodexSessionId(exchanges, sessionId); - if (codexSessionId) { - try { - const result = await callCodex(buildCodexSummaryPrompt(), codexSessionId, getCodexModel(exchanges)); - return extractSummary(result); - } - catch (error) { - console.log(` Codex summarizer unavailable, falling back to transcript text: ${error instanceof Error ? error.message : String(error)}`); - } + const codexSessionId = allowResume ? getCodexSessionId(exchanges, sessionId) : undefined; + if (exchanges.some(exchange => exchange.harness === 'codex')) { + // Filtered text must not recover unscreened source context. + const isolated = !codexSessionId; + const prompt = isolated + ? `${SUMMARIZER_CONTEXT_MARKER}. Summarize only the following admitted transcript in 2-4 factual sentences inside . Do not inspect files, use tools or recover any original session.\n\n${formatConversationText(exchanges)}` + : buildCodexSummaryPrompt(); + const result = await callCodex(prompt, isolated ? undefined : codexSessionId, getCodexModel(exchanges), isolated); + return extractSummary(result); } // Cost safety (#104): everything below this point calls the metered Claude API - // path. If a stray global ANTHROPIC_API_KEY would be billed and the user hasn't + // path. Codex conversations return or fail before reaching this point. + // If a stray global ANTHROPIC_API_KEY would be billed and the user hasn't // acknowledged it, warn once and proceed (don't silently spend without notice). - // Placed AFTER the Codex attempt so pure-Codex users are never warned needlessly. if (wouldBillMeteredApi() && !meteredApiOptIn()) { warnMeteredApiOnce(); } @@ -652,7 +773,7 @@ export async function summarizeConversation(exchanges, sessionId) { // would fail on every one before the no-resume retry kicks in. Treat // missing harness as Claude for backward compatibility with old archives. const isClaudeSession = exchanges.some(e => e.harness === 'claude' || e.harness === undefined); - const claudeSessionId = !codexSessionId && isClaudeSession ? sessionId : undefined; + const claudeSessionId = allowResume && isClaudeSession ? sessionId : undefined; const cwd = claudeSessionId ? exchanges.find(e => e.cwd)?.cwd : undefined; const conversationText = claudeSessionId ? '' // When resuming, no need to include conversation text - it's already in context diff --git a/dist/sync-cli.js b/dist/sync-cli.js index 26172c70..4ef03b53 100644 --- a/dist/sync-cli.js +++ b/dist/sync-cli.js @@ -9,6 +9,7 @@ import { spawn } from 'child_process'; import fs from 'fs'; import { formatLogLine, getSyncLogPath, getSyncLockPath } from './logging.js'; import { acquireFileLock, readLockHolder, releaseFileLock } from './file-lock.js'; +import { FindingsTally, formatFindings, getRedactionSettings, loadRedactor } from './redaction.js'; const args = process.argv.slice(2); // Reentrancy guard (#87): if this sync was triggered by a SessionStart hook // inside a Claude subprocess that the summarizer just spawned, exit silently. @@ -39,10 +40,13 @@ Sync conversations from Claude Code, Codex, and opencode transcript sources to a This command: 1. Exports opencode sessions from its SQLite database when available -2. Copies new or updated .jsonl files to conversation archive +2. Copies new or updated .jsonl files to conversation archive, redacting secrets 3. Generates embeddings for semantic search 4. Updates the search index +Secrets are replaced with [REDACTED:] tokens before anything is archived, +indexed, embedded, or summarized. See EPISODIC_MEMORY_REDACTION* in the README. + Only processes files that are new or have been modified since last sync. Safe to run multiple times - subsequent runs are fast no-ops. @@ -142,8 +146,21 @@ if (isBackground) { console.log(`Sync started in background. Log: ${logPath}`); process.exit(0); } +// Load redaction rules before anything is exported, copied, or indexed. In +// strict mode (the default) a bad rules file stops the sync here: fail closed +// rather than archive unredacted text. +let redactor; +try { + redactor = loadRedactor(); +} +catch (error) { + console.error(`episodic-memory: ${error instanceof Error ? error.message : String(error)}`); + console.error('episodic-memory: refusing to sync without redaction (strict mode). Fix the rules file, ' + + 'or set EPISODIC_MEMORY_REDACTION_STRICT=0 to sync unredacted, or EPISODIC_MEMORY_REDACTION=off.'); + process.exit(1); +} if (!onlyHarnesses || onlyHarnesses.includes('opencode')) { - const opencodeExport = exportOpencodeSessions(); + const opencodeExport = exportOpencodeSessions({ redactor }); if (opencodeExport.exported > 0 || opencodeExport.skipped > 0) { console.log(`opencode export: ${opencodeExport.exported} exported, ${opencodeExport.skipped} skipped`); } @@ -195,15 +212,23 @@ console.log(`Sources: ${sourceDirs.join(', ')}`); console.log(`Destination: ${destDir}\n`); async function syncAll() { const totals = { copied: 0, skipped: 0, indexed: 0, summarized: 0, errors: [], sourcesWithSummaryWork: 0, totalNeedingSummaries: 0 }; + const redactions = new FindingsTally(); for (const sourceDir of sourceDirs) { - const result = await syncConversations(sourceDir, destDir, { ...syncOptions, summaryLimit }); + const result = await syncConversations(sourceDir, destDir, { ...syncOptions, summaryLimit, redactor }); + redactions.add(result.redactions); totals.copied += result.copied; totals.skipped += result.skipped; totals.indexed += result.indexed; totals.summarized += result.summarized; totals.errors.push(...result.errors); } - console.log(`\n✅ Sync complete!`); + // Admission failure is a partial run, not successful synchronization. + // Finish migration before emitting a success claim as well. + if (totals.errors.length === 0) + await runEmbeddingMigrationPhase(); + else + process.exitCode = 1; + console.log(totals.errors.length ? '\nSync finished with errors.' : '\n✅ Sync complete!'); console.log(` Copied: ${totals.copied}`); console.log(` Skipped: ${totals.skipped}`); console.log(` Indexed: ${totals.indexed}`); @@ -213,6 +238,16 @@ async function syncAll() { else { console.log(` Summarized: ${totals.summarized}`); } + // Rule IDs and counts only — never matched values. + if (redactor) { + console.log(` Redaction: ${formatFindings(redactions.toArray())}`); + } + else if (getRedactionSettings().enabled) { + console.log(' Redaction: NOT APPLIED (rules failed to load; EPISODIC_MEMORY_REDACTION_STRICT=0)'); + } + else { + console.log(' Redaction: off (EPISODIC_MEMORY_REDACTION=off)'); + } if (totals.errors.length > 0) { console.log(`\n⚠️ Errors: ${totals.errors.length}`); totals.errors.forEach(err => console.log(` ${err.file}: ${err.error}`)); @@ -223,10 +258,6 @@ async function syncAll() { console.log(` Check your API configuration (EPISODIC_MEMORY_API_BASE_URL / ANTHROPIC_API_KEY).`); } } - // After regular sync, do a batch of embedding migration if any rows are - // still on the old encoder. Lock-protected; if another process is already - // migrating, this is a no-op. - await runEmbeddingMigrationPhase(); } const MIGRATION_BATCH_SIZE = parseInt(process.env.EPISODIC_MEMORY_MIGRATION_BATCH || '500', 10); async function runEmbeddingMigrationPhase() { @@ -246,6 +277,7 @@ async function runEmbeddingMigrationPhase() { } catch (err) { console.error('episodic-memory: migration phase error:', err instanceof Error ? err.message : err); + throw err; } finally { db.close(); diff --git a/dist/sync.d.ts b/dist/sync.d.ts index 658e98ad..a5d46c58 100644 --- a/dist/sync.d.ts +++ b/dist/sync.d.ts @@ -1,13 +1,5 @@ -/** - * Stream and scan for any exclusion marker, carrying an overlap between - * chunks so a marker split across a boundary is still found. A single - * fs.readFileSync(path, 'utf-8') throws ERR_STRING_TOO_LONG above Node's - * ~512 MB max string length; the old catch returned false (fail OPEN), - * silently indexing a file whose DO NOT INDEX marker we never read (#152). - * Streaming confirms cleanliness at any size, and a real read error now - * fails CLOSED (skip) rather than open. - */ -export declare function shouldSkipConversation(filePath: string): boolean; +import { type RedactionFinding, type Redactor } from './redaction.js'; +export { shouldSkipConversation } from './record-admission.js'; export interface SyncResult { copied: number; skipped: number; @@ -17,11 +9,18 @@ export interface SyncResult { file: string; error: string; }>; + /** Rule IDs and counts only — never matched values. */ + redactions: RedactionFinding[]; } export interface SyncOptions { skipIndex?: boolean; skipSummaries?: boolean; summaryLimit?: number; + /** + * Redactor applied at the archive write. `undefined` loads it from the + * environment (loadRedactor); `null` means redaction is off. + */ + redactor?: Redactor | null; } /** * Derive sync options from the process environment. diff --git a/dist/sync.js b/dist/sync.js index 0629761d..2a68ea45 100644 --- a/dist/sync.js +++ b/dist/sync.js @@ -1,56 +1,12 @@ import fs from 'fs'; +import { archiveAdmittedConversation, readRecordExclusions } from './record-admission.js'; import path from 'path'; -import { StringDecoder } from 'string_decoder'; -import { SUMMARIZER_CONTEXT_MARKER } from './constants.js'; import { getExcludedProjects, findJsonlFiles, statIfExists } from './paths.js'; import { formatErrorSentinel, shouldQueueForSummary } from './summary-sentinel.js'; import { getMaxMessageBytes, isOversizeExchange } from './message-size.js'; -const EXCLUSION_MARKERS = [ - 'DO NOT INDEX THIS CHAT', - 'Only use NO_INSIGHTS_FOUND', - SUMMARIZER_CONTEXT_MARKER, -]; -const MARKER_SCAN_CHUNK_BYTES = 1 << 20; // 1 MiB -/** - * Stream and scan for any exclusion marker, carrying an overlap between - * chunks so a marker split across a boundary is still found. A single - * fs.readFileSync(path, 'utf-8') throws ERR_STRING_TOO_LONG above Node's - * ~512 MB max string length; the old catch returned false (fail OPEN), - * silently indexing a file whose DO NOT INDEX marker we never read (#152). - * Streaming confirms cleanliness at any size, and a real read error now - * fails CLOSED (skip) rather than open. - */ -export function shouldSkipConversation(filePath) { - const maxMarkerLen = Math.max(...EXCLUSION_MARKERS.map(m => m.length)); - let fd; - try { - fd = fs.openSync(filePath, 'r'); - const buf = Buffer.allocUnsafe(MARKER_SCAN_CHUNK_BYTES); - const decoder = new StringDecoder('utf8'); - let carry = ''; - let bytesRead; - while ((bytesRead = fs.readSync(fd, buf, 0, buf.length, null)) > 0) { - const window = carry + decoder.write(buf.subarray(0, bytesRead)); - if (EXCLUSION_MARKERS.some(marker => window.includes(marker))) { - return true; - } - carry = window.slice(Math.max(0, window.length - (maxMarkerLen - 1))); - } - const tail = carry + decoder.end(); - return EXCLUSION_MARKERS.some(marker => tail.includes(marker)); - } - catch { - return true; // fail closed (#152) - } - finally { - if (fd !== undefined) { - try { - fs.closeSync(fd); - } - catch { } - } - } -} +import { FindingsTally, loadRedactor } from './redaction.js'; +export { shouldSkipConversation } from './record-admission.js'; +import { shouldSkipConversation } from './record-admission.js'; /** * True when a transcript contains at least one message line in any supported * harness format. Summarizer-spawned Agent SDK sessions materialize as @@ -122,33 +78,6 @@ function hasConversationContent(filePath) { export function buildSyncOptionsFromEnv(env) { return { skipSummaries: env.EPISODIC_MEMORY_SKIP_SUMMARIES === '1' }; } -function copyIfNewer(src, dest) { - // Ensure destination directory exists - const destDir = path.dirname(dest); - if (!fs.existsSync(destDir)) { - fs.mkdirSync(destDir, { recursive: true }); - } - // Check if destination exists and is up-to-date - if (fs.existsSync(dest)) { - const srcStat = fs.statSync(src); - const destStat = fs.statSync(dest); - if (destStat.mtimeMs >= srcStat.mtimeMs) { - return false; // Dest is current, skip - } - } - // Atomic copy: temp file + rename - const tempDest = dest + '.tmp.' + process.pid; - fs.copyFileSync(src, tempDest); - fs.renameSync(tempDest, dest); // Atomic on same filesystem - // Preserve source mtime: harnesses without per-message timestamps (Cursor - // agent transcripts) fall back to file mtime. Round up to the next whole - // millisecond — utimes can't always represent the source's sub-millisecond - // precision, and a dest mtime even fractionally older would defeat the - // skip-if-current check above on every subsequent sync. - const srcStat = fs.statSync(src); - fs.utimesSync(dest, srcStat.atimeMs / 1000, Math.ceil(srcStat.mtimeMs) / 1000); - return true; -} export function extractSessionIdFromPath(filePath) { // Extract session ID from Claude filename or Codex rollout filename. const basename = path.basename(filePath, '.jsonl'); @@ -160,13 +89,20 @@ export function extractSessionIdFromPath(filePath) { return null; } export async function syncConversations(sourceDir, destDir, options = {}) { + const exclusions = readRecordExclusions(); const result = { copied: 0, skipped: 0, indexed: 0, summarized: 0, - errors: [] + errors: [], + redactions: [] }; + // Load the redactor before touching the archive. In strict mode (the + // default) a rules-load failure throws here, so nothing unredacted is + // archived, indexed, or summarized (fail closed). + const redactor = options.redactor === undefined ? loadRedactor() : options.redactor; + const tally = new FindingsTally(); // Ensure source directory exists if (!fs.existsSync(sourceDir)) { return result; @@ -198,7 +134,7 @@ export async function syncConversations(sourceDir, destDir, options = {}) { result.skipped++; continue; } - const wasCopied = copyIfNewer(srcFile, destFile); + const wasCopied = await archiveAdmittedConversation(srcFile, destFile, exclusions, redactor, tally); if (wasCopied) { result.copied++; filesToIndex.push(destFile); @@ -212,10 +148,10 @@ export async function syncConversations(sourceDir, destDir, options = {}) { if (!options.skipSummaries) { const summaryPath = destFile.replace('.jsonl', '-summary.txt'); if (shouldQueueForSummary(summaryPath) && !shouldSkipConversation(destFile)) { - // sessionId enables Claude session-resume summarization; when the - // filename has no UUID to extract (e.g. subagent transcripts named - // agent-.jsonl), queue anyway — summarizeConversation falls - // back to summarizing from the transcript text. + // sessionId enables source-session summarization. When the filename + // has no UUID (e.g. agent-.jsonl), queue anyway: Codex uses an + // isolated transcript-only thread; other harnesses use their own + // transcript-text route without resuming a source session. const sessionId = extractSessionIdFromPath(destFile) ?? undefined; filesToSummarize.push({ path: destFile, sessionId }); } @@ -229,6 +165,7 @@ export async function syncConversations(sourceDir, destDir, options = {}) { } } } + result.redactions = tally.toArray(); // Index copied files (unless skipIndex is set) if (!options.skipIndex && filesToIndex.length > 0) { const { parseConversation } = await import('./parser.js'); @@ -343,7 +280,10 @@ export async function syncConversations(sourceDir, destDir, options = {}) { continue; } console.log(` Summarizing ${path.basename(filePath)} (${exchanges.length} exchanges)...`); - const summary = await summarizeConversation(exchanges, sessionId); + // With redaction on, never resume/fork the session: those paths hand + // the model the unredacted source transcript instead of these + // (redacted) exchanges. + const summary = await summarizeConversation(exchanges, sessionId, { allowResume: redactor === null }); const summaryPath = filePath.replace('.jsonl', '-summary.txt'); fs.writeFileSync(summaryPath, summary, 'utf-8'); result.summarized++; diff --git a/dist/verify.d.ts b/dist/verify.d.ts index 9558c879..4a46897a 100644 --- a/dist/verify.d.ts +++ b/dist/verify.d.ts @@ -3,6 +3,9 @@ export interface VerificationResult { path: string; reason: string; }>; + unindexed: Array<{ + path: string; + }>; orphaned: Array<{ uuid: string; path: string; @@ -12,6 +15,17 @@ export interface VerificationResult { fileTime: number; dbTime: number; }>; + screeningRejected: Array<{ + path: string; + }>; + screeningUnavailable: Array<{ + path: string; + }>; + archiveRefreshes: Array<{ + path: string; + fileTime: number; + dbTime: number; + }>; corrupted: Array<{ path: string; error: string; diff --git a/dist/verify.js b/dist/verify.js index dd4535d2..c27656a1 100644 --- a/dist/verify.js +++ b/dist/verify.js @@ -1,14 +1,22 @@ import fs from 'fs'; import path from 'path'; +import Database from 'better-sqlite3'; import { parseConversation } from './parser.js'; -import { initDatabase, getAllExchanges, getFileLastIndexed } from './db.js'; -import { getArchiveDir, getExcludedProjects, findJsonlFiles, statIfExists } from './paths.js'; +import { getAllExchanges } from './db.js'; +import { getArchiveDir, getDbPath, getExcludedProjects, findJsonlFiles, statIfExists } from './paths.js'; import { isErroredSentinel } from './summary-sentinel.js'; +import { readRecordExclusions } from './record-admission.js'; +import { loadRedactor } from './redaction.js'; export async function verifyIndex() { + readRecordExclusions(); const result = { missing: [], + unindexed: [], orphaned: [], outdated: [], + screeningRejected: [], + screeningUnavailable: [], + archiveRefreshes: [], corrupted: [] }; const archiveDir = getArchiveDir(); @@ -18,8 +26,13 @@ export async function verifyIndex() { if (!fs.existsSync(archiveDir)) { return result; } - // Initialize database once for all checks - const db = initDatabase(); + // Verification must not create or migrate the index it is measuring. + const dbPath = getDbPath(); + const db = fs.existsSync(dbPath) ? new Database(dbPath, { readonly: true, fileMustExist: true }) : null; + const indexState = db?.prepare(` + SELECT MAX(line_end) AS maxLineEnd, MAX(last_indexed) AS lastIndexed + FROM exchanges WHERE archive_path = ? + `); const projects = fs.readdirSync(archiveDir); const excludedProjects = getExcludedProjects(); const excludedDirSet = new Set(excludedProjects); @@ -47,42 +60,51 @@ export async function verifyIndex() { // re-attempts it rather than reporting the conversation as healthy. if (!fs.existsSync(summaryPath)) { result.missing.push({ path: conversationPath, reason: 'No summary file' }); - continue; } - if (isErroredSentinel(fs.readFileSync(summaryPath, 'utf-8'))) { + else if (isErroredSentinel(fs.readFileSync(summaryPath, 'utf-8'))) { result.missing.push({ path: conversationPath, reason: 'Previous summarization failed (error sentinel)' }); - continue; } - // Check if file is outdated (modified after last_indexed) - const lastIndexed = getFileLastIndexed(db, conversationPath); - if (lastIndexed !== null) { - const fileStat = fs.statSync(conversationPath); - if (fileStat.mtimeMs > lastIndexed) { - result.outdated.push({ - path: conversationPath, - fileTime: fileStat.mtimeMs, - dbTime: lastIndexed - }); - } - } - // Try parsing to detect corruption + const state = indexState?.get(conversationPath); + const maxLineEnd = state?.maxLineEnd ?? null; + if (maxLineEnd === null) + result.unindexed.push({ path: conversationPath }); + // Parse independently of summary state and mtime. Only exchanges past + // the indexed high-water mark are a real index delta. try { - await parseConversation(conversationPath, project, conversationPath); + const exchanges = await parseConversation(conversationPath, project, conversationPath); + if (maxLineEnd !== null) { + const fileTime = fs.statSync(conversationPath).mtimeMs; + const dbTime = state?.lastIndexed ?? 0; + if (exchanges.some(exchange => exchange.lineEnd > maxLineEnd)) { + result.outdated.push({ path: conversationPath, fileTime, dbTime }); + } + else if (state?.lastIndexed != null && fileTime > state.lastIndexed) { + result.archiveRefreshes.push({ path: conversationPath, fileTime, dbTime }); + } + } } catch (error) { - result.corrupted.push({ - path: conversationPath, - error: error instanceof Error ? error.message : String(error) - }); + const message = error instanceof Error ? error.message : String(error); + if (message === 'Record content rejected by screening' || + message === 'Record screening byte limit exceeded' || + message === 'Record screening nesting limit exceeded') { + result.screeningRejected.push({ path: conversationPath }); + } + else if (message === 'Record screening unavailable') { + result.screeningUnavailable.push({ path: conversationPath }); + } + else { + result.corrupted.push({ path: conversationPath, error: message }); + } } } } console.log(`Verified ${totalChecked} conversations.`); // Check for orphaned database entries - const dbExchanges = getAllExchanges(db); - db.close(); + const dbExchanges = db ? getAllExchanges(db) : []; + db?.close(); for (const exchange of dbExchanges) { - if (!foundFiles.has(exchange.archivePath)) { + if (!foundFiles.has(exchange.archivePath) && !statIfExists(exchange.archivePath)?.isFile()) { result.orphaned.push({ uuid: exchange.id, path: exchange.archivePath @@ -92,7 +114,14 @@ export async function verifyIndex() { return result; } export async function repairIndex(issues) { + if (issues.screeningRejected.length + issues.screeningUnavailable.length > 0) { + throw new Error('Resolve record screening failures before repair'); + } + readRecordExclusions(); console.log('Repairing index...'); + // Load before touching the index: strict mode fails closed here, as in sync + // and index. + const redactor = loadRedactor(); // To avoid circular dependencies, we import the indexer functions dynamically const { initDatabase, insertExchange, deleteExchange } = await import('./db.js'); const { parseConversation } = await import('./parser.js'); @@ -106,10 +135,10 @@ export async function repairIndex(issues) { deleteExchange(db, orphan.uuid); } // Re-index missing and outdated conversations - const toReindex = [ - ...issues.missing.map(m => m.path), - ...issues.outdated.map(o => o.path) - ]; + const toReindex = [...new Set([ + ...issues.missing.map(m => m.path), + ...issues.outdated.map(o => o.path) + ])]; for (const conversationPath of toReindex) { console.log(`Re-indexing: ${conversationPath}`); try { @@ -125,7 +154,8 @@ export async function repairIndex(issues) { } // Generate/update summary const summaryPath = conversationPath.replace('.jsonl', '-summary.txt'); - const summary = await summarizeConversation(exchanges); + // Under redaction, don't let the Codex fork fallback read the source rollout. + const summary = await summarizeConversation(exchanges, undefined, { allowResume: redactor === null }); fs.writeFileSync(summaryPath, summary, 'utf-8'); console.log(` Created summary: ${summary.split(/\s+/).length} words`); // Index exchanges @@ -138,6 +168,8 @@ export async function repairIndex(issues) { } catch (error) { console.error(`Failed to re-index ${conversationPath}:`, error); + db.close(); + throw error; } } db.close(); diff --git a/docs/CODEX.md b/docs/CODEX.md index c3d83f97..793a3b76 100644 --- a/docs/CODEX.md +++ b/docs/CODEX.md @@ -90,6 +90,6 @@ The test creates an isolated temporary `CODEX_HOME`, copies your existing Codex ## Summaries -Codex summaries use `codex app-server`, `thread/fork`, and `ephemeral: true`. This matters: `codex exec --ephemeral resume ` was tested and still appended to the resumed rollout, so it is not the quality bar for summarization. +Codex summaries with a session ID use `codex app-server`, `thread/fork`, and `ephemeral: true`. This matters: `codex exec --ephemeral resume ` was tested and still appended to the resumed rollout, so it is not the quality bar for summarization. Without a session ID, an isolated ephemeral `thread/start` receives only the admitted transcript text and requires Codex `0.154.0` or newer. -If the Codex app-server summarizer is unavailable or below the support floor, Episodic Memory logs the reason and falls back to transcript-text summarization instead of silently skipping the conversation. +If the Codex app-server summarizer is unavailable or below the applicable support floor, Episodic Memory records a retryable summary failure. It does not send Codex transcripts to Claude. Exchanges can still be indexed and searched without a generated summary. diff --git a/docs/REDACTION.md b/docs/REDACTION.md new file mode 100644 index 00000000..b2d0746f --- /dev/null +++ b/docs/REDACTION.md @@ -0,0 +1,186 @@ +# Secret redaction + +episodic-memory replaces secrets in your conversations with typed tokens +before it archives, indexes, embeds, or summarizes them: + +```text +AccountKey=<88-char key> → AccountKey=[REDACTED:connection-string-secret] +"ClientSecret": "" → "ClientSecret": "[REDACTED:quoted-secret-assignment]" + → +Authorization: Bearer → Authorization: Bearer [REDACTED:jwt] +``` + +Only the value is replaced, so the rest of the conversation stays searchable, +and the token says what kind of value was there. +`episodic-memory search --text "[REDACTED:azure-storage-key]"` finds every +conversation where a storage key was pasted. + +Redaction is on by default and fails closed: if the rules can't load, sync +doesn't run. + +## Where it happens + +Every harness's transcripts (Claude Code, Codex, Cursor, opencode, OMP) are +copied into the conversation archive before indexing and summarization. Record +exclusions and screening apply first, using the source's physical line numbers. +The archive writer then redacts admitted lines +([`archiveAdmittedConversation`](../src/record-admission.ts) → +[`redactJsonlLine`](../src/redaction.ts)). +Everything downstream reads the archive: + +| Where | Redacted? | +| ---------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | +| Conversation archive (`~/.config/superpowers/conversation-archive`) | Yes, line by line as it's copied | +| SQLite index: message text and tool inputs/outputs | Yes, parsed from the archive ([indexer.ts](../src/indexer.ts) included) | +| Embeddings | Yes, built from redacted text | +| Summaries (sent to a model) | Yes. Summarizer resume and Codex fork are turned off, because both read the original transcript ([summarizer.ts](../src/summarizer.ts)) | +| opencode and legacy Cursor staging exports | Yes, when written ([opencode-sync.ts](../src/opencode-sync.ts), [cursor-legacy.ts](../src/cursor-legacy.ts)) | +| `show`, MCP `read` | Yes, they read the archive | +| Logs | Rule IDs and counts only | +| The harness's own files (`~/.claude/projects`, `~/.codex/sessions`, …) | **No.** They belong to the harness; use its retention settings | + +With summarizer resume off, Codex conversations use an isolated, ephemeral Codex +thread containing only the admitted transcript; Claude conversations use the +Claude Agent SDK's transcript-text path. A Codex failure does not fall back to +Claude. Set `EPISODIC_MEMORY_SKIP_SUMMARIES=1` to skip summaries; they are +display-only, so search is unaffected. + +## How a secret is recognized + +The engine is [src/redaction.ts](../src/redaction.ts) and the default rules are +[src/redaction-rules.ts](../src/redaction-rules.ts). Each archive line is parsed +as JSON and passes through three layers: + +1. **Shape.** Values whose format gives them away: private keys, JWTs, + provider-prefixed keys, Entra client secrets (`…Q~…`), 88-character Azure + storage keys, SAS `sig=` values. +2. **Key context.** Values with no recognizable format, caught by the name + they're assigned to: `Password=` in a connection string, `"ClientSecret": "…"`, + ``, `{"name": "DB_PASSWORD", "value": "…"}`, + `password: …`. +3. **Field name.** In parsed JSON, any string whose field name is + secret-looking is replaced whole, including tool inputs and MCP results. + +A **secret-looking name** _ends_ in `secret`, `password`, `passwd`, +`passphrase`, `apikey`, `accesskey`, `accountkey`, `privatekey`, `token`, +`credential(s)` or a similar key word, in any case and with any separators +(`AzureAd:ClientSecret`, `DB_PASSWORD2`). `TokenEndpoint` and `passwordPolicy` +don't count. Placeholders (`${X}`, `$(X)`, `#{X}#`, `{{x}}`, ``, `%X%`, +`"string"`, `"*****"`) are left alone. + +Git SHAs and GUIDs are allowlisted for the shape rules, so commit hashes and +tenant, client and object IDs stay searchable. Key-context rules ignore the +allowlist: a GUID in a `password` slot is a secret. An entropy fallback exists +but is off by default. + +| Rules | Catch | +| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | +| `private-key-block`, `jwt`, `anthropic-api-key`, `openai-api-key`, `github-token`, `aws-access-key-id`, `aws-secret-access-key`, `slack-token`, `google-api-key`, `npm-token`, `azure-client-secret`, `azure-storage-key`, `azure-sas-token` | Shape | +| `connection-string-secret` | `AccountKey=`, `SharedAccessKey=`, `Password=`, `Pwd=` and similar in connection strings, any case | +| `url-credentials`, `bearer-token`, `basic-auth` | `scheme://user:password@host`, `Authorization` headers | +| `azure-keyvault-secret`, `name-value-secret` | Key Vault bundles; `{name, value}` pairs (`az … appsettings list`, Kubernetes `env`, ARM parameters) | +| `xml-appsettings-secret`, `xml-secret-element`, `xml-secret-attribute` | `web.config`, `…`, publish-profile `userPWD="…"` | +| `quoted-secret-assignment` | `"ClientSecret": "…"`, `ClientSecret = "…"`, `apiKey: '…'`; no spaces in the value | +| `secret-assignment` | `password: …`, `CLIENT_SECRET=…` (YAML, dotenv, decrypted SOPS); 8+ characters with a digit, code like `env.X` skipped | +| `secret-field` | Layer 3: a secret-named JSON field, replaced whole | + +`episodic-memory redact --print-default-rules` prints the full definitions. + +## Guarantees + +- **Fails closed.** In strict mode (the default), sync, index, `index repair` + and import stop if the rules can't load. +- **Idempotent.** Tokens never match again. When a match runs into an existing + token, only the text outside it is redacted, so re-running after adding a + rule is safe. +- **Structure-preserving.** One output line per input line, valid JSON stays + valid, and lines with no secrets stay byte-for-byte identical. Index line + ranges and MCP `read` ranges stay correct. Excluded lines remain blank, and + archives are created with private permissions. Source transcripts are unchanged. +- **Values are never printed.** Logs and reports name rules and counts; the + review report adds the value's shape, never the value. + +## Cleaning up existing data + +New syncs redact what they copy. To redact what was archived and indexed +before, or after you add a rule: + +```bash +episodic-memory redact --rewrite --dry-run # what would change; writes nothing +episodic-memory redact --rewrite --dry-run --report # each hit, to check for false positives +episodic-memory redact --rewrite # apply +``` + +`--rewrite` ([src/redact-rewrite.ts](../src/redact-rewrite.ts)) redacts the +archive, the staging exports and the index in place, re-embeds only the rows +that changed, and deletes summaries built from unredacted text so the next +sync regenerates them. It takes the sync lock. A dry run opens the index +read-only. + +`--report` prints one hit per value: + +```text + -work-contoso/4f1c….jsonl:212 quoted-secret-assignment len=40 aA9- H=4.9 + …"AzureAd": { "ClientId": "…", "ClientSecret": "[REDACTED:quoted-secret-assignment]", "TenantId… +``` + +The shape is the length, the character classes (`a` lower, `A` upper, `9` +digits, `-` symbols, `_` spaces) and the entropy in bits per character. A +random key is long with entropy around 4.5 or more; a word or placeholder is +short or low. + +## Configuration + +| Variable | Default | Meaning | +| ---------------------------------- | ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | +| `EPISODIC_MEMORY_REDACTION` | `on` | `off` turns redaction off; record exclusions and screening still apply. Resume is allowed only without a record policy. | +| `EPISODIC_MEMORY_REDACTION_RULES` | `/redaction-rules.json` if present | Custom rules file; a missing file you named is an error | +| `EPISODIC_MEMORY_REDACTION_STRICT` | `1` | `0` continues **unredacted**, with a warning, when the rules fail to load | + +Custom rules extend the defaults: + +```jsonc +{ + "rules": [ + // a rule with a default's id replaces it + { + "id": "contoso-api-key", + "pattern": "\\bctso_[A-Za-z0-9]{32}\\b", + "keywords": ["ctso_"], + }, + ], + "disableRules": ["basic-auth"], + "allowlist": [{ "id": "build-ids", "pattern": "build-[0-9]{8}" }], + "entropy": { "enabled": true }, + "secretFields": { + "keyPattern": "secret|password|token|credentials?|connectionstring", + }, + "includeDefaults": true, +} +``` + +A rule can also set `flags` (`imsu`), `secretGroup` (redact only that capture +group) and `useAllowlist: false`. Every rule is validated on load; an invalid +one is a load failure, which strict mode treats as fatal. Try rules with +`echo 'password: hunter2hunter2' | episodic-memory redact --stdin`. + +## Limits + +This is pattern and context matching, not named-entity recognition: a value +is caught by its format, by the name it's assigned to, or by the field it's +in. That covers how secrets reach transcripts in practice (pasted config, CLI +output, connection strings, tool results), and it is deterministic, cheap +enough for every line of every sync, and each token names the rule that fired. +What it misses: + +- A secret in prose with no name or shape: "the password is hunter2". +- An unquoted letters-only value (`password: hunter`): `secret-assignment` + needs a digit so ordinary English doesn't trigger it. +- Code-like values (`config.password`, `getPassword()`), skipped on purpose. +- Quoted values with spaces, so UI strings like `"Invalid password"` survive. +- Settings with a shapeless value and a name that isn't secret-looking + (`Stripe`, `ConnectionStrings__Default`). Add the name to + `secretFields.keyPattern` or write a rule. + +The entropy fallback narrows some of these gaps, at the cost of false +positives. diff --git a/src/cursor-legacy.ts b/src/cursor-legacy.ts index a62f4b56..598be439 100644 --- a/src/cursor-legacy.ts +++ b/src/cursor-legacy.ts @@ -3,6 +3,7 @@ import os from 'os'; import path from 'path'; import Database from 'better-sqlite3'; import { detectCursorCwd } from './parser.js'; +import { loadRedactor, redactJsonlLine, type Redactor } from './redaction.js'; /** * Import legacy Cursor conversations from Cursor's global SQLite store @@ -141,6 +142,8 @@ export interface CursorLegacyImportOptions { force?: boolean; /** Report what would be exported without writing files. */ dryRun?: boolean; + /** `undefined` loads from the environment (strict mode throws on bad rules); `null` disables. */ + redactor?: Redactor | null; } export interface CursorLegacyImportResult { @@ -160,6 +163,7 @@ export function importCursorLegacy(options: CursorLegacyImportOptions): CursorLe errors: [], }; + const redactor = options.redactor === undefined ? loadRedactor() : options.redactor; const liveIds = options.liveTranscriptIds ?? new Set(); const db = new Database(options.dbPath, { readonly: true, fileMustExist: true }); @@ -264,15 +268,20 @@ export function importCursorLegacy(options: CursorLegacyImportOptions): CursorLe if (!options.dryRun) { // Re-serialize with cwd now that it's known (it's derived from the // whole conversation's tool calls). - const finalLines = cwd + const withCwd = cwd ? lines.map(line => JSON.stringify({ ...JSON.parse(line), cwd })) : lines; + // The export dir is a plugin-owned plaintext copy, so redact it at + // write time like the archive (docs/REDACTION.md). + const finalLines = redactor + ? withCwd.map(line => redactJsonlLine(line, redactor, { source: 'cursor-legacy', path: outFile })) + : withCwd; fs.mkdirSync(path.dirname(outFile), { recursive: true }); fs.writeFileSync(outFile, finalLines.join('\n') + '\n', 'utf-8'); // Stamp the conversation's end time so mtime-based fallbacks and - // copyIfNewer comparisons reflect when it actually happened. + // archive freshness comparisons reflect when it actually happened. const mtime = toIsoTimestamp(composer.lastUpdatedAt) ?? lastTimestamp; const mtimeDate = new Date(mtime); diff --git a/src/db.ts b/src/db.ts index 8c7a73ad..04cb6ab9 100644 --- a/src/db.ts +++ b/src/db.ts @@ -1,4 +1,5 @@ import Database from 'better-sqlite3'; +import { admitExchange } from './record-admission.js'; import { ConversationExchange } from './types.js'; import path from 'path'; import fs from 'fs'; @@ -104,6 +105,17 @@ export function migrateToolCallsCascade(db: Database.Database): void { console.log(' tool_calls migration complete.'); } +/** + * Open the existing index read-only, without creating it, migrating it, or + * changing its journal mode. For dry runs that must not write. Returns null + * when there is no index yet. + */ +export function openDatabaseReadOnly(): Database.Database | null { + const dbPath = getDbPath(); + if (!fs.existsSync(dbPath)) return null; + return new Database(dbPath, { readonly: true, fileMustExist: true }); +} + export function initDatabase(): Database.Database { const dbPath = getDbPath(); @@ -216,6 +228,9 @@ export function insertExchange( embedding: number[], toolNames?: string[] ): void { + const admitted = admitExchange(exchange); + if (!admitted) return; + exchange = admitted; const now = Date.now(); const stmt = db.prepare(` diff --git a/src/index-cli.ts b/src/index-cli.ts index 272ecdba..524d91c1 100644 --- a/src/index-cli.ts +++ b/src/index-cli.ts @@ -10,16 +10,16 @@ import { acquireFileLock, readLockHolder, releaseFileLock } from './file-lock.js const command = process.argv[2]; -// Serialize the whole CLI with `sync-cli`: every command can initialize or -// migrate the database, and rebuild also deletes the database and summaries. +// Serialize the whole CLI with `sync-cli`: writers mutate the index/archive, +// and verification must not claim a consistent result while one is active. const syncLockPath = getSyncLockPath(); const syncLock = acquireFileLock(syncLockPath); if (!syncLock) { const holder = readLockHolder(syncLockPath); const holderLabel = holder !== null ? `pid ${holder}` : 'another process'; - // stderr keeps this out of stdout consumers; status 0 so hooks don't fail. + // A skipped write may be retried by a hook; a skipped verifier is not a result. console.error(`episodic-memory: sync already running (${holderLabel}); skipping`); - process.exit(0); + process.exit(command === 'verify' ? 2 : 0); } const releaseSyncLockOnce = () => { if ((releaseSyncLockOnce as any).done) return; @@ -71,17 +71,30 @@ async function main() { console.log('\n=== Verification Results ==='); console.log(`Missing summaries: ${issues.missing.length}`); + console.log(`Unindexed archives: ${issues.unindexed.length}`); console.log(`Orphaned entries: ${issues.orphaned.length}`); console.log(`Outdated files: ${issues.outdated.length}`); + console.log(`Screening-rejected archives: ${issues.screeningRejected.length}`); + console.log(`Screening-unavailable archives: ${issues.screeningUnavailable.length}`); + console.log(`Archive refreshes (no new exchanges): ${issues.archiveRefreshes.length}`); console.log(`Corrupted files: ${issues.corrupted.length}`); if (issues.missing.length > 0) { console.log('\nMissing summaries:'); issues.missing.forEach(m => console.log(` ${m.path}`)); } + if (issues.unindexed.length > 0) { + console.log('\nUnindexed archives:'); + issues.unindexed.forEach(item => console.log(` ${item.path}`)); + } + if (issues.outdated.length > 0) { + console.log('\nArchives with appended exchanges:'); + issues.outdated.forEach(item => console.log(` ${item.path}`)); + } - if (issues.missing.length + issues.orphaned.length + issues.outdated.length + issues.corrupted.length > 0) { - console.log('\nRun with --repair to fix these issues.'); + if (issues.missing.length + issues.unindexed.length + issues.orphaned.length + issues.outdated.length + + issues.screeningRejected.length + issues.screeningUnavailable.length + issues.corrupted.length > 0) { + console.log('\nReview the reported categories before repair; --repair may regenerate embeddings and AI summaries.'); process.exit(1); } else { console.log('\n✅ Index is healthy!'); @@ -92,9 +105,18 @@ async function main() { console.log('Verifying conversation index...'); const repairIssues = await verifyIndex(); + if (repairIssues.screeningRejected.length + repairIssues.screeningUnavailable.length > 0) { + console.error('Record screening failures require review; repair was not started.'); + process.exitCode = 1; + break; + } + if (repairIssues.unindexed.length > 0) { + console.log(`${repairIssues.unindexed.length} unindexed archive(s) require separate review; --repair does not select them by this status alone.`); + process.exitCode = 1; + } if (repairIssues.missing.length + repairIssues.orphaned.length + repairIssues.outdated.length > 0) { await repairIndex(repairIssues); - } else { + } else if (repairIssues.unindexed.length === 0) { console.log('✅ No issues to repair!'); } break; diff --git a/src/indexer.ts b/src/indexer.ts index 173ef679..db41aff9 100644 --- a/src/indexer.ts +++ b/src/indexer.ts @@ -1,4 +1,5 @@ import fs from 'fs'; +import { archiveAdmittedConversation, readRecordExclusions } from './record-admission.js'; import path from 'path'; import os from 'os'; import { initDatabase, insertExchange } from './db.js'; @@ -9,6 +10,7 @@ import { ConversationExchange } from './types.js'; import { getArchiveDir, getExcludedProjects, getConversationSourceDirs, findJsonlFiles, statIfExists } from './paths.js'; import { formatErrorSentinel, shouldQueueForSummary } from './summary-sentinel.js'; import { getMaxMessageBytes, isOversizeExchange } from './message-size.js'; +import { FindingsTally, formatFindings, loadRedactor, type Redactor } from './redaction.js'; // Set max output tokens for Claude SDK (used by summarizer) process.env.CLAUDE_CODE_MAX_OUTPUT_TOKENS = '20000'; @@ -38,12 +40,24 @@ function sessionIdForSummary(exchanges: ConversationExchange[]): string | undefi return exchanges.find(exchange => exchange.sessionId)?.sessionId; } +// Resume/fork would summarize the unredacted source transcript; see sync.ts. +function summarizeOptions(redactor: Redactor | null) { + return { allowResume: redactor === null }; +} + +function logRedactions(tally: FindingsTally): void { + if (tally.total > 0) console.log(` Redaction: ${formatFindings(tally.toArray())}`); +} + export async function indexConversations( limitToProject?: string, maxConversations?: number, concurrency: number = 1, noSummaries: boolean = false ): Promise { + readRecordExclusions(); + const redactor = loadRedactor(); + const tally = new FindingsTally(); console.log('Initializing database...'); const db = initDatabase(); @@ -113,16 +127,14 @@ export async function indexConversations( // Source transcripts can vanish mid-run (Claude Code cleanup). Skip loudly. let exchanges; try { - // Copy to archive (ensure parent dirs exist for subagent files) - if (!fs.existsSync(archivePath)) { - fs.mkdirSync(path.dirname(archivePath), { recursive: true }); - fs.copyFileSync(sourcePath, archivePath); + if (await archiveAdmittedConversation(sourcePath, archivePath, undefined, redactor, tally)) { console.log(` Archived: ${file}`); } // Parse conversation - exchanges = await parseConversation(sourcePath, project, archivePath); + exchanges = await parseConversation(archivePath, project, archivePath); } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') { db.close(); throw error; } console.log(` Skipped ${file} (read failed: ${error instanceof Error ? error.message : error})`); continue; } @@ -150,7 +162,7 @@ export async function indexConversations( await processBatch(needsSummary, async (conv) => { try { - const summary = await summarizeConversation(conv.exchanges, sessionIdForSummary(conv.exchanges)); + const summary = await summarizeConversation(conv.exchanges, sessionIdForSummary(conv.exchanges), summarizeOptions(redactor)); fs.writeFileSync(conv.summaryPath, summary, 'utf-8'); const wordCount = summary.split(/\s+/).length; console.log(` ✓ ${conv.file}: ${wordCount} words`); @@ -193,6 +205,7 @@ export async function indexConversations( // Check if we hit the limit if (maxConversations && conversationsProcessed >= maxConversations) { console.log(`\nReached limit of ${maxConversations} conversations`); + logRedactions(tally); db.close(); console.log(`✅ Indexing complete! Conversations: ${conversationsProcessed}, Exchanges: ${totalExchanges}`); return; @@ -205,12 +218,16 @@ export async function indexConversations( console.log(` Skipped ${oversizeSkipped} oversize exchange(s) (> ${maxMessageBytes} bytes; set EPISODIC_MEMORY_MAX_MESSAGE_BYTES to change) — likely embedded-transcript payloads (#139)`); } + logRedactions(tally); db.close(); console.log(`\n✅ Indexing complete! Conversations: ${conversationsProcessed}, Exchanges: ${totalExchanges}`); } export async function indexSession(sessionId: string, concurrency: number = 1, noSummaries: boolean = false): Promise { + readRecordExclusions(); console.log(`Indexing session: ${sessionId}`); + const redactor = loadRedactor(); + const tally = new FindingsTally(); // Find the conversation file for this session const sourceDirs = getConversationSourceDirs(); @@ -246,14 +263,12 @@ export async function indexSession(sessionId: string, concurrency: number = 1, n // Archive + parse — source may vanish mid-run (Claude Code cleanup). let exchanges; try { - if (!fs.existsSync(archivePath)) { - fs.mkdirSync(path.dirname(archivePath), { recursive: true }); - fs.copyFileSync(sourcePath, archivePath); - } - exchanges = await parseConversation(sourcePath, project, archivePath); + await archiveAdmittedConversation(sourcePath, archivePath, undefined, redactor, tally); + exchanges = await parseConversation(archivePath, project, archivePath); } catch (error) { - console.log(`Skipped ${file} (read failed: ${error instanceof Error ? error.message : error})`); db.close(); + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error; + console.log(`Skipped ${file} (read failed: ${error instanceof Error ? error.message : error})`); break; } @@ -263,7 +278,7 @@ export async function indexSession(sessionId: string, concurrency: number = 1, n if (!noSummaries && shouldQueueForSummary(summaryPath)) { fs.mkdirSync(path.dirname(summaryPath), { recursive: true }); try { - const summary = await summarizeConversation(exchanges, sessionIdForSummary(exchanges)); + const summary = await summarizeConversation(exchanges, sessionIdForSummary(exchanges), summarizeOptions(redactor)); fs.writeFileSync(summaryPath, summary, 'utf-8'); console.log(`Summary: ${summary.split(/\s+/).length} words`); } catch (error) { @@ -297,6 +312,7 @@ export async function indexSession(sessionId: string, concurrency: number = 1, n console.log(` Skipped ${oversizeSkipped} oversize exchange(s) (> ${maxMessageBytes} bytes; set EPISODIC_MEMORY_MAX_MESSAGE_BYTES to change) — likely embedded-transcript payloads (#139)`); } + logRedactions(tally); console.log(`✅ Indexed session ${sessionId}: ${exchanges.length} exchanges`); } @@ -313,10 +329,14 @@ export async function indexSession(sessionId: string, concurrency: number = 1, n } export async function indexUnprocessed(concurrency: number = 1, noSummaries: boolean = false): Promise { + readRecordExclusions(); console.log('Finding unprocessed conversations...'); if (concurrency > 1) console.log(`Concurrency: ${concurrency}`); if (noSummaries) console.log('⚠️ Running in no-summaries mode (skipping AI summaries)'); + const redactor = loadRedactor(); + const tally = new FindingsTally(); + const db = initDatabase(); await initEmbeddings(); @@ -361,17 +381,11 @@ export async function indexUnprocessed(concurrency: number = 1, noSummaries: boo ).get(archivePath) as { maxLine: number }; const maxIndexedLine = hw.maxLine; - // Ensure parent dirs exist for subagent files try { - fs.mkdirSync(path.dirname(archivePath), { recursive: true }); - - // Refresh the archive when the source may have grown beyond what we've seen. - if (!fs.existsSync(archivePath) || maxIndexedLine > 0) { - fs.copyFileSync(sourcePath, archivePath); - } + await archiveAdmittedConversation(sourcePath, archivePath, undefined, redactor, tally); // Parse and filter to exchanges past the high-water mark - const exchanges = await parseConversation(sourcePath, project, archivePath); + const exchanges = await parseConversation(archivePath, project, archivePath); const newExchanges = maxIndexedLine > 0 ? exchanges.filter(e => e.lineStart > maxIndexedLine) : exchanges; @@ -379,6 +393,7 @@ export async function indexUnprocessed(concurrency: number = 1, noSummaries: boo unprocessed.push({ project, file, sourcePath, archivePath, summaryPath, exchanges: newExchanges }); } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') { db.close(); throw error; } console.log(` Skipped ${file} (read failed: ${error instanceof Error ? error.message : error})`); continue; } @@ -386,6 +401,8 @@ export async function indexUnprocessed(concurrency: number = 1, noSummaries: boo } } // end sourceDir loop + logRedactions(tally); + if (unprocessed.length === 0) { console.log('✅ All conversations are already processed!'); db.close(); @@ -402,7 +419,7 @@ export async function indexUnprocessed(concurrency: number = 1, noSummaries: boo await processBatch(needsSummary, async (conv) => { try { - const summary = await summarizeConversation(conv.exchanges, sessionIdForSummary(conv.exchanges)); + const summary = await summarizeConversation(conv.exchanges, sessionIdForSummary(conv.exchanges), summarizeOptions(redactor)); fs.writeFileSync(conv.summaryPath, summary, 'utf-8'); const wordCount = summary.split(/\s+/).length; console.log(` ✓ ${conv.project}/${conv.file}: ${wordCount} words`); diff --git a/src/mcp-server.ts b/src/mcp-server.ts index c1cbd0e3..1dce206c 100644 --- a/src/mcp-server.ts +++ b/src/mcp-server.ts @@ -23,6 +23,7 @@ import { import { formatConversationAsMarkdown } from './show.js'; import { VERSION } from './version.js'; import fs from 'fs'; +import { readAdmittedConversation } from './record-admission.js'; // Zod Schemas for Input Validation @@ -291,7 +292,7 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => { } // Read and format conversation with optional line range - const jsonlContent = fs.readFileSync(params.path, 'utf-8'); + const jsonlContent = await readAdmittedConversation(params.path, params.startLine, params.endLine); const markdownContent = formatConversationAsMarkdown( jsonlContent, params.startLine, diff --git a/src/opencode-sync.ts b/src/opencode-sync.ts index b61f8ef4..eca6876c 100644 --- a/src/opencode-sync.ts +++ b/src/opencode-sync.ts @@ -2,6 +2,7 @@ import fs from 'fs'; import path from 'path'; import Database from 'better-sqlite3'; import { getOpencodeDbPath, getOpencodeTranscriptDir } from './paths.js'; +import { loadRedactor, redactJsonlLine, type Redactor } from './redaction.js'; export interface OpencodeExportResult { exported: number; @@ -92,7 +93,8 @@ function shouldExportSession(filePath: string, sessionUpdatedMs: number): boolea function writeSessionTranscript( db: Database.Database, session: OpencodeSessionRow, - filePath: string + filePath: string, + redactor: Redactor | null ): void { const messages = db.prepare(` SELECT id, session_id, time_created, time_updated, data @@ -173,9 +175,15 @@ function writeSessionTranscript( })); } + // The staging transcript is a plugin-owned plaintext copy, so redact it at + // write time like the archive (docs/REDACTION.md). + const output = redactor + ? lines.map(line => redactJsonlLine(line, redactor, { source: 'opencode', path: filePath })) + : lines; + fs.mkdirSync(path.dirname(filePath), { recursive: true }); const tempPath = `${filePath}.tmp.${process.pid}`; - fs.writeFileSync(tempPath, `${lines.join('\n')}\n`, 'utf-8'); + fs.writeFileSync(tempPath, `${output.join('\n')}\n`, 'utf-8'); fs.renameSync(tempPath, filePath); const mtime = dateFromMillis(session.time_updated); fs.utimesSync(filePath, mtime, mtime); @@ -184,7 +192,10 @@ function writeSessionTranscript( export function exportOpencodeSessions(options: { dbPath?: string; transcriptDir?: string; + /** `undefined` loads from the environment (strict mode throws on bad rules); `null` disables. */ + redactor?: Redactor | null; } = {}): OpencodeExportResult { + const redactor = options.redactor === undefined ? loadRedactor() : options.redactor; const dbPath = options.dbPath || getOpencodeDbPath(); const transcriptDir = options.transcriptDir || getOpencodeTranscriptDir(); const result: OpencodeExportResult = { @@ -234,7 +245,7 @@ export function exportOpencodeSessions(options: { result.skipped++; continue; } - writeSessionTranscript(db, session, filePath); + writeSessionTranscript(db, session, filePath, redactor); result.exported++; } catch (error) { result.errors.push({ diff --git a/src/parser.ts b/src/parser.ts index a416a113..28ca15c7 100644 --- a/src/parser.ts +++ b/src/parser.ts @@ -1,8 +1,8 @@ import fs from 'fs'; -import readline from 'readline'; import path from 'path'; import { ConversationExchange, ConversationHarness, ToolCall } from './types.js'; import crypto from 'crypto'; +import { admittedConversationLines } from './record-admission.js'; interface JSONLMessage { type: string; @@ -91,11 +91,7 @@ interface ExchangeBuilder { } async function detectConversationHarness(filePath: string): Promise { - const fileStream = fs.createReadStream(filePath); - const rl = readline.createInterface({ - input: fileStream, - crlfDelay: Infinity - }); + const rl = admittedConversationLines(filePath); for await (const line of rl) { if (!line.trim()) continue; @@ -175,11 +171,7 @@ async function parseClaudeConversation( archivePath: string ): Promise { const exchanges: ConversationExchange[] = []; - const fileStream = fs.createReadStream(filePath); - const rl = readline.createInterface({ - input: fileStream, - crlfDelay: Infinity - }); + const rl = admittedConversationLines(filePath); let lineNumber = 0; let currentExchange: ExchangeBuilder | null = null; @@ -479,11 +471,7 @@ async function parseOpencodeConversation( archivePath: string ): Promise { const exchanges: ConversationExchange[] = []; - const fileStream = fs.createReadStream(filePath); - const rl = readline.createInterface({ - input: fileStream, - crlfDelay: Infinity - }); + const rl = admittedConversationLines(filePath); let lineNumber = 0; let sessionId: string | undefined; @@ -687,11 +675,7 @@ async function parseOmpConversation( archivePath: string ): Promise { const exchanges: ConversationExchange[] = []; - const fileStream = fs.createReadStream(filePath); - const rl = readline.createInterface({ - input: fileStream, - crlfDelay: Infinity - }); + const rl = admittedConversationLines(filePath); let sessionId: string | undefined; let cwd: string | undefined; @@ -811,11 +795,7 @@ async function parseCodexConversation( archivePath: string ): Promise { const exchanges: ConversationExchange[] = []; - const fileStream = fs.createReadStream(filePath); - const rl = readline.createInterface({ - input: fileStream, - crlfDelay: Infinity - }); + const rl = admittedConversationLines(filePath); let lineNumber = 0; let sessionId: string | undefined; @@ -955,7 +935,8 @@ async function parseCodexConversation( const timestamp = parsed.timestamp || new Date().toISOString(); if (parsed.type === 'session_meta' && payload) { - sessionId = payload.id || sessionId; + // Forked transcripts may retain ancestor headers after the primary one. + sessionId = sessionId || payload.id; cwd = payload.cwd || cwd; gitBranch = payload.git?.branch || gitBranch; agentVersion = payload.cli_version || agentVersion; @@ -1092,14 +1073,10 @@ async function parseCursorConversation( archivePath: string ): Promise { const exchanges: ConversationExchange[] = []; - const fileStream = fs.createReadStream(filePath); - const rl = readline.createInterface({ - input: fileStream, - crlfDelay: Infinity - }); + const rl = admittedConversationLines(filePath); // Live Cursor transcripts carry no per-message timestamps; fall back to the - // file mtime (preserved from the source by sync's copyIfNewer). Legacy + // file mtime (preserved from the source by the archive writer). Legacy // exports from import-cursor-history embed real per-message timestamps. let fallbackTimestamp: string; try { diff --git a/src/record-admission.ts b/src/record-admission.ts new file mode 100644 index 00000000..6d368acf --- /dev/null +++ b/src/record-admission.ts @@ -0,0 +1,581 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import crypto from 'node:crypto'; +import os from 'node:os'; +import { spawnSync } from 'node:child_process'; +import { SUMMARIZER_CONTEXT_MARKER } from './constants.js'; +import type { ConversationExchange } from './types.js'; +import { getSuperpowersDir } from './paths.js'; +import { redactJsonlLine, type FindingsTally, type Redactor } from './redaction.js'; + +const EXCLUSION_MARKERS = [ + 'DO NOT INDEX THIS CHAT', + 'Only use NO_INSIGHTS_FOUND', + SUMMARIZER_CONTEXT_MARKER, +]; + +const MARKER_SCAN_CHUNK_BYTES = 1 << 20; // 1 MiB + +/** Scan decoded instruction records without retaining the whole transcript. */ +export function shouldSkipConversation(filePath: string): boolean { + let fd: number | undefined; + try { + fd = fs.openSync(filePath, fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW); + const initial = fs.fstatSync(fd); + const excluded = descriptorHasExclusionMarker(fd); + assertUnchangedInput(filePath, initial); + return excluded; + } catch { + return true; + } finally { + if (fd !== undefined) fs.closeSync(fd); + } +} + +function instructionText(record: unknown): string { + if (!object(record)) return ''; + // Codex records both the response item and its user-message event. + if (record.type === 'event_msg' && object(record.payload)) { + return record.payload.type === 'user_message' && typeof record.payload.message === 'string' + ? record.payload.message : ''; + } + const message = record.type === 'response_item' ? record.payload : record.message; + if (!object(message) || (message.role ?? record.role) !== 'user') return ''; + const content = record.type === 'opencode_message' ? record.parts : message.content; + if (typeof content === 'string') return content; + if (!Array.isArray(content)) return ''; + // Claude tool results are wrapped in a user envelope, but are not instructions. + return content.filter(part => object(part) && ['text', 'input_text'].includes(String(part.type)) + && typeof part.text === 'string').map(part => part.text).join('\n'); +} + +function descriptorHasExclusionMarker(fd: number, byteLimit = Infinity, digest?: crypto.Hash): boolean { + if (!fs.fstatSync(fd).isFile()) throw new Error('Conversation input must be a regular file'); + const buffer = Buffer.allocUnsafe(MARKER_SCAN_CHUNK_BYTES); + const decoder = new TextDecoder('utf-8', { fatal: true }); + let pending = Buffer.alloc(0); + let position = 0; + const excluded = (bytes: Buffer) => { + const line = decoder.decode(bytes); + let text: string; + try { text = instructionText(JSON.parse(line)); } + catch { text = line; } // Preserve conservative handling of legacy raw transcripts. + return EXCLUSION_MARKERS.some(marker => text.includes(marker)); + }; + let count: number; + while (position < byteLimit && (count = fs.readSync(fd, buffer, 0, Math.min(buffer.length, byteLimit - position), position)) > 0) { + position += count; + digest?.update(buffer.subarray(0, count)); + pending = Buffer.concat([pending, buffer.subarray(0, count)]); + let newline: number; + while ((newline = pending.indexOf(10)) !== -1) { + if (newline > 64 * 1024 * 1024) throw new Error('Conversation record exceeds admission byte limit'); + if (excluded(pending.subarray(0, newline))) return true; + pending = pending.subarray(newline + 1); + } + if (pending.length > 64 * 1024 * 1024) throw new Error('Conversation record exceeds admission byte limit'); + } + if (position < byteLimit && byteLimit !== Infinity) throw new Error('Conversation changed during record admission'); + return pending.length > 0 && excluded(pending); +} + +function assertUnchangedInput(file: string, initial: fs.Stats): void { + const current = fs.lstatSync(file); + if (!current.isFile() || initial.dev !== current.dev || initial.ino !== current.ino || + initial.size !== current.size || initial.mtimeMs !== current.mtimeMs || initial.ctimeMs !== current.ctimeMs) { + throw new Error('Conversation changed during record admission'); + } +} + +function assertUnchangedSearchPrefix(file: string, initial: fs.Stats, expectedDigest: string): void { + // A live transcript may grow after the search snapshot. Rehash the pinned + // prefix only when metadata changes, so appended bytes cannot hide a rewrite. + const changed = () => new Error('Conversation changed during record admission'); + let current: fs.Stats; + try { current = fs.lstatSync(file); } catch { throw changed(); } + if (!current.isFile() || initial.dev !== current.dev || initial.ino !== current.ino || current.size < initial.size) { + throw changed(); + } + if (current.size === initial.size && current.mtimeMs === initial.mtimeMs && current.ctimeMs === initial.ctimeMs) return; + + let fd: number | undefined; + try { + fd = fs.openSync(file, fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW); + const opened = fs.fstatSync(fd); + if (!opened.isFile() || opened.dev !== initial.dev || opened.ino !== initial.ino || opened.size < initial.size) throw changed(); + const digest = crypto.createHash('sha256'); + const buffer = Buffer.allocUnsafe(MARKER_SCAN_CHUNK_BYTES); + let position = 0; + while (position < initial.size) { + const count = fs.readSync(fd, buffer, 0, Math.min(buffer.length, initial.size - position), position); + if (count <= 0) throw changed(); + digest.update(buffer.subarray(0, count)); + position += count; + } + if (digest.digest('hex') !== expectedDigest) throw changed(); + current = fs.lstatSync(file); + if (!current.isFile() || current.dev !== initial.dev || current.ino !== initial.ino || current.size < initial.size) throw changed(); + } catch { throw changed(); } + finally { if (fd !== undefined) fs.closeSync(fd); } +} + +function assertUnchangedPolicy(expected: string): void { + if (JSON.stringify(readRecordExclusions()) !== expected) { + throw new Error('Record exclusion policy changed during record admission'); + } +} + +function prepareArchiveParent(directory: string): string { + try { + const info = fs.lstatSync(directory); + if (!info.isDirectory() || info.isSymbolicLink()) throw new Error('Archive parent must be a non-symlink directory'); + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error; + const parent = path.dirname(directory); + if (parent === directory) throw error; + prepareArchiveParent(parent); + fs.mkdirSync(directory, { mode: 0o700 }); + } + return fs.realpathSync(directory); +} + +interface ExchangeExclusion { + session_id: string; + transcript: string; + line_start: number; + line_end: number; +} + +interface ToolExclusion { + session_id: string; + transcript: string; + call_id: string; +} + +export interface RecordExclusions { + version: 1; + exchanges: ExchangeExclusion[]; + tool_calls: ToolExclusion[]; + screening?: { + engine: 'gitleaks'; + executable: string; + sha256: string; + timeout_ms: number; + max_record_bytes: number; + }; +} + +const POLICY_ERROR = 'Invalid or unreadable record exclusion policy'; +const UUID = /[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}/ig; + +function object(value: unknown): value is Record { + return value !== null && typeof value === 'object' && !Array.isArray(value); +} + +function keys(value: Record, expected: string[]): boolean { + return Object.keys(value).length === expected.length && expected.every(key => key in value); +} + +function transcriptName(value: unknown): value is string { + return identity(value) && !/[\\/]/.test(value) && value.endsWith('.jsonl') && value !== '.jsonl'; +} + +function identity(value: unknown): value is string { + return typeof value === 'string' && value.length > 0 && value.length <= 512 && !/[\x00-\x1f]/.test(value); +} + +/** Keep scanner inputs local to one record, including JSON-escaped text. */ +function screeningText(value: unknown, depth = 0): string { + if (depth > 24) throw new Error('Record screening nesting limit exceeded'); + if (typeof value === 'string') { + let decoded: unknown; + try { decoded = JSON.parse(value); } catch { return value; } + if (typeof decoded === 'string' || object(decoded) || Array.isArray(decoded)) { + return screeningText(decoded, depth + 1); + } + return value; + } + if (Array.isArray(value)) return value.map(item => screeningText(item, depth + 1)).join('\n'); + if (object(value)) { + // Generic credential rules need an assignment delimiter, not just a nearby + // keyword. Decode nested text without destroying its key/value relation. + return Object.entries(value).map(([key, item]) => `${key} = ${screeningText(item, depth + 1)}`).join('\n'); + } + return value == null ? '' : String(value); +} + +/** One pinned, short-lived local scanner context; no secret-bearing cache. */ +function createRecordScreener(policy: RecordExclusions | null): { scan(value: unknown): void; close(): void } { + const config = policy?.screening; + if (!config) return { scan() {}, close() {} }; + const unavailable = () => new Error('Record screening unavailable'); + let initial: fs.Stats; + let fd: number | undefined; + try { + if (fs.realpathSync(config.executable) !== config.executable) throw unavailable(); + fd = fs.openSync(config.executable, fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW); + initial = fs.fstatSync(fd); + if (!initial.isFile() || initial.size < 1 || initial.size > 134_217_728 || + (initial.mode & 0o022) !== 0 || + (typeof process.getuid === 'function' && initial.uid !== 0 && initial.uid !== process.getuid())) { + throw unavailable(); + } + fs.accessSync(config.executable, fs.constants.X_OK); + const digest = crypto.createHash('sha256').update(fs.readFileSync(fd)).digest('hex'); + if (digest !== config.sha256) throw unavailable(); + } catch { throw unavailable(); } + finally { if (fd !== undefined) fs.closeSync(fd); } + let directory: string; + try { + directory = fs.mkdtempSync(path.join(os.tmpdir(), 'episodic-screen-')); + fs.chmodSync(directory, 0o700); + } catch { throw unavailable(); } + const unchanged = () => { + try { + const current = fs.lstatSync(config.executable); + if (!current.isFile() || current.dev !== initial.dev || current.ino !== initial.ino || + current.size !== initial.size || current.mtimeMs !== initial.mtimeMs || + current.ctimeMs !== initial.ctimeMs) throw unavailable(); + } catch { throw unavailable(); } + }; + return { + scan(value: unknown) { + const raw = typeof value === 'string' ? value : JSON.stringify(value); + if (raw === undefined) return; + if (Buffer.byteLength(raw) > config.max_record_bytes) throw new Error('Record screening byte limit exceeded'); + const input = screeningText(value); + if (Buffer.byteLength(input) > config.max_record_bytes) throw new Error('Record screening byte limit exceeded'); + if (!input.trim()) return; + unchanged(); + const result = spawnSync(config.executable, [ + 'stdin', '--redact', '--no-banner', '--log-level', 'error', + '--ignore-gitleaks-allow', '--exit-code', '10', '--report-format', 'json', '--report-path', '-', + ], { + input, encoding: 'utf8', cwd: directory, shell: false, + env: { HOME: directory, USERPROFILE: directory, PATH: path.dirname(config.executable) }, + timeout: config.timeout_ms, killSignal: 'SIGKILL', maxBuffer: 1_048_576, + windowsHide: true, + }); + unchanged(); + // Exit zero alone is not a clean scan. Validate the full report and never + // forward stdout/stderr, findings, fingerprints or input into errors. + if (result.error || result.signal || ![0, 10].includes(result.status ?? -1)) throw unavailable(); + let report: unknown; + try { report = JSON.parse(result.stdout); } catch { throw unavailable(); } + if (!Array.isArray(report) || (result.status === 0 && report.length !== 0) || + (result.status === 10 && (report.length === 0 || !report.every(item => object(item) && typeof item.RuleID === 'string')))) { + throw unavailable(); + } + if (result.status === 10) throw new Error('Record content rejected by screening'); + }, + close() { fs.rmSync(directory, { recursive: true, force: true }); }, + }; +} + +export function readRecordExclusions(): RecordExclusions | null { + const file = path.join(getSuperpowersDir(), 'record-exclusions.json'); + let fd: number | undefined; + try { + fd = fs.openSync(file, fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW); + const info = fs.fstatSync(fd); + if (!info.isFile() || info.size > 4 * 1024 * 1024) throw new Error(POLICY_ERROR); + const value: unknown = JSON.parse(fs.readFileSync(fd, 'utf8')); + if (!object(value) || !keys(value, 'screening' in value + ? ['version', 'exchanges', 'tool_calls', 'screening'] : ['version', 'exchanges', 'tool_calls']) || value.version !== 1 || + !Array.isArray(value.exchanges) || !Array.isArray(value.tool_calls)) throw new Error(POLICY_ERROR); + if ('screening' in value) { + const s = value.screening; + if (!object(s) || !keys(s, ['engine', 'executable', 'sha256', 'timeout_ms', 'max_record_bytes']) || + s.engine !== 'gitleaks' || typeof s.executable !== 'string' || !path.isAbsolute(s.executable) || + /[\x00-\x1f]/.test(s.executable) || typeof s.sha256 !== 'string' || !/^[a-f0-9]{64}$/.test(s.sha256) || + !Number.isSafeInteger(s.timeout_ms) || Number(s.timeout_ms) < 50 || Number(s.timeout_ms) > 30_000 || + !Number.isSafeInteger(s.max_record_bytes) || Number(s.max_record_bytes) < 1 || Number(s.max_record_bytes) > 8_388_608) { + throw new Error(POLICY_ERROR); + } + } + for (const row of value.exchanges) { + if (!object(row) || !keys(row, ['session_id', 'transcript', 'line_start', 'line_end']) || !identity(row.session_id) || !transcriptName(row.transcript) || + !Number.isSafeInteger(row.line_start) || !Number.isSafeInteger(row.line_end) || + (row.line_start as number) < 1 || (row.line_end as number) < (row.line_start as number)) throw new Error(POLICY_ERROR); + } + for (const row of value.tool_calls) { + if (!object(row) || !keys(row, ['session_id', 'transcript', 'call_id']) || !identity(row.session_id) || !transcriptName(row.transcript) || !identity(row.call_id)) throw new Error(POLICY_ERROR); + } + return value as unknown as RecordExclusions; + } catch (error) { + if (fd === undefined && (error as NodeJS.ErrnoException).code === 'ENOENT') return null; + throw new Error(POLICY_ERROR); + } finally { + if (fd !== undefined) fs.closeSync(fd); + } +} + +export function excludedSession(policy: RecordExclusions | null, sessionId?: string): boolean { + return !!sessionId && !!policy && (policy.exchanges.some(x => x.session_id === sessionId) || + policy.tool_calls.some(x => x.session_id === sessionId)); +} + +function sessionFromPath(file: string): string | undefined { + return path.basename(file).match(UUID)?.at(-1); +} + +function sessionFromRecord(value: any): string | undefined { + if (value?.type === 'session_meta') return value.payload?.id; + if (value?.type === 'session') return value.id; + if (value?.type === 'opencode_session') return value.session?.id; + return value?.sessionId ?? value?.sessionID ?? value?.message?.sessionID; +} + +function blockedCall(policy: RecordExclusions, sessionId: string, transcript: string, callId: unknown): boolean { + return typeof callId === 'string' && policy.tool_calls.some(x => x.session_id === sessionId && x.transcript === transcript && x.call_id === callId); +} + +function filterRecord(line: string, number: number, policy: RecordExclusions | null, transcript: string, sessionId?: string): string { + if (!policy || !sessionId) return line; + if (policy.exchanges.some(x => x.session_id === sessionId && x.transcript === transcript && number >= x.line_start && number <= x.line_end)) return ''; + let value: any; + try { value = JSON.parse(line); } catch { return line; } + if (blockedCall(policy, sessionId, transcript, value?.payload?.call_id) || blockedCall(policy, sessionId, transcript, value?.call_id)) return ''; + // Claude and Cursor may combine safe text and tool payloads in one physical record. + // Preserve the record and safe blocks rather than excluding the entire exchange. + const content = value?.message?.content; + if (Array.isArray(content)) { + const kept = content.filter(block => !blockedCall(policy, sessionId, transcript, block?.id ?? block?.tool_use_id)); + if (kept.length !== content.length) { + return JSON.stringify({ ...value, message: { ...value.message, content: kept } }); + } + } + if (Array.isArray(value?.parts)) { + const kept = value.parts.filter((part: any) => !blockedCall(policy, sessionId, transcript, part?.callID ?? part?.id)); + if (kept.length !== value.parts.length) return JSON.stringify({ ...value, parts: kept }); + } + return line; +} + +/** Keep byte delimiters and physical line coordinates stable in derived copies. */ +async function* admittedRecords(file: string, policy: RecordExclusions | null, startLine = 1, endLine = Infinity): AsyncGenerator<{ line: string; ending: string }> { + const policySnapshot = JSON.stringify(policy); + assertUnchangedPolicy(policySnapshot); + const input = fs.openSync(file, fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW); + const initial = fs.fstatSync(input); + let stream: fs.ReadStream | undefined; + let screener: ReturnType | undefined; + const decoder = new TextDecoder('utf-8', { fatal: true }); + let pending = Buffer.alloc(0); + let sessionId = sessionFromPath(file); + let primaryIdentitySeen = false; + let number = 0; + const admit = (bytes: Buffer, terminated: boolean) => { + number++; + const crlf = terminated && bytes.at(-1) === 13; + const line = decoder.decode(crlf ? bytes.subarray(0, -1) : bytes); + let recordSession: string | undefined; + try { recordSession = sessionFromRecord(JSON.parse(line)); } catch { /* Parser owns malformed records. */ } + if (recordSession && !primaryIdentitySeen) { + if (sessionId && recordSession !== sessionId && (excludedSession(policy, sessionId) || excludedSession(policy, recordSession))) { + throw new Error('Record exclusion session identity mismatch'); + } + sessionId = recordSession; + primaryIdentitySeen = true; + } + const filtered = number >= startLine && number <= endLine + ? filterRecord(line, number, policy, path.basename(file), sessionId) : ''; + screener?.scan(filtered); + return { line: filtered, ending: terminated ? (crlf ? '\r\n' : '\n') : '' }; + }; + try { + // The marker scan and record stream share one descriptor and identity. + // A path replacement cannot substitute a different unscanned transcript. + if (descriptorHasExclusionMarker(input)) return; + assertUnchangedInput(file, initial); + screener = createRecordScreener(policy); + stream = fs.createReadStream(file, { fd: input, autoClose: true, start: 0 }); + for await (const chunk of stream) { + assertUnchangedInput(file, initial); + assertUnchangedPolicy(policySnapshot); + pending = Buffer.concat([pending, chunk]); + let newline: number; + while ((newline = pending.indexOf(10)) !== -1) { + yield admit(pending.subarray(0, newline), true); + pending = pending.subarray(newline + 1); + } + if (pending.length > 64 * 1024 * 1024) throw new Error('Conversation record exceeds admission byte limit'); + } + if (pending.length) yield admit(pending, false); + assertUnchangedInput(file, initial); + assertUnchangedPolicy(policySnapshot); + } finally { + if (stream) stream.destroy(); + else fs.closeSync(input); + screener?.close(); + } +} + +/** Read original inputs without mutation, preserving their physical line numbers. */ +export async function* admittedConversationLines(file: string, policy = readRecordExclusions()): AsyncGenerator { + if (!fs.lstatSync(file).isFile()) throw new Error('Conversation input must be a regular non-symlink file'); + if (shouldSkipConversation(file)) return; + for await (const record of admittedRecords(file, policy)) yield record.line; +} + +/** Read the admitted display input without changing original physical coordinates. */ +export async function readAdmittedConversation(file: string, startLine = 1, endLine = Infinity): Promise { + if (!Number.isSafeInteger(startLine) || startLine < 1 || + (endLine !== Infinity && (!Number.isSafeInteger(endLine) || endLine < startLine))) { + throw new Error('Invalid conversation line range'); + } + const lines: string[] = []; + for await (const record of admittedRecords(file, readRecordExclusions(), startLine, endLine)) lines.push(record.line); + return lines.join('\n'); +} + +/** Enforce the same rule for direct callers that bypass transcript parsing. */ +export function admitExchange(exchange: ConversationExchange, policy = readRecordExclusions()): ConversationExchange | null { + if (!policy) return exchange; + // Historical Codex rows may carry a fork ancestor's ID. Neither that metadata + // nor the filename may override a denial bound to the other known identity. + const sessionIds = [exchange.sessionId, sessionFromPath(exchange.archivePath)].filter((id): id is string => !!id); + const transcript = path.basename(exchange.archivePath); + if (policy.exchanges.some(x => sessionIds.includes(x.session_id) && x.transcript === transcript && exchange.lineStart <= x.line_end && exchange.lineEnd >= x.line_start)) return null; + const kept = exchange.toolCalls?.filter(call => !sessionIds.some(id => blockedCall(policy, id, transcript, call.id))); + const admitted = kept && kept.length !== exchange.toolCalls?.length ? { ...exchange, toolCalls: kept } : exchange; + const screener = createRecordScreener(policy); + try { + // These are separate original records, not one keyword-prefilter fragment. + screener.scan(admitted.userMessage); + screener.scan(admitted.assistantMessage); + for (const call of admitted.toolCalls ?? []) screener.scan(call); + const { userMessage, assistantMessage, toolCalls, ...metadata } = admitted; + screener.scan(metadata); + return admitted; + } finally { screener.close(); } +} + +/** Publish only admitted bytes. Never write an unfiltered temporary copy. */ +export async function archiveAdmittedConversation( + source: string, + destination: string, + policy = readRecordExclusions(), + redactor: Redactor | null = null, + tally?: FindingsTally, +): Promise { + const policySnapshot = JSON.stringify(policy); + assertUnchangedPolicy(policySnapshot); + if (shouldSkipConversation(source)) return false; + if (path.resolve(source) === path.resolve(destination)) throw new Error('Source and derived archive must be distinct'); + const original = fs.lstatSync(source); + if (!original.isFile()) throw new Error('Conversation source must be a regular non-symlink file'); + let target: fs.Stats | undefined; + if (fs.existsSync(destination)) { + target = fs.lstatSync(destination); + if (!target.isFile() || target.dev === original.dev && target.ino === original.ino) { + throw new Error('Source and derived archive must be distinct regular files'); + } + } + if (!policy && !redactor && target && target.mtimeMs >= original.mtimeMs && target.size === original.size) return false; + const initial = fs.statSync(source); + const parent = prepareArchiveParent(path.dirname(path.resolve(destination))); + const parentIdentity = fs.statSync(parent); + destination = path.join(parent, path.basename(destination)); + const temp = destination + '.tmp.' + process.pid + '.' + crypto.randomUUID(); + const fd = fs.openSync(temp, 'wx', 0o600); + const hash = crypto.createHash('sha256'); + try { + for await (const record of admittedRecords(source, policy)) { + // Exclusions use source coordinates; only admitted text reaches redaction. + const line = redactor + ? redactJsonlLine(record.line, redactor, { source: 'archive', path: source }, tally) + : record.line; + const bytes = Buffer.from(line + record.ending); + hash.update(bytes); + fs.writeFileSync(fd, bytes); + } + fs.fsyncSync(fd); + } catch (error) { + fs.closeSync(fd); + fs.unlinkSync(temp); + throw error; + } + fs.closeSync(fd); + try { + const final = fs.statSync(source); + if (initial.dev !== final.dev || initial.ino !== final.ino || initial.size !== final.size || initial.mtimeMs !== final.mtimeMs) { + throw new Error('Conversation changed during record admission'); + } + if (fs.existsSync(destination)) { + const existing = crypto.createHash('sha256'); + for await (const chunk of fs.createReadStream(destination)) existing.update(chunk); + if (existing.digest('hex') === hash.digest('hex')) return false; + } + assertUnchangedInput(source, original); + assertUnchangedPolicy(policySnapshot); + const currentParent = fs.lstatSync(parent); + if (!currentParent.isDirectory() || currentParent.dev !== parentIdentity.dev || currentParent.ino !== parentIdentity.ino) { + throw new Error('Archive parent changed during record admission'); + } + fs.renameSync(temp, destination); + fs.utimesSync(destination, initial.atimeMs / 1000, Math.ceil(initial.mtimeMs) / 1000); + return true; + } finally { if (fs.existsSync(temp)) fs.unlinkSync(temp); } +} + +/** Request-local admission: retained search rows are not trusted merely because they were indexed. */ +export function createSearchAdmission() { + const policy = readRecordExclusions(); + const policySnapshot = JSON.stringify(policy); + const files = new Map(); + const checkFile = (file: string) => { + assertUnchangedPolicy(policySnapshot); + const cached = files.get(file); + if (cached) return !cached.excluded; + let fd: number | undefined; + try { + let pathInfo: fs.Stats; + try { pathInfo = fs.lstatSync(file); } + catch (error) { + // A transcript removed since indexing has nothing left to verify or expose: + // withhold its retained rows instead of failing the whole search. + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error; + files.set(file, { excluded: true }); + return false; + } + if (!pathInfo.isFile()) throw new Error('Conversation changed during record admission'); + fd = fs.openSync(file, fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW); + const initial = fs.fstatSync(fd); + if (!initial.isFile() || initial.dev !== pathInfo.dev || initial.ino !== pathInfo.ino || + !Number.isSafeInteger(initial.size)) throw new Error('Conversation changed during record admission'); + const hash = crypto.createHash('sha256'); + // The observed size is this request's snapshot boundary. Records appended + // later are admitted by the next request, including conversation opt-outs. + const excluded = descriptorHasExclusionMarker(fd, initial.size, hash); + files.set(file, { initial, excluded, digest: excluded ? undefined : hash.digest('hex') }); + return !excluded; + } catch { throw new Error('Conversation changed during record admission'); } + finally { if (fd !== undefined) fs.closeSync(fd); } + }; + return { + admit(exchange: ConversationExchange): ConversationExchange | null { + if (!checkFile(exchange.archivePath)) return null; + const admitted = admitExchange(exchange, policy); + assertUnchangedPolicy(policySnapshot); + return admitted; + }, + summary(exchange: ConversationExchange, text: string): string | undefined { + // An old summary has no record-level provenance. A filtered source may have + // contributed any of its text, so do not expose that derived summary. + const ids = [exchange.sessionId, sessionFromPath(exchange.archivePath)]; + const transcript = path.basename(exchange.archivePath); + if (policy && [...policy.exchanges, ...policy.tool_calls].some(row => + row.transcript === transcript && ids.includes(row.session_id))) return undefined; + return admitExchange({ ...exchange, userMessage: text, assistantMessage: '', toolCalls: [] }, policy) + ? text : undefined; + }, + verify() { + assertUnchangedPolicy(policySnapshot); + for (const [file, { initial, digest }] of files) { + if (initial !== undefined && digest !== undefined) assertUnchangedSearchPrefix(file, initial, digest); + } + assertUnchangedPolicy(policySnapshot); + }, + }; +} diff --git a/src/redact-cli.ts b/src/redact-cli.ts new file mode 100644 index 00000000..01c33ea4 --- /dev/null +++ b/src/redact-cli.ts @@ -0,0 +1,151 @@ +import fs from 'fs'; +import { getArchiveDir, getCursorLegacyExportDir, getOpencodeTranscriptDir } from './paths.js'; +import { acquireFileLock, readLockHolder, releaseFileLock } from './file-lock.js'; +import { getSyncLockPath } from './logging.js'; +import { + DEFAULT_REDACTION_CONFIG, + FindingsTally, + formatFindings, + getRedactionSettings, + loadRedactor, + redactJsonlLine, + type Redactor, +} from './redaction.js'; + +const args = process.argv.slice(2); + +const HELP = ` +Usage: episodic-memory redact [--rewrite [--dry-run [--report]]] [--stdin] [--print-default-rules] + +Secret redaction for the conversation archive and index. + +OPTIONS: + --rewrite Re-run redaction over the existing archive, staging exports, + and search index (in place). Rows that change are re-embedded, + and summaries built from unredacted text are deleted (the next + sync regenerates them). Run once after upgrading, and again + after adding rules. + --dry-run With --rewrite: report what would change, write nothing. + The index is opened read-only and is not migrated. + --report With --rewrite --dry-run: list every value that would be + redacted: where it is, the rule, its shape (length, + character classes, entropy) and the redacted text around + it. Use it to spot false positives before applying. + --stdin Redact stdin to stdout, one JSONL/text line at a time, and + print rule counts to stderr. Handy for testing rules. + --print-default-rules Print the bundled rules as JSON (a starting point for + redaction-rules.json). + --help, -h Show this help + +ENVIRONMENT: + EPISODIC_MEMORY_REDACTION on (default) | off + EPISODIC_MEMORY_REDACTION_RULES custom rules file (default: /redaction-rules.json) + EPISODIC_MEMORY_REDACTION_STRICT 1 (default) fails closed on a bad rules file | 0 passes through + +Output names rule IDs, counts and value shapes only; matched values are never printed. +`; + +function fail(message: string): never { + console.error(`episodic-memory: ${message}`); + process.exit(1); +} + +function requireRedactor(): Redactor { + if (!getRedactionSettings().enabled) { + fail('redaction is off (EPISODIC_MEMORY_REDACTION=off); unset it to use this command.'); + } + let redactor: Redactor | null; + try { + // Always strict here: a rewrite with no rules would be a silent no-op. + redactor = loadRedactor({ ...process.env, EPISODIC_MEMORY_REDACTION_STRICT: '1' }); + } catch (error) { + fail(error instanceof Error ? error.message : String(error)); + } + return redactor!; +} + +async function runRewrite(dryRun: boolean, report: boolean): Promise { + if (report && !dryRun) fail('--report needs --dry-run: review the hits, then apply without it.'); + const redactor = requireRedactor(); + + // Share sync's single-instance lock so a background sync can't write + // between our reads and renames. + const lockPath = getSyncLockPath(); + const lock = acquireFileLock(lockPath); + if (!lock) { + const holder = readLockHolder(lockPath); + fail(`a sync or index run is in progress (${holder !== null ? `pid ${holder}` : 'another process'}); try again when it finishes.`); + } + const release = () => releaseFileLock(lock); + process.on('exit', release); + + const { rewriteArchive } = await import('./redact-rewrite.js'); + let embeddingsReady = false; + const embed = async (user: string, assistant: string, toolNames?: string[]) => { + const embeddings = await import('./embeddings.js'); + if (!embeddingsReady) { + await embeddings.initEmbeddings(); + embeddingsReady = true; + } + return embeddings.generateExchangeEmbedding(user, assistant, toolNames); + }; + + const archiveDir = getArchiveDir(); + const stagingDirs = [getOpencodeTranscriptDir(), getCursorLegacyExportDir()].filter(d => fs.existsSync(d)); + console.log(`Redacting${dryRun ? ' (dry run)' : ''}: ${archiveDir}`); + for (const dir of stagingDirs) console.log(` + staging: ${dir}`); + + const result = await rewriteArchive({ + archiveDir, + stagingDirs, + redactor, + embed, + dryRun, + report: report + ? hit => console.log(` ${hit.location} ${hit.ruleId} ${hit.shape} + ${hit.context}`) + : undefined, + log: message => console.log(` ${message}`), + }); + + console.log(`\n${dryRun ? 'Would redact' : 'Redacted'} across archive, staging, and index: ${formatFindings(result.findings)}`); + if (dryRun && (result.filesRewritten || result.rowsUpdated || result.summariesRemoved || result.stagingFilesRewritten)) { + console.log('Run without --dry-run to apply.'); + } +} + +async function runStdin(): Promise { + const redactor = requireRedactor(); + const chunks: Buffer[] = []; + for await (const chunk of process.stdin) chunks.push(chunk as Buffer); + const input = Buffer.concat(chunks).toString('utf-8'); + const tally = new FindingsTally(); + const output = input.split('\n').map(line => redactJsonlLine(line, redactor, { source: 'stdin', path: '-' }, tally)); + process.stdout.write(output.join('\n')); + console.error(formatFindings(tally.toArray())); +} + +async function main(): Promise { + if (args.length === 0 || args.includes('--help') || args.includes('-h')) { + console.log(HELP); + return; + } + if (args.includes('--print-default-rules')) { + console.log(JSON.stringify(DEFAULT_REDACTION_CONFIG, null, 2)); + return; + } + if (args.includes('--stdin')) { + await runStdin(); + return; + } + if (args.includes('--rewrite')) { + await runRewrite(args.includes('--dry-run'), args.includes('--report')); + return; + } + fail(`unknown option(s): ${args.join(' ')}. Try: episodic-memory redact --help`); +} + +main().catch(error => { + console.error('Error:', error instanceof Error ? error.message : error); + process.exit(1); +}); diff --git a/src/redact-rewrite.ts b/src/redact-rewrite.ts new file mode 100644 index 00000000..3218bd2b --- /dev/null +++ b/src/redact-rewrite.ts @@ -0,0 +1,349 @@ +import fs from 'fs'; +import path from 'path'; +import { initDatabase, openDatabaseReadOnly } from './db.js'; +import { recordReembedded } from './embedding-migration.js'; +import { + copyFileRedacted, + describeShape, + findRedactionTokens, + FindingsTally, + redactJsonlLine, + type RedactionContext, + type RedactionFinding, + type Redactor, +} from './redaction.js'; + +/** + * Backfill for `episodic-memory redact --rewrite`: re-run redaction over data + * written before redaction existed (or before a rule was added). + * + * 1. Archive: every .jsonl is redacted in place. Line count and mtime are + * preserved, so index line ranges stay valid and sync still sees the + * archive as current. + * 2. Staging dirs (opencode / legacy Cursor exports): same, in place. + * 3. Index: every exchanges/tool_calls row is redacted in place. Changed rows + * are re-embedded from the redacted text, so no vector is left that was + * derived from a secret. + * 4. Summaries: a `-summary.txt` for a conversation that had findings (in the + * archive or the index), or one that matches a rule itself, is deleted. + * It was generated from unredacted text, and the next sync regenerates it + * from the redacted archive. + * + * Idempotent: a second run finds nothing to change. A dry run opens the index + * read-only, so it doesn't create, migrate or otherwise touch it. + */ + +export type EmbedFn = (user: string, assistant: string, toolNames?: string[]) => Promise; + +export interface RewriteOptions { + archiveDir: string; + redactor: Redactor; + embed: EmbedFn; + /** Count what would change without writing anything. */ + dryRun?: boolean; + /** + * Called once per value that would be redacted, for reviewing hits before + * applying them. Never receives the value itself. + */ + report?: (hit: RedactionHit) => void; + /** Plugin-owned staging dirs to redact in place (not indexed). */ + stagingDirs?: string[]; + log?: (message: string) => void; +} + +export interface RewriteResult { + filesScanned: number; + filesRewritten: number; + stagingFilesRewritten: number; + rowsUpdated: number; + summariesRemoved: number; + /** Rule IDs and counts only — never matched values. */ + findings: RedactionFinding[]; +} + +/** One redacted value, described without revealing it. */ +export interface RedactionHit { + /** Archive-relative file and 1-based line, or `index:# `. */ + location: string; + ruleId: string; + /** Length, character classes and entropy (see describeShape). */ + shape: string; + /** Redacted text around the token, on one line. */ + context: string; +} + +const SUMMARY_SUFFIX = '-summary.txt'; +const CONTEXT_BEFORE = 60; +const CONTEXT_AFTER = 20; +const PAGE_SIZE = 500; + +function walk(dir: string): string[] { + const out: string[] = []; + let entries: fs.Dirent[]; + try { + entries = fs.readdirSync(dir, { withFileTypes: true }); + } catch { + return out; + } + for (const entry of entries) { + const full = path.join(dir, entry.name); + if (entry.isDirectory()) out.push(...walk(full)); + else if (entry.isFile()) out.push(full); + } + return out; +} + +function summaryPathFor(jsonlPath: string): string { + return jsonlPath.replace(/\.jsonl$/, SUMMARY_SUFFIX); +} + +type Match = { ruleId: string; value: string; absorbedBy?: number }; + +// In report mode each hit is written as its own numbered token, e.g. +// `[REDACTED:jwt--hit3]`, so it can be found in the output exactly; tokens that +// were already in the text have no number. Numbers are stripped for display. +const HIT_TOKEN = /\[REDACTED:([a-z0-9][a-z0-9-]*?)--hit(\d+)\]/g; + +/** `value` with any numbered tokens in it replaced by the values they stand for. */ +function restoreHits(value: string, matches: Match[]): string { + return value.replace(HIT_TOKEN, (_token, _ruleId, n: string) => matches[Number(n)].value); +} + +/** + * Report each hit with the redacted text around its token. A hit whose token + * was swallowed by a later one (a secret field redacted whole after a text rule + * hit part of it) is reported at the token that swallowed it. + */ +function reportHits( + redacted: string, + matches: Match[], + location: string, + report: (hit: RedactionHit) => void +): string { + let display = ''; + let pos = 0; + const spans = new Map(); + for (const t of findRedactionTokens(redacted)) { + const numbered = /^(.*)--hit(\d+)$/.exec(t.ruleId); + display += redacted.slice(pos, t.start); + const shown = numbered ? `[REDACTED:${numbered[1]}]` : redacted.slice(t.start, t.end); + if (numbered && !spans.has(Number(numbered[2]))) { + spans.set(Number(numbered[2]), [display.length, display.length + shown.length]); + } + display += shown; + pos = t.end; + } + display += redacted.slice(pos); + + const spanOf = (n: number): [number, number] | undefined => { + for (let i: number | undefined = n; i !== undefined; i = matches[i].absorbedBy) { + const span = spans.get(i); + if (span) return span; + } + return undefined; + }; + const oneLine = (text: string) => text.replace(/\s+/g, ' '); + const hits = matches.map((m, n) => ({ m, span: spanOf(n) })); + hits.sort((a, b) => (a.span?.[0] ?? Infinity) - (b.span?.[0] ?? Infinity)); + for (const { m, span } of hits) { + const [start, end] = span ?? [0, 0]; + const before = display.slice(Math.max(0, start - CONTEXT_BEFORE), start); + report({ + location, + ruleId: m.ruleId, + shape: describeShape(m.value), + context: span + ? oneLine(`${start > CONTEXT_BEFORE ? '…' : ''}${before}${display.slice(start, end + CONTEXT_AFTER)}`) + : '', + }); + } + return display; +} + +/** + * Redact with `run`, reporting each hit when `report` is set. Returns + * the redacted text with ordinary tokens either way. + */ +function redactReporting( + ctx: RedactionContext, + location: string, + report: ((hit: RedactionHit) => void) | undefined, + run: (ctx: RedactionContext) => string +): string { + if (!report) return run(ctx); + const matches: Match[] = []; + const out = run({ + ...ctx, + onMatch: (ruleId, value) => { + const n = matches.length; + for (const [, , k] of value.matchAll(HIT_TOKEN)) matches[Number(k)].absorbedBy = n; + matches.push({ ruleId, value: restoreHits(value, matches) }); + return `[REDACTED:${ruleId}--hit${n}]`; + }, + }); + return matches.length > 0 ? reportHits(out, matches, location, report) : out; +} + +/** Report every hit in a JSONL file, line by line. Writes nothing. */ +function reportFile(file: string, label: string, redactor: Redactor, report: (hit: RedactionHit) => void): void { + const lines = fs.readFileSync(file, 'utf-8').split('\n'); + const ctx = { source: 'rewrite', path: file }; + lines.forEach((line, i) => { + redactReporting(ctx, `${label}:${i + 1}`, report, c => redactJsonlLine(line, redactor, c)); + }); +} + +/** Redact one JSONL file in place. Returns the number of values redacted. */ +function rewriteFileInPlace(file: string, redactor: Redactor, dryRun: boolean, tally: FindingsTally): number { + const fileTally = new FindingsTally(); + const temp = `${file}.redact.${process.pid}`; + try { + copyFileRedacted(file, temp, redactor, { source: 'rewrite', path: file }, fileTally); + if (fileTally.total > 0 && !dryRun) { + const stat = fs.statSync(file); + fs.renameSync(temp, file); + // Same rounding as archiveAdmittedConversation: never leave the archive older than its source. + fs.utimesSync(file, stat.atimeMs / 1000, Math.ceil(stat.mtimeMs) / 1000); + } + } finally { + try { fs.unlinkSync(temp); } catch {} + } + tally.add(fileTally.toArray()); + return fileTally.total; +} + +export async function rewriteArchive(options: RewriteOptions): Promise { + const { archiveDir, redactor, embed } = options; + const dryRun = options.dryRun === true; + // Reporting reads files before redaction; after a real rewrite there'd be nothing to find. + if (options.report && !dryRun) throw new Error('rewriteArchive: report requires dryRun'); + const log = options.log ?? (() => {}); + const tally = new FindingsTally(); + const result: RewriteResult = { + filesScanned: 0, + filesRewritten: 0, + stagingFilesRewritten: 0, + rowsUpdated: 0, + summariesRemoved: 0, + findings: [], + }; + // Conversations whose summary was built from unredacted text. + const staleSummaries = new Set(); + + // 1. Archive files. + const archiveFiles = walk(archiveDir); + for (const file of archiveFiles.filter(f => f.endsWith('.jsonl'))) { + result.filesScanned++; + if (rewriteFileInPlace(file, redactor, dryRun, tally) > 0) { + result.filesRewritten++; + staleSummaries.add(summaryPathFor(file)); + if (options.report) reportFile(file, path.relative(archiveDir, file), redactor, options.report); + } + } + log(`Archive: ${result.filesRewritten} of ${result.filesScanned} file(s) ${dryRun ? 'would be ' : ''}rewritten`); + + // 2. Staging dirs. + for (const dir of options.stagingDirs ?? []) { + for (const file of walk(dir).filter(f => f.endsWith('.jsonl'))) { + if (rewriteFileInPlace(file, redactor, dryRun, tally) > 0) { + result.stagingFilesRewritten++; + if (options.report) reportFile(file, `staging:${path.relative(dir, file)}`, redactor, options.report); + } + } + } + if (options.stagingDirs?.length) { + log(`Staging exports: ${result.stagingFilesRewritten} file(s) ${dryRun ? 'would be ' : ''}rewritten`); + } + + // 3. Index rows. Page by rowid so writes between pages don't disturb the scan. + // A dry run reads the index as it is; opening it normally would migrate it. + const db = dryRun ? openDatabaseReadOnly() : initDatabase(); + if (db) try { + const page = db.prepare( + 'SELECT rowid AS rid, id, user_message, assistant_message, archive_path FROM exchanges WHERE rowid > ? ORDER BY rowid LIMIT ?' + ); + const toolsFor = db.prepare('SELECT id, tool_name, tool_input, tool_result FROM tool_calls WHERE exchange_id = ? ORDER BY rowid'); + const updateExchange = db.prepare('UPDATE exchanges SET user_message = ?, assistant_message = ? WHERE id = ?'); + const updateTool = db.prepare('UPDATE tool_calls SET tool_input = ?, tool_result = ? WHERE id = ?'); + + let lastRowid = 0; + for (;;) { + const rows = page.all(lastRowid, PAGE_SIZE) as Array<{ + rid: number; id: string; user_message: string; assistant_message: string; archive_path: string; + }>; + if (rows.length === 0) break; + lastRowid = rows[rows.length - 1].rid; + + for (const row of rows) { + const rowTally = new FindingsTally(); + const ctx = { source: 'index', path: row.archive_path }; + const where = `index:${path.relative(archiveDir, row.archive_path)}#${row.id}`; + const redactText = (text: string, field: string) => + redactReporting(ctx, `${where} ${field}`, options.report, c => { + const r = redactor.redact(text, c); + rowTally.add(r.findings); + return r.text; + }); + const user = { text: redactText(row.user_message, 'user') }; + const assistant = { text: redactText(row.assistant_message, 'assistant') }; + + const tools = toolsFor.all(row.id) as Array<{ id: string; tool_name: string; tool_input: string | null; tool_result: string | null }>; + const toolUpdates: Array<{ id: string; input: string | null; result: string | null }> = []; + for (const tool of tools) { + // tool_input is JSON text; redactJsonlLine keeps it valid JSON. + const toolInput = tool.tool_input; + const input = toolInput === null ? null : redactReporting( + ctx, `${where} ${tool.tool_name} input`, options.report, + c => redactJsonlLine(toolInput, redactor, c, rowTally) + ); + const output = tool.tool_result === null ? null : redactText(tool.tool_result, `${tool.tool_name} result`); + if (input !== tool.tool_input || output !== tool.tool_result) { + toolUpdates.push({ id: tool.id, input, result: output }); + } + } + + if (rowTally.total === 0) continue; + tally.add(rowTally.toArray()); + result.rowsUpdated++; + staleSummaries.add(summaryPathFor(row.archive_path)); + if (dryRun) continue; + + const toolNames = tools.length > 0 ? tools.map(t => t.tool_name) : undefined; + const embedding = await embed(user.text, assistant.text, toolNames); + db.transaction(() => { + updateExchange.run(user.text, assistant.text, row.id); + for (const t of toolUpdates) updateTool.run(t.input, t.result, t.id); + recordReembedded(db, row.id, embedding); + })(); + } + } + } finally { + db.close(); + } + log(`Index: ${result.rowsUpdated} exchange(s) ${dryRun ? 'would be ' : ''}redacted and re-embedded`); + + // 4. Summaries: stale ones, plus any summary that itself matches a rule. + for (const file of archiveFiles.filter(f => f.endsWith(SUMMARY_SUFFIX))) { + if (staleSummaries.has(file)) continue; + let text: string; + try { + text = fs.readFileSync(file, 'utf-8'); + } catch { + continue; + } + const r = redactor.redact(text, { source: 'summary', path: file }); + if (r.findings.length > 0) { + tally.add(r.findings); + staleSummaries.add(file); + } + } + for (const summary of staleSummaries) { + if (!fs.existsSync(summary)) continue; + result.summariesRemoved++; + if (!dryRun) fs.unlinkSync(summary); + } + log(`Summaries: ${result.summariesRemoved} ${dryRun ? 'would be ' : ''}removed (regenerated from redacted text on the next sync)`); + + result.findings = tally.toArray(); + return result; +} diff --git a/src/redaction-rules.ts b/src/redaction-rules.ts new file mode 100644 index 00000000..7f5490b0 --- /dev/null +++ b/src/redaction-rules.ts @@ -0,0 +1,272 @@ +import type { RedactionConfig } from './redaction.js'; + +/** + * Bundled default redaction rules. + * + * Patterns are ported from gitleaks' rule set (https://github.com/gitleaks/gitleaks, + * MIT), trimmed to the credentials that realistically show up in Claude Code / + * Codex transcripts for an Azure-heavy stack, and rewritten for JavaScript + * regex (lookbehind instead of gitleaks' consuming boundary groups, so + * adjacent matches aren't swallowed). + * + * Order matters: rules run top to bottom, and a later rule never re-matches + * inside an earlier rule's `[REDACTED:...]` token. Specific shapes go first so + * the token names the most precise rule; the generic `secret-assignment` + * keyword rule goes last. + * + * `secretGroup` replaces only that capture group, so surrounding context + * (connection-string server names, SAS URL paths, usernames) stays searchable. + * + * `keywords` is a case-insensitive prefilter: a rule only runs on text that + * contains at least one keyword. It keeps the per-string cost low on large + * transcripts and bounds the generic rules' work. + * + * Users extend or override these with `redaction-rules.json`; see + * docs/REDACTION.md. `episodic-memory redact --print-default-rules` dumps + * this object as JSON. + */ +// Names that mark the value next to them as a secret, for the key-context +// rules below. A name must END in one of these (optionally plus digits), so +// tokenType, secretName, passwordPolicy, TokenEndpoint and maxTokens don't +// count. `pwd` is deliberately absent (the PWD env var); `userPWD` is handled +// by xml-secret-attribute. +const SECRET_NAME = + String.raw`(?:secret|password|passwd|passphrase|api[_-]?key|access[_-]?key|account[_-]?key|private[_-]?key|` + + String.raw`shared[_-]?(?:access[_-]?)?key|primary[_-]?key|secondary[_-]?key|master[_-]?key|signing[_-]?key|` + + String.raw`subscription[_-]?key|client[_-]?key|encryption[_-]?key|token|credentials?)\d*`; + +// Value is not a template/placeholder: ${X} $(X) $X #{X}# {{x}} %X% __X__, +// an existing token, a type name (Swagger "string"), or a mask (*****). +const NOT_PLACEHOLDER = + String.raw`(?!\[REDACTED:|\$\{|\$\(|\$[A-Za-z_]\w*["'<\s]|#\{|\{\{|<[^>]*>|%[A-Za-z_]\w*%|__[A-Za-z0-9_]+__|` + + String.raw`(?:string|null|true|false|none|undefined|\*+)["'<])`; + +// A JSON string value (escapes allowed), captured. +const JSON_STRING_VALUE = String.raw`"${NOT_PLACEHOLDER}([^"\\]+(?:\\.[^"\\]*)*)"`; +// Sibling members inside the same object: strings, or anything but braces/quotes. +const SAME_OBJECT = String.raw`(?:[^{}"]|"(?:[^"\\]|\\.)*"){0,400}?`; +const KEY_VAULT_ID = + String.raw`"id"[ \t]*:[ \t]*"https://[^"\s]+\.vault\.(?:azure\.net|azure\.cn|usgovcloudapi\.net|microsoftazure\.de)/secrets/[^"\s]*"`; + +export const DEFAULT_REDACTION_CONFIG: RedactionConfig = { + rules: [ + { + id: 'private-key-block', + description: 'PEM/OpenSSH private key block; a truncated block (no END line) is redacted to the end of the text', + pattern: String.raw`-----BEGIN[A-Z0-9 _-]{0,40}PRIVATE KEY(?: BLOCK)?-----[\s\S]*?(?:-----END[A-Z0-9 _-]{0,40}PRIVATE KEY(?: BLOCK)?-----|$)`, + keywords: ['private key'], + }, + { + id: 'connection-string-secret', + description: 'Secret value inside an Azure/ADO.NET connection string; account, server and database names are kept. Keywords are case-insensitive, as in ADO.NET; a path after Pwd= is the shell PWD variable and is kept; the value needs a letter or digit, so Markdown like `Password=` is kept', + pattern: String.raw`\b(?:(?:AccountKey|SharedAccessKey|SharedAccessSignature|SharedSecret|ClientSecret|Password)=|Pwd=(?![/~]))(?![\[$<{%])(?=[^;"'\s]*[A-Za-z0-9])([^;"'\s]+)`, + flags: 'i', + secretGroup: 1, + keywords: ['accountkey', 'sharedaccess', 'sharedsecret', 'clientsecret', 'password', 'pwd'], + }, + { + id: 'azure-sas-token', + description: 'Signature of an Azure SAS token; the resource URL and other SAS parameters are kept', + pattern: String.raw`\bsig=([A-Za-z0-9%+/=_-]{16,})`, + secretGroup: 1, + keywords: ['sig='], + }, + { + id: 'jwt', + description: 'JSON Web Token (Entra ID / Azure access tokens, id tokens)', + pattern: String.raw`\beyJ[A-Za-z0-9_-]{8,}\.eyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}`, + keywords: ['eyj'], + }, + { + id: 'anthropic-api-key', + pattern: String.raw`\bsk-ant-[a-z]{2,10}\d{2}-[A-Za-z0-9_-]{32,}`, + keywords: ['sk-ant-'], + }, + { + id: 'openai-api-key', + pattern: String.raw`\bsk-(?:[a-z]+-)?[A-Za-z0-9_-]{16,}T3BlbkFJ[A-Za-z0-9_-]{16,}`, + keywords: ['t3blbkfj'], + }, + { + id: 'github-token', + description: 'GitHub classic, OAuth, app and fine-grained tokens', + pattern: String.raw`\b(?:gh[pousr]_[A-Za-z0-9]{36,255}|github_pat_[A-Za-z0-9_]{22,255})\b`, + keywords: ['ghp_', 'gho_', 'ghu_', 'ghs_', 'ghr_', 'github_pat_'], + }, + { + id: 'aws-access-key-id', + pattern: String.raw`\b(?:AKIA|ASIA|ABIA|ACCA)[A-Z2-7]{16}\b`, + keywords: ['akia', 'asia', 'abia', 'acca'], + }, + { + id: 'aws-secret-access-key', + pattern: String.raw`aws[_-]?secret[_-]?access[_-]?key["']?[ \t]*[:=][ \t]*["']?([A-Za-z0-9/+]{40})(?![A-Za-z0-9/+])`, + flags: 'i', + secretGroup: 1, + keywords: ['secret'], + }, + { + id: 'slack-token', + pattern: String.raw`\bxox[abposr]-[A-Za-z0-9-]{10,}`, + keywords: ['xox'], + }, + { + id: 'google-api-key', + pattern: String.raw`\bAIza[0-9A-Za-z_-]{35}(?![0-9A-Za-z_-])`, + keywords: ['aiza'], + }, + { + id: 'npm-token', + pattern: String.raw`\bnpm_[A-Za-z0-9]{36}\b`, + keywords: ['npm_'], + }, + { + id: 'azure-client-secret', + description: 'Entra ID (Azure AD) application client secret: 3 chars, a digit, "Q~", 31-34 chars. A trailing "." may follow, so a secret ending a sentence still matches', + pattern: String.raw`(?]+:(?![\[$<{%])([^\s@/"'<>]+)@`, + secretGroup: 1, + keywords: ['://'], + }, + { + id: 'bearer-token', + pattern: String.raw`\bBearer\s+([A-Za-z0-9\-._~+/]{20,}=*)`, + flags: 'i', + secretGroup: 1, + keywords: ['bearer'], + }, + { + id: 'basic-auth', + pattern: String.raw`\bAuthorization[ \t]*:[ \t]*Basic\s+([A-Za-z0-9+/]{8,}={0,2})`, + flags: 'i', + secretGroup: 1, + keywords: ['basic'], + }, + { + id: 'azure-keyvault-secret', + description: 'The "value" of a Key Vault secret bundle (az keyvault secret show/set, SDK JSON), whatever the secret is named', + pattern: + KEY_VAULT_ID + String.raw`(?:[^{}]|\{[^{}]*\}){0,800}?"value"[ \t]*:[ \t]*` + JSON_STRING_VALUE + + String.raw`|"value"[ \t]*:[ \t]*` + JSON_STRING_VALUE + String.raw`(?=(?:[^{}]|\{[^{}]*\}){0,800}?` + KEY_VAULT_ID + ')', + secretGroup: [1, 2], + useAllowlist: false, + keywords: ['.vault.'], + }, + { + id: 'name-value-secret', + description: + 'The "value" of a {"name"/"key": , "value": ...} object, in either order: ' + + 'az webapp/functionapp config appsettings list, Kubernetes env, ARM/Bicep parameters', + pattern: + String.raw`"(?:name|key)"[ \t]*:[ \t]*"[^"\\]{0,80}?` + SECRET_NAME + '"' + SAME_OBJECT + + String.raw`"value"[ \t]*:[ \t]*` + JSON_STRING_VALUE + + String.raw`|"value"[ \t]*:[ \t]*` + JSON_STRING_VALUE + '(?=' + SAME_OBJECT + + String.raw`"(?:name|key)"[ \t]*:[ \t]*"[^"\\]{0,80}?` + SECRET_NAME + '")', + flags: 'i', + secretGroup: [1, 2], + useAllowlist: false, + keywords: ['"value"'], + }, + { + id: 'xml-appsettings-secret', + description: 'web.config / app.config , either attribute order', + pattern: + String.raw`]*?\bkey[ \t]*=[ \t]*"[^"]{0,80}?` + SECRET_NAME + String.raw`"[^>]*?\bvalue[ \t]*=[ \t]*"` + + NOT_PLACEHOLDER + String.raw`([^"]+)"` + + String.raw`|]*?\bvalue[ \t]*=[ \t]*"` + NOT_PLACEHOLDER + String.raw`([^"]+)"(?=[^>]*?\bkey[ \t]*=[ \t]*"[^"]{0,80}?` + + SECRET_NAME + '")', + flags: 'i', + secretGroup: [1, 2], + useAllowlist: false, + keywords: [']*)?>` + NOT_PLACEHOLDER + String.raw`([^<]{1,4096})`, + flags: 'i', + secretGroup: 2, + useAllowlist: false, + keywords: [', %X%, {{x}}) and code (calls, dotted member access)', + pattern: + String.raw`\b[A-Za-z0-9_.-]{0,40}?(?:secret|password|passwd|passphrase|api[_-]?key|access[_-]?key|private[_-]?key|token)` + + String.raw`["']?[ \t]*[:=][ \t]*["']?(?![\[$<{%])(?=[^\s"',;&\\]*\d)` + + // Not a dotted identifier path (env.AUTH0_SECRET, this.config.token2): code, not a value. + String.raw`(?![A-Za-z_$][\w$]*(?:\.[A-Za-z_$][\w$]*)+(?:$|[\s"',;&\\)}\]]))` + + String.raw`([^\s"',;&\\()<>{}\[\]]{8,})(?=$|[\s"',;&\\)}\]])`, + flags: 'i', + secretGroup: 1, + keywords: ['secret', 'passw', 'passphrase', 'key', 'token'], + }, + ], + allowlist: [ + { + id: 'git-sha', + description: 'Full or short git commit SHA', + pattern: String.raw`\b[0-9a-f]{7,40}\b`, + }, + { + id: 'guid', + description: 'GUID/UUID (Entra tenant, client and object IDs, subscription IDs)', + pattern: String.raw`\b[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\b`, + }, + ], + entropy: { + enabled: false, + minLength: 32, + threshold: 4.5, + requireKeyword: true, + keywords: ['secret', 'key', 'token', 'password', 'passwd', 'credential', 'signature'], + window: 40, + }, + secretFields: { + enabled: true, + // Normalized key (lowercase letters/digits only, trailing digits dropped) + // must end with one of these. Harness keys (signature, apiKeySource, + // input_tokens, token_count) don't. + keyPattern: + 'secret|password|passwd|userpwd|passphrase|apikey|accesskey|accountkey|privatekey|sharedkey|primarykey|' + + 'secondarykey|masterkey|signingkey|subscriptionkey|clientkey|encryptionkey|token|credentials?', + }, +}; diff --git a/src/redaction.ts b/src/redaction.ts new file mode 100644 index 00000000..999f0136 --- /dev/null +++ b/src/redaction.ts @@ -0,0 +1,849 @@ +import fs from 'fs'; +import path from 'path'; +import { StringDecoder } from 'string_decoder'; +import { getSuperpowersDir } from './paths.js'; +import { DEFAULT_REDACTION_CONFIG } from './redaction-rules.js'; + +/** + * Inline secret redaction. + * + * Secrets are replaced with typed tokens (`[REDACTED:]`) at the archive + * write, the one point every harness's transcripts pass through (see + * docs/REDACTION.md). Everything downstream (SQLite text, + * tool_calls, embeddings, summaries, show/read) reads the archive, so it only + * ever sees redacted text. + * + * Invariants: + * - Findings carry rule IDs and counts only. A matched value is never logged, + * returned, or thrown. + * - Idempotent: a `[REDACTED:...]` token never matches a rule, so redacting + * redacted text is a no-op. + * - JSONL structure is preserved: one output line per input line, valid JSON in + * gives valid JSON out, and unchanged lines are byte-identical. + */ + +export { DEFAULT_REDACTION_CONFIG }; + +/** Fork default. Flip to false for an opt-in upstream build. */ +export const REDACTION_ENABLED_BY_DEFAULT = true; + +export const REDACTION_RULES_FILENAME = 'redaction-rules.json'; + +const RULE_ID_PATTERN = /^[a-z0-9][a-z0-9-]*$/; +const TOKEN_PATTERN = /\[REDACTED:[a-z0-9][a-z0-9-]*\]/g; +const TOKEN_PREFIX = '[REDACTED:'; +const ALLOWED_FLAGS = /^[imsu]*$/; +const ENTROPY_RULE_ID = 'high-entropy'; +const FIELD_RULE_ID = 'secret-field'; +const RESERVED_IDS = new Set([ENTROPY_RULE_ID, FIELD_RULE_ID]); + +/** Values that are templates or type names, not secrets. Never redacted by field context. */ +const PLACEHOLDER_VALUE = + /^(?:\$\{.*\}|\$\(.*\)|\$[A-Za-z_]\w*|#\{.*\}#?|\{\{.*\}\}|<[^>]*>|%[A-Za-z_]\w*%|__[A-Za-z0-9_]+__|string|null|true|false|none|undefined|\*+)$/i; +const KEY_VAULT_SECRET_ID = /^https:\/\/[^/\s]+\.vault\.(?:azure\.net|azure\.cn|usgovcloudapi\.net|microsoftazure\.de)\/secrets\//i; + +export interface RedactionRuleSpec { + id: string; + pattern: string; + /** Extra RegExp flags from [imsu]; `g` and `d` are always added. */ + flags?: string; + /** Case-insensitive prefilter: skip the rule unless the text contains one. */ + keywords?: string[]; + /** + * Replace only this capture group instead of the whole match. An array means + * "the first of these groups that participated" (for either-order patterns). + */ + secretGroup?: number | number[]; + /** Apply the allowlist to this rule's matches (default true). Key-context rules turn it off: a GUID in a password slot is a secret. */ + useAllowlist?: boolean; + description?: string; +} + +export interface AllowlistSpec { + id: string; + /** Matched against the whole candidate secret; a full match keeps it. */ + pattern: string; + flags?: string; + description?: string; +} + +export interface EntropySpec { + enabled: boolean; + minLength: number; + /** Shannon entropy in bits per character. */ + threshold: number; + /** Only fire when a keyword appears shortly before the candidate, on the same line. */ + requireKeyword: boolean; + keywords: string[]; + /** How many characters before the candidate to search for a keyword. */ + window: number; +} + +/** + * Field-name context for parsed JSON (transcript lines, structured tool/MCP + * results, tool inputs). A string value is redacted whole when its own key, or + * the `name`/`key` of a `{name, value}` pair, matches `keyPattern`. + */ +export interface SecretFieldsSpec { + enabled: boolean; + /** + * Matched (anchored at the end) against the key lowercased with everything + * but letters and digits removed and trailing digits dropped, so + * `AzureAd:ClientSecret`, `client_secret` and `DB_PASSWORD2` all normalize to + * something ending in a keyword. + */ + keyPattern: string; +} + +export interface RedactionConfig { + rules: RedactionRuleSpec[]; + allowlist: AllowlistSpec[]; + entropy: EntropySpec; + secretFields: SecretFieldsSpec; +} + +/** Shape of a user `redaction-rules.json`. Every field is optional. */ +export interface RedactionRulesFile { + /** Merge with the bundled defaults (default true). */ + includeDefaults?: boolean; + /** Added after the defaults; a rule with a default's id replaces it in place. */ + rules?: RedactionRuleSpec[]; + /** Default rule ids to turn off. */ + disableRules?: string[]; + /** Added to the default allowlist (same id replaces). */ + allowlist?: AllowlistSpec[]; + entropy?: Partial; + secretFields?: Partial; +} + +export interface RedactionContext { + source: string; + path: string; + /** + * Called with each value as it is redacted. For local review tooling + * (`redact --report`) only: the value is the secret itself, so never log it. + * May return the token to write instead of `[REDACTED:]`, e.g. one + * numbered per hit; it must still have the token's shape. + */ + onMatch?: (ruleId: string, value: string) => string | void; +} + +export interface RedactionFinding { + ruleId: string; + count: number; +} + +export interface RedactionResult { + text: string; + findings: RedactionFinding[]; +} + +export interface Redactor { + redact(text: string, ctx?: RedactionContext): RedactionResult; + readonly ruleIds: string[]; + /** True when a JSON key / setting name marks its value as a secret (secretFields). */ + isSecretField(name: string): boolean; +} + +export interface RedactionSettings { + enabled: boolean; + strict: boolean; + /** Explicit EPISODIC_MEMORY_REDACTION_RULES path, if set. */ + rulesPath?: string; +} + +/** Rules could not be loaded or are invalid. Messages describe config only. */ +export class RedactionConfigError extends Error { + constructor(message: string) { + super(message); + this.name = 'RedactionConfigError'; + } +} + +// --------------------------------------------------------------------------- +// Settings and loading +// --------------------------------------------------------------------------- + +const OFF_VALUES = new Set(['off', '0', 'false', 'no', 'disabled']); +const ON_VALUES = new Set(['on', '1', 'true', 'yes', 'enabled']); + +/** Unknown values fall back to the default, which is the safe side for both switches. */ +function parseToggle(raw: string | undefined, fallback: boolean): boolean { + const value = raw?.trim().toLowerCase(); + if (!value) return fallback; + if (OFF_VALUES.has(value)) return false; + if (ON_VALUES.has(value)) return true; + return fallback; +} + +/** + * EPISODIC_MEMORY_REDACTION on (fork default) | off + * EPISODIC_MEMORY_REDACTION_RULES path to a custom rules file + * EPISODIC_MEMORY_REDACTION_STRICT 1 (default) | 0; strict fails closed on a rules-load error + */ +export function getRedactionSettings(env: NodeJS.ProcessEnv = process.env): RedactionSettings { + return { + enabled: parseToggle(env.EPISODIC_MEMORY_REDACTION, REDACTION_ENABLED_BY_DEFAULT), + strict: parseToggle(env.EPISODIC_MEMORY_REDACTION_STRICT, true), + rulesPath: env.EPISODIC_MEMORY_REDACTION_RULES || undefined, + }; +} + +function readRulesFile(settings: RedactionSettings): RedactionRulesFile { + let rulesPath = settings.rulesPath; + if (!rulesPath) { + const candidate = path.join(getSuperpowersDir(), REDACTION_RULES_FILENAME); + if (!fs.existsSync(candidate)) return {}; + rulesPath = candidate; + } + let raw: string; + try { + raw = fs.readFileSync(rulesPath, 'utf-8'); + } catch (error) { + throw new RedactionConfigError( + `Redaction rules failed to load from ${rulesPath}: ${error instanceof Error ? error.message : String(error)}` + ); + } + try { + return JSON.parse(raw) as RedactionRulesFile; + } catch (error) { + throw new RedactionConfigError( + `Redaction rules file ${rulesPath} is not valid JSON: ${error instanceof Error ? error.message : String(error)}` + ); + } +} + +function replaceById(base: T[], additions: T[]): T[] { + const out = [...base]; + for (const item of additions) { + const at = out.findIndex(existing => existing.id === item.id); + if (at >= 0) out[at] = item; + else out.push(item); + } + return out; +} + +/** + * Merge a rules file with the bundled defaults and validate the result. + * Throws RedactionConfigError on any invalid rule. + */ +export function loadRedactionConfig(file: RedactionRulesFile): RedactionConfig { + if (!file || typeof file !== 'object' || Array.isArray(file)) { + throw new RedactionConfigError('Redaction rules file must contain a JSON object'); + } + for (const key of ['rules', 'allowlist', 'disableRules'] as const) { + if (file[key] !== undefined && !Array.isArray(file[key])) { + throw new RedactionConfigError(`Redaction rules file: "${key}" must be an array`); + } + } + for (const key of ['entropy', 'secretFields'] as const) { + if (file[key] !== undefined && (typeof file[key] !== 'object' || file[key] === null || Array.isArray(file[key]))) { + throw new RedactionConfigError(`Redaction rules file: "${key}" must be an object`); + } + } + + const includeDefaults = file.includeDefaults !== false; + const disabled = new Set(file.disableRules ?? []); + const baseRules = includeDefaults ? DEFAULT_REDACTION_CONFIG.rules : []; + const baseAllowlist = includeDefaults ? DEFAULT_REDACTION_CONFIG.allowlist : []; + + const config: RedactionConfig = { + rules: replaceById(baseRules, file.rules ?? []).filter(rule => !disabled.has(rule.id)), + allowlist: replaceById(baseAllowlist, file.allowlist ?? []), + entropy: { ...DEFAULT_REDACTION_CONFIG.entropy, ...(file.entropy ?? {}) }, + secretFields: { ...DEFAULT_REDACTION_CONFIG.secretFields, ...(file.secretFields ?? {}) }, + }; + compileConfig(config); // validate + return config; +} + +/** + * Load the active redactor from the environment. + * + * Returns null when redaction is off. When the rules can't be loaded: strict + * mode (the default) throws RedactionConfigError so callers fail closed; + * non-strict mode warns and returns null, so text passes through unredacted. + */ +export function loadRedactor( + env: NodeJS.ProcessEnv = process.env, + warn: (message: string) => void = message => console.error(message) +): Redactor | null { + const settings = getRedactionSettings(env); + if (!settings.enabled) return null; + try { + return createRedactor(loadRedactionConfig(readRulesFile(settings))); + } catch (error) { + const err = error instanceof RedactionConfigError + ? error + : new RedactionConfigError(`Redaction rules failed to load: ${error instanceof Error ? error.message : String(error)}`); + if (settings.strict) throw err; + warn( + `episodic-memory: ${err.message}. EPISODIC_MEMORY_REDACTION_STRICT=0, so conversations ` + + 'will be archived and indexed WITHOUT redaction this run.' + ); + return null; + } +} + +// --------------------------------------------------------------------------- +// Compilation +// --------------------------------------------------------------------------- + +interface CompiledRule { + id: string; + regex: RegExp; + keywords?: string[]; + secretGroups: number[]; + useAllowlist: boolean; +} + +interface CompiledConfig { + rules: CompiledRule[]; + allowlist: RegExp[]; + entropy: EntropySpec; + /** null when secretFields is disabled. */ + secretField: RegExp | null; +} + +function checkFlags(flags: string | undefined, what: string): string { + const value = flags ?? ''; + if (typeof value !== 'string' || !ALLOWED_FLAGS.test(value)) { + throw new RedactionConfigError(`${what}: flags must be a combination of "imsu"`); + } + return value; +} + +function compileRule(spec: RedactionRuleSpec): CompiledRule { + if (!spec || typeof spec !== 'object') { + throw new RedactionConfigError('Redaction rule must be an object'); + } + if (typeof spec.id !== 'string' || !RULE_ID_PATTERN.test(spec.id)) { + throw new RedactionConfigError( + `Redaction rule id ${JSON.stringify(spec.id)} is invalid: use lowercase letters, digits and dashes` + ); + } + if (RESERVED_IDS.has(spec.id)) { + throw new RedactionConfigError(`Redaction rule id "${spec.id}" is reserved`); + } + const what = `Redaction rule "${spec.id}"`; + if (typeof spec.pattern !== 'string' || spec.pattern.length === 0) { + throw new RedactionConfigError(`${what}: pattern must be a non-empty string`); + } + const flags = checkFlags(spec.flags, what); + + let regex: RegExp; + let groupCount: number; + try { + regex = new RegExp(spec.pattern, flags + 'gd'); + groupCount = new RegExp(`(?:${spec.pattern})|`, flags).exec('')!.length - 1; + } catch (error) { + throw new RedactionConfigError(`${what}: invalid regular expression (${error instanceof Error ? error.message : String(error)})`); + } + if (new RegExp(spec.pattern, flags).test('')) { + throw new RedactionConfigError(`${what}: pattern matches the empty string`); + } + + const secretGroups = Array.isArray(spec.secretGroup) ? spec.secretGroup : [spec.secretGroup ?? 0]; + if (secretGroups.length === 0) { + throw new RedactionConfigError(`${what}: secretGroup must not be an empty array`); + } + for (const group of secretGroups) { + if (!Number.isInteger(group) || group < 0 || group > groupCount) { + throw new RedactionConfigError(`${what}: secretGroup ${group} does not exist in the pattern (${groupCount} group(s))`); + } + } + if (spec.useAllowlist !== undefined && typeof spec.useAllowlist !== 'boolean') { + throw new RedactionConfigError(`${what}: useAllowlist must be a boolean`); + } + + let keywords: string[] | undefined; + if (spec.keywords !== undefined) { + if (!Array.isArray(spec.keywords) || spec.keywords.some(k => typeof k !== 'string' || k.length === 0)) { + throw new RedactionConfigError(`${what}: keywords must be an array of non-empty strings`); + } + keywords = spec.keywords.length > 0 ? spec.keywords.map(k => k.toLowerCase()) : undefined; + } + + return { id: spec.id, regex, keywords, secretGroups, useAllowlist: spec.useAllowlist !== false }; +} + +function compileAllowlist(spec: AllowlistSpec): RegExp { + const what = `Redaction allowlist entry ${JSON.stringify(spec?.id)}`; + if (!spec || typeof spec.pattern !== 'string' || spec.pattern.length === 0) { + throw new RedactionConfigError(`${what}: pattern must be a non-empty string`); + } + const flags = checkFlags(spec.flags, what); + try { + return new RegExp(`^(?:${spec.pattern})$`, flags); + } catch (error) { + throw new RedactionConfigError(`${what}: invalid regular expression (${error instanceof Error ? error.message : String(error)})`); + } +} + +function compileConfig(config: RedactionConfig): CompiledConfig { + const seen = new Set(); + const rules = config.rules.map(spec => { + const rule = compileRule(spec); + if (seen.has(rule.id)) throw new RedactionConfigError(`Duplicate redaction rule id "${rule.id}"`); + seen.add(rule.id); + return rule; + }); + const allowlist = config.allowlist.map(compileAllowlist); + + const e = config.entropy; + if ( + typeof e.enabled !== 'boolean' || typeof e.requireKeyword !== 'boolean' || + !Number.isInteger(e.minLength) || e.minLength < 8 || + typeof e.threshold !== 'number' || !(e.threshold > 0) || + !Number.isInteger(e.window) || e.window < 0 || + !Array.isArray(e.keywords) || e.keywords.some(k => typeof k !== 'string' || !k) + ) { + throw new RedactionConfigError( + 'Redaction entropy settings are invalid (need enabled, requireKeyword: boolean; minLength >= 8; threshold > 0; window >= 0; keywords: string[])' + ); + } + const f = config.secretFields; + if (!f || typeof f.enabled !== 'boolean' || typeof f.keyPattern !== 'string' || f.keyPattern.length === 0) { + throw new RedactionConfigError('Redaction secretFields settings are invalid (need enabled: boolean; keyPattern: non-empty string)'); + } + let secretField: RegExp | null = null; + if (f.enabled) { + try { + secretField = new RegExp(`(?:${f.keyPattern})$`); + } catch (error) { + throw new RedactionConfigError(`Redaction secretFields.keyPattern: invalid regular expression (${error instanceof Error ? error.message : String(error)})`); + } + if (secretField.test('')) throw new RedactionConfigError('Redaction secretFields.keyPattern matches the empty string'); + } + + return { rules, allowlist, entropy: { ...e, keywords: e.keywords.map(k => k.toLowerCase()) }, secretField }; +} + +// --------------------------------------------------------------------------- +// Redaction engine +// --------------------------------------------------------------------------- + +function tokenFor(ruleId: string): string { + return `${TOKEN_PREFIX}${ruleId}]`; +} + +const WHOLE_TOKEN = /^\[REDACTED:[a-z0-9][a-z0-9-]*\]$/; + +/** The token for one redacted value: onMatch's, if it returns a valid one. */ +function tokenForMatch(ruleId: string, value: string, onMatch: RedactionContext['onMatch']): string { + const custom = onMatch?.(ruleId, value); + return typeof custom === 'string' && WHOLE_TOKEN.test(custom) ? custom : tokenFor(ruleId); +} + +function tokenSpans(text: string): Array<[number, number]> { + const spans: Array<[number, number]> = []; + for (const m of text.matchAll(TOKEN_PATTERN)) spans.push([m.index!, m.index! + m[0].length]); + return spans; +} + +function overlapsAny(spans: Array<[number, number]>, start: number, end: number): boolean { + for (const [s, e] of spans) if (start < e && end > s) return true; + return false; +} + +/** + * Rewrite text[start, end) keeping the existing tokens in it and replacing each + * stretch between them that has a letter or digit with `token`. Returns null + * when nothing outside the tokens needs redacting, which keeps this idempotent. + */ +function redactAroundTokens( + text: string, + spans: Array<[number, number]>, + start: number, + end: number, + token: () => string +): string | null { + let out = ''; + let pos = start; + let replaced = false; + const flush = (to: number) => { + const piece = text.slice(pos, to); + if (/[A-Za-z0-9]/.test(piece)) { + out += token(); + replaced = true; + } else { + out += piece; + } + }; + for (const [s, e] of spans) { + if (e <= start || s >= end) continue; + if (s > pos) flush(s); + out += text.slice(Math.max(s, pos), Math.min(e, end)); + pos = Math.min(e, end); + } + if (pos < end) flush(end); + return replaced ? out : null; +} + +/** + * Describe a value without revealing it: length, character classes and + * Shannon entropy (bits per char), e.g. "len=40 aA9- H=4.9". Lets someone + * reviewing hits tell a random key from a word or a placeholder. + */ +export function describeShape(value: string): string { + const classes = + (/[a-z]/.test(value) ? 'a' : '') + + (/[A-Z]/.test(value) ? 'A' : '') + + (/[0-9]/.test(value) ? '9' : '') + + (/[^A-Za-z0-9\s]/.test(value) ? '-' : '') + + (/\s/.test(value) ? '_' : ''); + return `len=${value.length} ${classes || '?'} H=${shannonEntropy(value).toFixed(1)}`; +} + +/** Each `[REDACTED:]` token in `text`, left to right. */ +export function findRedactionTokens(text: string): Array<{ ruleId: string; start: number; end: number }> { + const out: Array<{ ruleId: string; start: number; end: number }> = []; + for (const m of text.matchAll(TOKEN_PATTERN)) { + out.push({ ruleId: m[0].slice(TOKEN_PREFIX.length, -1), start: m.index!, end: m.index! + m[0].length }); + } + return out; +} + +function shannonEntropy(text: string): number { + const counts = new Map(); + for (const ch of text) counts.set(ch, (counts.get(ch) ?? 0) + 1); + let entropy = 0; + for (const n of counts.values()) { + const p = n / text.length; + entropy -= p * Math.log2(p); + } + return entropy; +} + +/** + * Replace each accepted match (or its secretGroup) with a token. A candidate + * that overlaps existing tokens keeps them, and only the text around them is + * redacted (so re-running is a no-op). A candidate is skipped when it fully + * matches an allowlist pattern. + */ +function applyRule( + text: string, + rule: CompiledRule, + isAllowed: (s: string) => boolean, + onMatch?: RedactionContext['onMatch'] +): { text: string; count: number } { + const spans = text.includes(TOKEN_PREFIX) ? tokenSpans(text) : null; + let out = ''; + let last = 0; + let count = 0; + for (const m of text.matchAll(rule.regex)) { + let range: [number, number] | undefined; + for (const group of rule.secretGroups) { + range = m.indices?.[group]; + if (range) break; + } + if (!range) continue; + const [start, end] = range; + if (end <= start || start < last) continue; + if (rule.useAllowlist && isAllowed(text.slice(start, end))) continue; + if (spans && overlapsAny(spans, start, end)) { + let token: string | undefined; + const rewritten = redactAroundTokens(text, spans, start, end, () => + (token ??= tokenForMatch(rule.id, text.slice(start, end).replace(TOKEN_PATTERN, ''), onMatch))); + if (rewritten === null) continue; + out += text.slice(last, start) + rewritten; + last = end; + count++; + continue; + } + out += text.slice(last, start) + tokenForMatch(rule.id, text.slice(start, end), onMatch); + last = end; + count++; + } + return count === 0 ? { text, count } : { text: out + text.slice(last), count }; +} + +function applyEntropy( + text: string, + spec: EntropySpec, + isAllowed: (s: string) => boolean, + onMatch?: RedactionContext['onMatch'] +): { text: string; count: number } { + const candidates = new RegExp(`[A-Za-z0-9+/=_.~-]{${spec.minLength},}`, 'g'); + const spans = text.includes(TOKEN_PREFIX) ? tokenSpans(text) : null; + let out = ''; + let last = 0; + let count = 0; + for (const m of text.matchAll(candidates)) { + const start = m.index!; + const end = start + m[0].length; + if (spans && overlapsAny(spans, start, end)) continue; + if (isAllowed(m[0])) continue; + if (shannonEntropy(m[0]) < spec.threshold) continue; + if (spec.requireKeyword) { + let before = text.slice(Math.max(0, start - spec.window), start); + before = before.slice(before.lastIndexOf('\n') + 1).toLowerCase(); + if (!spec.keywords.some(k => before.includes(k))) continue; + } + out += text.slice(last, start) + tokenForMatch(ENTROPY_RULE_ID, m[0], onMatch); + last = end; + count++; + } + return count === 0 ? { text, count } : { text: out + text.slice(last), count }; +} + +export function createRedactor(config: RedactionConfig): Redactor { + const compiled = compileConfig(config); + const isAllowed = (secret: string) => compiled.allowlist.some(re => re.test(secret)); + + return { + ruleIds: compiled.rules.map(r => r.id), + isSecretField(name: string): boolean { + if (!compiled.secretField || typeof name !== 'string') return false; + const normalized = name.toLowerCase().replace(/[^a-z0-9]/g, '').replace(/\d+$/, ''); + return normalized.length > 0 && compiled.secretField.test(normalized); + }, + redact(text: string, ctx?: RedactionContext): RedactionResult { + if (typeof text !== 'string' || text.length === 0) return { text, findings: [] }; + let current = text; + let lower: string | null = null; + const counts = new Map(); + + for (const rule of compiled.rules) { + if (rule.keywords) { + lower ??= current.toLowerCase(); + if (!rule.keywords.some(k => lower!.includes(k))) continue; + } + const r = applyRule(current, rule, isAllowed, ctx?.onMatch); + if (r.count > 0) { + current = r.text; + lower = null; + counts.set(rule.id, (counts.get(rule.id) ?? 0) + r.count); + } + } + + if (compiled.entropy.enabled) { + const r = applyEntropy(current, compiled.entropy, isAllowed, ctx?.onMatch); + if (r.count > 0) { + current = r.text; + counts.set(ENTROPY_RULE_ID, r.count); + } + } + + return { text: current, findings: [...counts].map(([ruleId, count]) => ({ ruleId, count })) }; + }, + }; +} + +// --------------------------------------------------------------------------- +// Findings +// --------------------------------------------------------------------------- + +/** Aggregates findings by rule id. Never holds matched values. */ +export class FindingsTally { + private counts = new Map(); + + add(findings: RedactionFinding[]): void { + for (const f of findings) this.counts.set(f.ruleId, (this.counts.get(f.ruleId) ?? 0) + f.count); + } + + get total(): number { + let n = 0; + for (const c of this.counts.values()) n += c; + return n; + } + + toArray(): RedactionFinding[] { + return [...this.counts].map(([ruleId, count]) => ({ ruleId, count })); + } +} + +/** "7 value(s) redacted (azure-storage-key: 2, jwt: 5)" */ +export function formatFindings(findings: RedactionFinding[]): string { + const total = findings.reduce((n, f) => n + f.count, 0); + if (total === 0) return 'no values redacted'; + const detail = [...findings] + .sort((a, b) => b.count - a.count || a.ruleId.localeCompare(b.ruleId)) + .map(f => `${f.ruleId}: ${f.count}`) + .join(', '); + return `${total} value(s) redacted (${detail})`; +} + +// --------------------------------------------------------------------------- +// JSON / JSONL / files +// --------------------------------------------------------------------------- + +/** + * Redact a parsed JSON tree in place: string values by the text rules, values + * by their field name (secretFields), and keys that are themselves secrets. + */ +function redactTree( + node: unknown, + redactor: Redactor, + ctx: RedactionContext | undefined, + tally: FindingsTally | undefined +): { value: unknown; changed: boolean } { + if (typeof node === 'string') { + const r = redactor.redact(node, ctx); + if (r.findings.length === 0) return { value: node, changed: false }; + tally?.add(r.findings); + return { value: r.text, changed: r.text !== node }; + } + if (node === null || typeof node !== 'object') return { value: node, changed: false }; + + let changed = false; + if (Array.isArray(node)) { + for (let i = 0; i < node.length; i++) { + const r = redactTree(node[i], redactor, ctx, tally); + if (r.changed) { node[i] = r.value; changed = true; } + } + return { value: node, changed }; + } + const obj = node as Record; + const keys = Object.keys(obj); + for (const key of keys) { + const r = redactTree(obj[key], redactor, ctx, tally); + if (r.changed) { + setOwn(obj, key, r.value); + changed = true; + } + } + + // Field-name context: the key (or a {name, value} pair's name, or a Key + // Vault secret id) says the value is a secret even when its shape matches + // no rule — e.g. a letters-only clientSecret field in a structured + // MCP result. Runs after the text rules; a value they redacted only in part + // is still replaced whole, since the field name says all of it is secret. + const redactWhole = (key: string) => { + const value = obj[key]; + if (!isRedactableWhole(value)) return; + setOwn(obj, key, tokenForMatch(FIELD_RULE_ID, value, ctx?.onMatch)); + tally?.add([{ ruleId: FIELD_RULE_ID, count: 1 }]); + changed = true; + }; + for (const key of keys) { + if (redactor.isSecretField(key)) redactWhole(key); + } + const lower = new Map(keys.map(k => [k.toLowerCase(), k])); + const valueKey = lower.get('value'); + if (valueKey !== undefined) { + const nameKey = lower.get('name') ?? lower.get('key'); + const nameValue = nameKey !== undefined ? obj[nameKey] : undefined; + const id = lower.has('id') ? obj[lower.get('id')!] : undefined; + if ( + (typeof nameValue === 'string' && redactor.isSecretField(nameValue)) || + (typeof id === 'string' && KEY_VAULT_SECRET_ID.test(id)) + ) { + redactWhole(valueKey); + } + } + + // Keys themselves: a secret used as a key (a token-keyed cache) is renamed + // to its token. Harness keys never match a rule, so structure is unchanged. + let renamed: Array<[string, unknown]> | null = null; + for (let i = 0; i < keys.length; i++) { + const r = redactor.redact(keys[i], ctx); + if (r.findings.length === 0) continue; + tally?.add(r.findings); + renamed ??= keys.map(k => [k, obj[k]]); + renamed[i][0] = r.text; + } + if (renamed) { + const rebuilt: Record = {}; + for (const [key, value] of renamed) { + let unique = key; + for (let n = 2; Object.prototype.hasOwnProperty.call(rebuilt, unique); n++) unique = `${key}#${n}`; + setOwn(rebuilt, unique, value); + } + return { value: rebuilt, changed: true }; + } + return { value: obj, changed }; +} + +function setOwn(obj: Record, key: string, value: unknown): void { + // defineProperty so a "__proto__" key from JSON.parse stays an own property. + Object.defineProperty(obj, key, { value, writable: true, enumerable: true, configurable: true }); +} + +// A value that is only tokens is already redacted; anything else in a secret +// field, even alongside a token, is replaced whole. +const ONLY_TOKENS = /^\s*(?:\[REDACTED:[a-z0-9][a-z0-9-]*\]\s*)+$/; + +function isRedactableWhole(value: unknown): value is string { + return typeof value === 'string' && + value.trim().length > 0 && + !ONLY_TOKENS.test(value) && + !PLACEHOLDER_VALUE.test(value.trim()); +} + +/** + * Redact one JSONL line. Valid JSON stays valid (string values are redacted + * and the line is re-serialized only if something changed); a line that isn't + * JSON (e.g. a torn last line mid-write) is redacted as raw text. A trailing + * `\r` is preserved. + */ +export function redactJsonlLine( + line: string, + redactor: Redactor, + ctx?: RedactionContext, + tally?: FindingsTally +): string { + const cr = line.endsWith('\r') ? '\r' : ''; + const body = cr ? line.slice(0, -1) : line; + if (body.trim().length === 0) return line; + + let parsed: unknown; + try { + parsed = JSON.parse(body); + } catch { + const r = redactor.redact(body, ctx); + if (r.findings.length === 0) return line; + tally?.add(r.findings); + return r.text + cr; + } + + const r = redactTree(parsed, redactor, ctx, tally); + return r.changed ? JSON.stringify(r.value) + cr : line; +} + +const COPY_CHUNK_BYTES = 1 << 20; // 1 MiB + +/** + * Stream `src` to `dest`, redacting line by line. Writes `dest` directly; + * callers that need atomicity write to a temp path and rename. Line count and + * trailing-newline shape match the source exactly, so archive line numbers + * (exchanges.line_start/line_end, MCP read ranges) stay valid. + */ +export function copyFileRedacted( + src: string, + dest: string, + redactor: Redactor, + ctx?: RedactionContext, + tally?: FindingsTally +): void { + const fdIn = fs.openSync(src, 'r'); + let fdOut: number | undefined; + try { + // Create with the source's mode, as copyFileSync would, so a 0600 + // transcript doesn't become world-readable in the archive. + fdOut = fs.openSync(dest, 'w', fs.fstatSync(fdIn).mode & 0o777); + const buf = Buffer.allocUnsafe(COPY_CHUNK_BYTES); + const decoder = new StringDecoder('utf8'); + let pending = ''; + let bytesRead: number; + while ((bytesRead = fs.readSync(fdIn, buf, 0, buf.length, null)) > 0) { + let scanFrom = pending.length; + pending += decoder.write(buf.subarray(0, bytesRead)); + const out: string[] = []; + let start = 0; + let nl: number; + while ((nl = pending.indexOf('\n', scanFrom)) !== -1) { + out.push(redactJsonlLine(pending.slice(start, nl), redactor, ctx, tally), '\n'); + start = nl + 1; + scanFrom = start; + } + if (out.length > 0) fs.writeSync(fdOut, out.join('')); + pending = pending.slice(start); + } + pending += decoder.end(); + if (pending.length > 0) fs.writeSync(fdOut, redactJsonlLine(pending, redactor, ctx, tally)); + } finally { + fs.closeSync(fdIn); + if (fdOut !== undefined) fs.closeSync(fdOut); + } +} diff --git a/src/search.ts b/src/search.ts index 786d5119..b6c5e3fb 100644 --- a/src/search.ts +++ b/src/search.ts @@ -1,5 +1,6 @@ import Database from 'better-sqlite3'; import { initDatabase } from './db.js'; +import { createSearchAdmission } from './record-admission.js'; import { initEmbeddings, generateQueryEmbedding } from './embeddings.js'; import { SearchResult, ConversationExchange, MultiConceptResult } from './types.js'; import { isErroredSentinel } from './summary-sentinel.js'; @@ -188,81 +189,77 @@ export async function searchConversations( if (after) validateISODate(after, '--after'); if (before) validateISODate(before, '--before'); + const admission = createSearchAdmission(); const db = initDatabase(); + const admitted = (row: any) => admission.admit(exchangeFromRow(row)) !== null; let results: any[] = []; const { sql: filterClause, params: filterParams } = buildSearchFilters(options); - if (mode === 'vector' || mode === 'both') { - // Vector similarity search. - // vec0 applies KNN before the WHERE clause and before our sidechain - // de-rank, so we over-fetch candidates and trim after the final ordering. - await initEmbeddings(); - const queryEmbedding = await generateQueryEmbedding(query); - const k = limit * 3; - - const stmt = db.prepare(` - SELECT - ${EXCHANGE_SELECT_COLUMNS}, - vec.distance - FROM vec_exchanges AS vec - JOIN exchanges AS e ON vec.id = e.id - WHERE vec.embedding MATCH ? - AND k = ? - ${sidechainClause} - ${filterClause} - ORDER BY (vec.distance + e.is_sidechain * ?) ASC - `); - - results = stmt.all( - Buffer.from(new Float32Array(queryEmbedding).buffer), - k, - ...filterParams, - SIDECHAIN_DISTANCE_PENALTY - ); - if (results.length > limit) { - results = results.slice(0, limit); + try { + if (mode === 'vector' || mode === 'both') { + // Order the filtered candidates before admission. The vec0 KNN API caps k; + // increasing k cannot find safe results beyond a large excluded prefix. + await initEmbeddings(); + const queryEmbedding = await generateQueryEmbedding(query); + const stmt = db.prepare(` + SELECT ${EXCHANGE_SELECT_COLUMNS}, + vec_distance_l2(vec.embedding, ?) AS retrieval_distance + FROM vec_exchanges AS vec + JOIN exchanges AS e ON vec.id = e.id + WHERE 1 = 1 ${sidechainClause} ${filterClause} + ORDER BY (retrieval_distance + e.is_sidechain * ?) ASC + `); + for (const row of stmt.iterate( + Buffer.from(new Float32Array(queryEmbedding).buffer), + ...filterParams, SIDECHAIN_DISTANCE_PENALTY + )) { + if (admitted(row)) results.push(row); + if (results.length === limit) break; + } } - } - if (mode === 'text' || mode === 'both') { - // Text search: AND of per-token LIKE patterns so multi-word queries match - // when every term appears somewhere in the exchange (any order, either field). - // See #127 — whole-query contiguous substring matching returned empty for - // typical multi-word searches that are not verbatim phrases. Sidechain rows - // are de-ranked (ordered after main-thread rows), not excluded (#128). - const { sql: textMatchSql, params: textMatchParams } = buildTextMatchClause(query); - const textStmt = db.prepare(` - SELECT - ${EXCHANGE_SELECT_COLUMNS}, - 0 as distance - FROM exchanges AS e - WHERE ${textMatchSql} - ${sidechainClause} - ${filterClause} - ORDER BY e.is_sidechain ASC, e.timestamp DESC - LIMIT ? - `); - - const textResults = textStmt.all(...textMatchParams, ...filterParams, limit); - - if (mode === 'both') { - // Merge and deduplicate by ID - const seenIds = new Set(results.map(r => r.id)); - for (const textResult of textResults) { - if (!seenIds.has((textResult as any).id)) { - results.push(textResult); + if (mode === 'text' || mode === 'both') { + // Text search: AND of per-token LIKE patterns so multi-word queries match + // when every term appears somewhere in the exchange (any order, either field). + // See #127 — whole-query contiguous substring matching returned empty for + // typical multi-word searches that are not verbatim phrases. Sidechain rows + // are de-ranked (ordered after main-thread rows), not excluded (#128). + const { sql: textMatchSql, params: textMatchParams } = buildTextMatchClause(query); + const textStmt = db.prepare(` + SELECT + ${EXCHANGE_SELECT_COLUMNS}, + 0 as retrieval_distance + FROM exchanges AS e + WHERE ${textMatchSql} + ${sidechainClause} + ${filterClause} + ORDER BY e.is_sidechain ASC, e.timestamp DESC + `); + + const textResults: any[] = []; + for (const row of textStmt.iterate(...textMatchParams, ...filterParams)) { + if (admitted(row)) textResults.push(row); + if (textResults.length === limit) break; + } + + if (mode === 'both') { + // Merge and deduplicate by ID + const seenIds = new Set(results.map(r => r.id)); + for (const textResult of textResults) { + if (!seenIds.has((textResult as any).id)) { + results.push(textResult); + } } + } else { + results = textResults; } - } else { - results = textResults; } - } - db.close(); + } finally { db.close(); } - return results.map((row: any) => { + const output = results.map((row: any) => { const exchange = exchangeFromRow(row); // Try to load summary if available. Skip error sentinels (#96) so failed @@ -272,7 +269,7 @@ export async function searchConversations( if (fs.existsSync(summaryPath)) { const raw = fs.readFileSync(summaryPath, 'utf-8'); if (!isErroredSentinel(raw)) { - summary = raw.trim(); + summary = admission.summary(exchange, raw.trim()); } } @@ -282,11 +279,13 @@ export async function searchConversations( return { exchange, - similarity: mode === 'text' ? undefined : l2DistanceToCosineSimilarity(row.distance), + similarity: mode === 'text' ? undefined : l2DistanceToCosineSimilarity(row.retrieval_distance), snippet, summary } as SearchResult & { summary?: string }; }); + admission.verify(); + return output; } // Helper function to count lines in a file efficiently diff --git a/src/show-cli.ts b/src/show-cli.ts index 397267e0..689e1aff 100644 --- a/src/show-cli.ts +++ b/src/show-cli.ts @@ -1,4 +1,4 @@ -import { readFileSync } from 'fs'; +import { readAdmittedConversation } from './record-admission.js'; import { formatConversationAsMarkdown, formatConversationAsHTML } from './show.js'; const args = process.argv.slice(2); @@ -46,7 +46,7 @@ if (!filePath) { } try { - const jsonl = readFileSync(filePath, 'utf-8'); + const jsonl = await readAdmittedConversation(filePath); if (format === 'html') { console.log(formatConversationAsHTML(jsonl)); diff --git a/src/show.ts b/src/show.ts index 06a34b45..acab547c 100644 --- a/src/show.ts +++ b/src/show.ts @@ -24,15 +24,16 @@ interface ConversationMessage { } export function formatConversationAsMarkdown(jsonl: string, startLine?: number, endLine?: number): string { - const allLines = jsonl.trim().split('\n').filter(line => line.trim()); + const allLines = jsonl.split('\n'); // Apply line range if specified (1-indexed, inclusive) - const lines = startLine !== undefined || endLine !== undefined + const selectedLines = startLine !== undefined || endLine !== undefined ? allLines.slice( startLine !== undefined ? startLine - 1 : 0, endLine !== undefined ? endLine : undefined ) : allLines; + const lines = selectedLines.filter(line => line.trim()); if (isCodexRollout(lines)) { return formatCodexConversationAsMarkdown(lines); diff --git a/src/summarizer.ts b/src/summarizer.ts index d7e3cc04..d8269d6d 100644 --- a/src/summarizer.ts +++ b/src/summarizer.ts @@ -1,5 +1,9 @@ import fs from 'fs'; +import os from 'node:os'; +import path from 'node:path'; +import { admitExchange, readRecordExclusions } from './record-admission.js'; import { ConversationExchange } from './types.js'; +import { getClaudeDir } from './paths.js'; import { query } from '@anthropic-ai/claude-agent-sdk'; import { SUMMARIZER_CONTEXT_MARKER } from './constants.js'; import { VERSION } from './version.js'; @@ -121,10 +125,11 @@ export interface CodexSummarizerCommand { command: string; args: string[]; prompt: string; - sessionId: string; + sessionId?: string; model?: string; versionArgs?: string[]; skipVersionCheck?: boolean; + isolated?: boolean; } /** @@ -330,6 +335,24 @@ function extractSummary(text: string): string { return text.trim(); } +function readClaudeSummaryAuthentication(): { apiKeyHelper?: string; baseUrl?: string } { + let settings: { apiKeyHelper?: unknown; env?: { ANTHROPIC_BASE_URL?: unknown } } | null; + try { + settings = JSON.parse(fs.readFileSync(path.join(getClaudeDir(), 'settings.json'), 'utf8')); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return {}; + // Parser and filesystem errors can contain private setting values or paths. + throw new Error('Cannot read Claude summarizer authentication settings'); + } + const apiKeyHelper = settings?.apiKeyHelper; + const baseUrl = settings?.env?.ANTHROPIC_BASE_URL; + if ((apiKeyHelper !== undefined && typeof apiKeyHelper !== 'string') || + (baseUrl !== undefined && typeof baseUrl !== 'string')) { + throw new Error('Cannot read Claude summarizer authentication settings'); + } + return { apiKeyHelper, baseUrl }; +} + /** * Build the options object passed to the Claude Agent SDK's query() for a * summarization call. @@ -348,10 +371,16 @@ export function buildSummarizerQueryOptions(args: { cwd?: string; }): Record { const { model, sessionId, cwd } = args; + const authentication = readClaudeSummaryAuthentication(); + const env = getApiEnv()!; + if (!env.ANTHROPIC_BASE_URL && authentication.baseUrl) { + env.ANTHROPIC_BASE_URL = authentication.baseUrl; + } return { model, max_tokens: 4096, - env: getApiEnv(), + env, + ...(authentication.apiKeyHelper ? { settings: { apiKeyHelper: authentication.apiKeyHelper } } : {}), resume: sessionId, persistSession: false, // Summarizers never call tools — disabling them stops a resumed live @@ -366,7 +395,7 @@ export function buildSummarizerQueryOptions(args: { // strictMcpConfig only blocks ~/.claude.json MCP servers; user settings // still load every enabled plugin (each with its own MCP server) plus the // user's Stop/Notification hooks (#136) into the subprocess. Summarizers - // need no plugins, hooks, or user settings at all. + // need no behavior settings. Only authentication is passed separately. settingSources: [], // Resume looks up the session under ~/.claude/projects//, so pass the recorded cwd when it still exists on disk. ...(cwd && fs.existsSync(cwd) ? { cwd } : {}), @@ -404,19 +433,34 @@ Bad: } export function buildCodexSummarizerCommand(args: { - sessionId: string; + sessionId?: string; prompt: string; model?: string; codexBin?: string; + isolated?: boolean; }): CodexSummarizerCommand { const command = args.codexBin || process.env.EPISODIC_MEMORY_CODEX_BIN || 'codex'; + if (args.isolated && args.sessionId) throw new Error('An isolated summary cannot resume a source session'); + const restrictions = args.isolated ? [ + ...['shell_tool', 'unified_exec', 'shell_snapshot', 'apps', 'plugins', 'remote_plugin', + 'hooks', 'multi_agent', 'multi_agent_v2', 'memories', 'goals', 'computer_use', + 'browser_use', 'browser_use_external', 'image_generation', 'workspace_dependencies', + 'skill_search', 'skill_mcp_dependency_install', 'tool_suggest', 'code_mode_host', + 'sleep_tool', 'view_image', 'default_mode_request_user_input'] + .map(feature => `features.${feature}=false`), + 'features.skip_host_skill_discovery=true', 'skills.bundled.enabled=false', + 'skills.include_instructions=false', 'project_doc_max_bytes=0', + 'tools.experimental_request_user_input.enabled=false', 'web_search="disabled"', + 'history.persistence="none"', + ].flatMap(value => ['-c', value]) : []; return { command, - args: ['app-server'], + args: ['app-server', ...restrictions], prompt: args.prompt, sessionId: args.sessionId, model: args.model, + ...(args.isolated ? { isolated: true } : {}), }; } @@ -447,6 +491,8 @@ interface PendingAppServerRequest { reject: (error: Error) => void; } +const CODEX_SUMMARY_TERMINATION_GRACE_MS = 1024; + function appServerTimeoutMs(): number { const configured = Number(process.env.EPISODIC_MEMORY_CODEX_SUMMARY_TIMEOUT_MS); return Number.isFinite(configured) && configured > 0 ? configured : 120000; @@ -484,6 +530,9 @@ async function assertSupportedCodexVersion(command: CodexSummarizerCommand): Pro const output = await readCommandOutput(command.command, command.versionArgs || ['--version']); const version = parseCodexCliVersion(output); + if (command.isolated && (!version || !versionMeetsMinimum(version, '0.154.0'))) { + throw new Error('Isolated record-policy summarization requires codex-cli >= 0.154.0'); + } if (!version || !versionMeetsMinimum(version)) { throw new Error(codexVersionRequirementMessage(output)); } @@ -506,11 +555,16 @@ function requireTurnId(result: any, method: string): string { } export async function runCodexCommand(command: CodexSummarizerCommand): Promise { + if (command.isolated && command.sessionId) throw new Error('An isolated summary cannot resume a source session'); await assertSupportedCodexVersion(command); + const summaryCwd = command.isolated + ? fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'episodic-memory-summary-'))) + : undefined; return new Promise((resolve, reject) => { const child = spawn(command.command, command.args, { env: getApiEnv(), + ...(summaryCwd ? { cwd: summaryCwd } : {}), stdio: ['pipe', 'pipe', 'pipe'] }); @@ -519,7 +573,9 @@ export async function runCodexCommand(command: CodexSummarizerCommand): Promise< let nextRequestId = 1; let targetTurnId: string | undefined; let finished = false; + let terminalOutcome: { error?: Error; result: string } | undefined; let timeout: NodeJS.Timeout | undefined; + let terminationTimeout: NodeJS.Timeout | undefined; const pending = new Map(); const lines = createInterface({ input: child.stdout }); @@ -535,12 +591,24 @@ export async function runCodexCommand(command: CodexSummarizerCommand): Promise< if (!child.killed) { child.kill('SIGTERM'); } + if (command.isolated && child.exitCode === null && child.signalCode === null) { + // Bound shutdown without removing the owned directory before close. + terminationTimeout = setTimeout(() => { + if (child.exitCode === null && child.signalCode === null) child.kill('SIGKILL'); + }, CODEX_SUMMARY_TERMINATION_GRACE_MS); + } }; const finish = (error: Error | undefined, result = '') => { if (finished) return; finished = true; cleanup(); + if (command.isolated) { + // Wait for process termination before removing its owned directory. + // Provider payloads may repeat input; never persist them in error sidecars. + terminalOutcome = { error: error ? new Error('Isolated Codex summary failed; provider details suppressed') : undefined, result }; + return; + } if (error) { reject(error); } else { @@ -622,6 +690,16 @@ export async function runCodexCommand(command: CodexSummarizerCommand): Promise< finish(new Error(detail)); } }); + child.on('close', () => { + if (terminationTimeout) clearTimeout(terminationTimeout); + if (!command.isolated) return; + try { + if (summaryCwd) fs.rmSync(summaryCwd, { recursive: true, force: true }); + if (terminalOutcome?.error) reject(terminalOutcome.error); + else if (terminalOutcome) resolve(terminalOutcome.result); + else reject(new Error('Isolated Codex summary ended without a terminal result')); + } catch { reject(new Error('Isolated Codex summary cleanup failed')); } + }); (async () => { try { @@ -637,15 +715,44 @@ export async function runCodexCommand(command: CodexSummarizerCommand): Promise< }); notify('initialized'); - const fork = await send('thread/fork', { - threadId: command.sessionId, + let isolation: Record = {}; + if (command.isolated) { + const configured = await send('config/read', { includeLayers: false }); + const servers = configured?.config?.mcp_servers; + if (!configured?.config || servers !== undefined && (!servers || typeof servers !== 'object' || Array.isArray(servers))) { + throw new Error('Cannot establish configured MCP boundary'); + } + const listed = await send('skills/list', { cwds: [summaryCwd], forceReload: true }); + if (!Array.isArray(listed?.data) || listed.data.some((entry: any) => !Array.isArray(entry.skills) || !Array.isArray(entry.errors) || entry.errors.length)) { + throw new Error('Cannot establish configured skill boundary'); + } + const skills = listed.data.flatMap((entry: any) => entry.skills).map((skill: any) => { + if (typeof skill.path !== 'string' || !path.isAbsolute(skill.path)) throw new Error('Invalid configured skill identity'); + return { path: skill.path, enabled: false }; + }); + isolation = { + cwd: summaryCwd, + dynamicTools: [], environments: [], selectedCapabilityRoots: [], + allowProviderModelFallback: false, + baseInstructions: 'Summarize only the admitted transcript. Do not use tools or recover external context.', + developerInstructions: 'Output only a concise factual summary inside .', + config: { + mcp_servers: Object.fromEntries(Object.keys(servers ?? {}).map(name => [name, { enabled: false }])), + 'skills.config': skills, + }, + }; + } + const method = command.sessionId ? 'thread/fork' : 'thread/start'; + const fork = await send(method, { + ...(command.sessionId ? { threadId: command.sessionId } : {}), ephemeral: true, excludeTurns: true, sandbox: 'read-only', approvalPolicy: 'never', ...(command.model ? { model: command.model } : {}), + ...isolation, }); - const forkThreadId = requireThreadId(fork, 'thread/fork'); + const forkThreadId = requireThreadId(fork, method); const turn = await send('turn/start', { threadId: forkThreadId, @@ -663,8 +770,8 @@ export async function runCodexCommand(command: CodexSummarizerCommand): Promise< }); } -async function callCodex(prompt: string, sessionId: string, model?: string): Promise { - const command = buildCodexSummarizerCommand({ sessionId, prompt, model }); +async function callCodex(prompt: string, sessionId?: string, model?: string, isolated = false): Promise { + const command = buildCodexSummarizerCommand({ sessionId, prompt, model, isolated }); return runCodexCommand(command); } @@ -680,7 +787,7 @@ function getCodexSessionId(exchanges: ConversationExchange[], sessionId?: string if (!exchanges.some(exchange => exchange.harness === 'codex')) { return undefined; } - return sessionId || exchanges.find(exchange => exchange.sessionId)?.sessionId; + return sessionId || exchanges.find(exchange => exchange.harness === 'codex' && exchange.sessionId)?.sessionId; } /** @@ -693,8 +800,8 @@ function getCodexSessionId(exchanges: ConversationExchange[], sessionId?: string * * Default to `undefined` so `app-server` uses the current Codex config * (`~/.codex/config.toml#model`). Operators can override via - * `EPISODIC_MEMORY_CODEX_MODEL` if they need a specific model id (e.g. an - * API-key user wanting `gpt-5.5-codex`). + * `EPISODIC_MEMORY_CODEX_MODEL` if they need a supported model id for their + * selected Codex route. * * See https://github.com/obra/episodic-memory/issues/98. */ @@ -702,7 +809,14 @@ export function getCodexModel(_exchanges: ConversationExchange[]): string | unde return process.env.EPISODIC_MEMORY_CODEX_MODEL || undefined; } -export async function summarizeConversation(exchanges: ConversationExchange[], sessionId?: string): Promise { +export interface SummarizeOptions { + allowResume?: boolean; +} + +export async function summarizeConversation(exchanges: ConversationExchange[], sessionId?: string, options: SummarizeOptions = {}): Promise { + const recordPolicy = readRecordExclusions(); + exchanges = exchanges.map(exchange => admitExchange(exchange, recordPolicy)).filter((exchange): exchange is ConversationExchange => exchange !== null); + const allowResume = options.allowResume !== false && recordPolicy === null; // Handle trivial conversations if (exchanges.length === 0) { return 'Trivial conversation with no substantive content.'; @@ -715,20 +829,21 @@ export async function summarizeConversation(exchanges: ConversationExchange[], s } } - const codexSessionId = getCodexSessionId(exchanges, sessionId); - if (codexSessionId) { - try { - const result = await callCodex(buildCodexSummaryPrompt(), codexSessionId, getCodexModel(exchanges)); - return extractSummary(result); - } catch (error) { - console.log(` Codex summarizer unavailable, falling back to transcript text: ${error instanceof Error ? error.message : String(error)}`); - } + const codexSessionId = allowResume ? getCodexSessionId(exchanges, sessionId) : undefined; + if (exchanges.some(exchange => exchange.harness === 'codex')) { + // Filtered text must not recover unscreened source context. + const isolated = !codexSessionId; + const prompt = isolated + ? `${SUMMARIZER_CONTEXT_MARKER}. Summarize only the following admitted transcript in 2-4 factual sentences inside . Do not inspect files, use tools or recover any original session.\n\n${formatConversationText(exchanges)}` + : buildCodexSummaryPrompt(); + const result = await callCodex(prompt, isolated ? undefined : codexSessionId, getCodexModel(exchanges), isolated); + return extractSummary(result); } // Cost safety (#104): everything below this point calls the metered Claude API - // path. If a stray global ANTHROPIC_API_KEY would be billed and the user hasn't + // path. Codex conversations return or fail before reaching this point. + // If a stray global ANTHROPIC_API_KEY would be billed and the user hasn't // acknowledged it, warn once and proceed (don't silently spend without notice). - // Placed AFTER the Codex attempt so pure-Codex users are never warned needlessly. if (wouldBillMeteredApi() && !meteredApiOptIn()) { warnMeteredApiOnce(); } @@ -742,7 +857,7 @@ export async function summarizeConversation(exchanges: ConversationExchange[], s const isClaudeSession = exchanges.some( e => e.harness === 'claude' || e.harness === undefined ); - const claudeSessionId = !codexSessionId && isClaudeSession ? sessionId : undefined; + const claudeSessionId = allowResume && isClaudeSession ? sessionId : undefined; const cwd = claudeSessionId ? exchanges.find(e => e.cwd)?.cwd : undefined; const conversationText = claudeSessionId ? '' // When resuming, no need to include conversation text - it's already in context diff --git a/src/sync-cli.ts b/src/sync-cli.ts index 297b20f8..c24e1184 100644 --- a/src/sync-cli.ts +++ b/src/sync-cli.ts @@ -14,6 +14,7 @@ import { spawn } from 'child_process'; import fs from 'fs'; import { formatLogLine, getSyncLogPath, getSyncLockPath } from './logging.js'; import { acquireFileLock, readLockHolder, releaseFileLock } from './file-lock.js'; +import { FindingsTally, formatFindings, getRedactionSettings, loadRedactor, type Redactor } from './redaction.js'; const args = process.argv.slice(2); @@ -48,10 +49,13 @@ Sync conversations from Claude Code, Codex, and opencode transcript sources to a This command: 1. Exports opencode sessions from its SQLite database when available -2. Copies new or updated .jsonl files to conversation archive +2. Copies new or updated .jsonl files to conversation archive, redacting secrets 3. Generates embeddings for semantic search 4. Updates the search index +Secrets are replaced with [REDACTED:] tokens before anything is archived, +indexed, embedded, or summarized. See EPISODIC_MEMORY_REDACTION* in the README. + Only processes files that are new or have been modified since last sync. Safe to run multiple times - subsequent runs are fast no-ops. @@ -157,8 +161,23 @@ if (isBackground) { process.exit(0); } +// Load redaction rules before anything is exported, copied, or indexed. In +// strict mode (the default) a bad rules file stops the sync here: fail closed +// rather than archive unredacted text. +let redactor: Redactor | null; +try { + redactor = loadRedactor(); +} catch (error) { + console.error(`episodic-memory: ${error instanceof Error ? error.message : String(error)}`); + console.error( + 'episodic-memory: refusing to sync without redaction (strict mode). Fix the rules file, ' + + 'or set EPISODIC_MEMORY_REDACTION_STRICT=0 to sync unredacted, or EPISODIC_MEMORY_REDACTION=off.' + ); + process.exit(1); +} + if (!onlyHarnesses || onlyHarnesses.includes('opencode')) { - const opencodeExport = exportOpencodeSessions(); + const opencodeExport = exportOpencodeSessions({ redactor }); if (opencodeExport.exported > 0 || opencodeExport.skipped > 0) { console.log(`opencode export: ${opencodeExport.exported} exported, ${opencodeExport.skipped} skipped`); } @@ -215,8 +234,11 @@ console.log(`Destination: ${destDir}\n`); async function syncAll() { const totals = { copied: 0, skipped: 0, indexed: 0, summarized: 0, errors: [] as Array<{file: string; error: string}>, sourcesWithSummaryWork: 0, totalNeedingSummaries: 0 }; + const redactions = new FindingsTally(); + for (const sourceDir of sourceDirs) { - const result = await syncConversations(sourceDir, destDir, { ...syncOptions, summaryLimit }); + const result = await syncConversations(sourceDir, destDir, { ...syncOptions, summaryLimit, redactor }); + redactions.add(result.redactions); totals.copied += result.copied; totals.skipped += result.skipped; totals.indexed += result.indexed; @@ -224,7 +246,12 @@ async function syncAll() { totals.errors.push(...result.errors); } - console.log(`\n✅ Sync complete!`); + // Admission failure is a partial run, not successful synchronization. + // Finish migration before emitting a success claim as well. + if (totals.errors.length === 0) await runEmbeddingMigrationPhase(); + else process.exitCode = 1; + + console.log(totals.errors.length ? '\nSync finished with errors.' : '\n✅ Sync complete!'); console.log(` Copied: ${totals.copied}`); console.log(` Skipped: ${totals.skipped}`); console.log(` Indexed: ${totals.indexed}`); @@ -233,6 +260,14 @@ async function syncAll() { } else { console.log(` Summarized: ${totals.summarized}`); } + // Rule IDs and counts only — never matched values. + if (redactor) { + console.log(` Redaction: ${formatFindings(redactions.toArray())}`); + } else if (getRedactionSettings().enabled) { + console.log(' Redaction: NOT APPLIED (rules failed to load; EPISODIC_MEMORY_REDACTION_STRICT=0)'); + } else { + console.log(' Redaction: off (EPISODIC_MEMORY_REDACTION=off)'); + } if (totals.errors.length > 0) { console.log(`\n⚠️ Errors: ${totals.errors.length}`); @@ -246,10 +281,6 @@ async function syncAll() { } } - // After regular sync, do a batch of embedding migration if any rows are - // still on the old encoder. Lock-protected; if another process is already - // migrating, this is a no-op. - await runEmbeddingMigrationPhase(); } const MIGRATION_BATCH_SIZE = parseInt(process.env.EPISODIC_MEMORY_MIGRATION_BATCH || '500', 10); @@ -270,6 +301,7 @@ async function runEmbeddingMigrationPhase(): Promise { } } catch (err) { console.error('episodic-memory: migration phase error:', err instanceof Error ? err.message : err); + throw err; } finally { db.close(); } diff --git a/src/sync.ts b/src/sync.ts index d9ee6810..815b0d4a 100644 --- a/src/sync.ts +++ b/src/sync.ts @@ -1,52 +1,13 @@ import fs from 'fs'; +import { archiveAdmittedConversation, readRecordExclusions } from './record-admission.js'; import path from 'path'; -import { StringDecoder } from 'string_decoder'; -import { SUMMARIZER_CONTEXT_MARKER } from './constants.js'; import { getExcludedProjects, findJsonlFiles, statIfExists } from './paths.js'; import { formatErrorSentinel, shouldQueueForSummary } from './summary-sentinel.js'; import { getMaxMessageBytes, isOversizeExchange } from './message-size.js'; +import { FindingsTally, loadRedactor, type RedactionFinding, type Redactor } from './redaction.js'; -const EXCLUSION_MARKERS = [ - 'DO NOT INDEX THIS CHAT', - 'Only use NO_INSIGHTS_FOUND', - SUMMARIZER_CONTEXT_MARKER, -]; - -const MARKER_SCAN_CHUNK_BYTES = 1 << 20; // 1 MiB - -/** - * Stream and scan for any exclusion marker, carrying an overlap between - * chunks so a marker split across a boundary is still found. A single - * fs.readFileSync(path, 'utf-8') throws ERR_STRING_TOO_LONG above Node's - * ~512 MB max string length; the old catch returned false (fail OPEN), - * silently indexing a file whose DO NOT INDEX marker we never read (#152). - * Streaming confirms cleanliness at any size, and a real read error now - * fails CLOSED (skip) rather than open. - */ -export function shouldSkipConversation(filePath: string): boolean { - const maxMarkerLen = Math.max(...EXCLUSION_MARKERS.map(m => m.length)); - let fd: number | undefined; - try { - fd = fs.openSync(filePath, 'r'); - const buf = Buffer.allocUnsafe(MARKER_SCAN_CHUNK_BYTES); - const decoder = new StringDecoder('utf8'); - let carry = ''; - let bytesRead: number; - while ((bytesRead = fs.readSync(fd, buf, 0, buf.length, null)) > 0) { - const window = carry + decoder.write(buf.subarray(0, bytesRead)); - if (EXCLUSION_MARKERS.some(marker => window.includes(marker))) { - return true; - } - carry = window.slice(Math.max(0, window.length - (maxMarkerLen - 1))); - } - const tail = carry + decoder.end(); - return EXCLUSION_MARKERS.some(marker => tail.includes(marker)); - } catch { - return true; // fail closed (#152) - } finally { - if (fd !== undefined) { try { fs.closeSync(fd); } catch {} } - } -} +export { shouldSkipConversation } from './record-admission.js'; +import { shouldSkipConversation } from './record-admission.js'; /** * True when a transcript contains at least one message line in any supported @@ -105,12 +66,19 @@ export interface SyncResult { indexed: number; summarized: number; errors: Array<{ file: string; error: string }>; + /** Rule IDs and counts only — never matched values. */ + redactions: RedactionFinding[]; } export interface SyncOptions { skipIndex?: boolean; skipSummaries?: boolean; summaryLimit?: number; // Max summaries to generate per run (default: 10) + /** + * Redactor applied at the archive write. `undefined` loads it from the + * environment (loadRedactor); `null` means redaction is off. + */ + redactor?: Redactor | null; } /** @@ -132,37 +100,6 @@ export function buildSyncOptionsFromEnv(env: NodeJS.ProcessEnv): SyncOptions { return { skipSummaries: env.EPISODIC_MEMORY_SKIP_SUMMARIES === '1' }; } -function copyIfNewer(src: string, dest: string): boolean { - // Ensure destination directory exists - const destDir = path.dirname(dest); - if (!fs.existsSync(destDir)) { - fs.mkdirSync(destDir, { recursive: true }); - } - - // Check if destination exists and is up-to-date - if (fs.existsSync(dest)) { - const srcStat = fs.statSync(src); - const destStat = fs.statSync(dest); - if (destStat.mtimeMs >= srcStat.mtimeMs) { - return false; // Dest is current, skip - } - } - - // Atomic copy: temp file + rename - const tempDest = dest + '.tmp.' + process.pid; - fs.copyFileSync(src, tempDest); - fs.renameSync(tempDest, dest); // Atomic on same filesystem - - // Preserve source mtime: harnesses without per-message timestamps (Cursor - // agent transcripts) fall back to file mtime. Round up to the next whole - // millisecond — utimes can't always represent the source's sub-millisecond - // precision, and a dest mtime even fractionally older would defeat the - // skip-if-current check above on every subsequent sync. - const srcStat = fs.statSync(src); - fs.utimesSync(dest, srcStat.atimeMs / 1000, Math.ceil(srcStat.mtimeMs) / 1000); - return true; -} - export function extractSessionIdFromPath(filePath: string): string | null { // Extract session ID from Claude filename or Codex rollout filename. const basename = path.basename(filePath, '.jsonl'); @@ -179,14 +116,22 @@ export async function syncConversations( destDir: string, options: SyncOptions = {} ): Promise { + const exclusions = readRecordExclusions(); const result: SyncResult = { copied: 0, skipped: 0, indexed: 0, summarized: 0, - errors: [] + errors: [], + redactions: [] }; + // Load the redactor before touching the archive. In strict mode (the + // default) a rules-load failure throws here, so nothing unredacted is + // archived, indexed, or summarized (fail closed). + const redactor = options.redactor === undefined ? loadRedactor() : options.redactor; + const tally = new FindingsTally(); + // Ensure source directory exists if (!fs.existsSync(sourceDir)) { return result; @@ -226,7 +171,7 @@ export async function syncConversations( continue; } - const wasCopied = copyIfNewer(srcFile, destFile); + const wasCopied = await archiveAdmittedConversation(srcFile, destFile, exclusions, redactor, tally); if (wasCopied) { result.copied++; filesToIndex.push(destFile); @@ -240,10 +185,10 @@ export async function syncConversations( if (!options.skipSummaries) { const summaryPath = destFile.replace('.jsonl', '-summary.txt'); if (shouldQueueForSummary(summaryPath) && !shouldSkipConversation(destFile)) { - // sessionId enables Claude session-resume summarization; when the - // filename has no UUID to extract (e.g. subagent transcripts named - // agent-.jsonl), queue anyway — summarizeConversation falls - // back to summarizing from the transcript text. + // sessionId enables source-session summarization. When the filename + // has no UUID (e.g. agent-.jsonl), queue anyway: Codex uses an + // isolated transcript-only thread; other harnesses use their own + // transcript-text route without resuming a source session. const sessionId = extractSessionIdFromPath(destFile) ?? undefined; filesToSummarize.push({ path: destFile, sessionId }); } @@ -257,6 +202,8 @@ export async function syncConversations( } } + result.redactions = tally.toArray(); + // Index copied files (unless skipIndex is set) if (!options.skipIndex && filesToIndex.length > 0) { const { parseConversation } = await import('./parser.js'); @@ -395,7 +342,10 @@ export async function syncConversations( } console.log(` Summarizing ${path.basename(filePath)} (${exchanges.length} exchanges)...`); - const summary = await summarizeConversation(exchanges, sessionId); + // With redaction on, never resume/fork the session: those paths hand + // the model the unredacted source transcript instead of these + // (redacted) exchanges. + const summary = await summarizeConversation(exchanges, sessionId, { allowResume: redactor === null }); const summaryPath = filePath.replace('.jsonl', '-summary.txt'); fs.writeFileSync(summaryPath, summary, 'utf-8'); diff --git a/src/verify.ts b/src/verify.ts index 0dd2c5db..9a904eab 100644 --- a/src/verify.ts +++ b/src/verify.ts @@ -1,22 +1,34 @@ import fs from 'fs'; import path from 'path'; +import Database from 'better-sqlite3'; import { parseConversation } from './parser.js'; -import { initDatabase, getAllExchanges, getFileLastIndexed } from './db.js'; -import { getArchiveDir, getExcludedProjects, findJsonlFiles, statIfExists } from './paths.js'; +import { getAllExchanges } from './db.js'; +import { getArchiveDir, getDbPath, getExcludedProjects, findJsonlFiles, statIfExists } from './paths.js'; import { isErroredSentinel } from './summary-sentinel.js'; +import { readRecordExclusions } from './record-admission.js'; +import { loadRedactor } from './redaction.js'; export interface VerificationResult { missing: Array<{ path: string; reason: string }>; + unindexed: Array<{ path: string }>; orphaned: Array<{ uuid: string; path: string }>; outdated: Array<{ path: string; fileTime: number; dbTime: number }>; + screeningRejected: Array<{ path: string }>; + screeningUnavailable: Array<{ path: string }>; + archiveRefreshes: Array<{ path: string; fileTime: number; dbTime: number }>; corrupted: Array<{ path: string; error: string }>; } export async function verifyIndex(): Promise { + readRecordExclusions(); const result: VerificationResult = { missing: [], + unindexed: [], orphaned: [], outdated: [], + screeningRejected: [], + screeningUnavailable: [], + archiveRefreshes: [], corrupted: [] }; @@ -30,8 +42,13 @@ export async function verifyIndex(): Promise { return result; } - // Initialize database once for all checks - const db = initDatabase(); + // Verification must not create or migrate the index it is measuring. + const dbPath = getDbPath(); + const db = fs.existsSync(dbPath) ? new Database(dbPath, { readonly: true, fileMustExist: true }) : null; + const indexState = db?.prepare(` + SELECT MAX(line_end) AS maxLineEnd, MAX(last_indexed) AS lastIndexed + FROM exchanges WHERE archive_path = ? + `); const projects = fs.readdirSync(archiveDir); const excludedProjects = getExcludedProjects(); @@ -68,34 +85,38 @@ export async function verifyIndex(): Promise { // re-attempts it rather than reporting the conversation as healthy. if (!fs.existsSync(summaryPath)) { result.missing.push({ path: conversationPath, reason: 'No summary file' }); - continue; - } - if (isErroredSentinel(fs.readFileSync(summaryPath, 'utf-8'))) { + } else if (isErroredSentinel(fs.readFileSync(summaryPath, 'utf-8'))) { result.missing.push({ path: conversationPath, reason: 'Previous summarization failed (error sentinel)' }); - continue; } - // Check if file is outdated (modified after last_indexed) - const lastIndexed = getFileLastIndexed(db, conversationPath); - if (lastIndexed !== null) { - const fileStat = fs.statSync(conversationPath); - if (fileStat.mtimeMs > lastIndexed) { - result.outdated.push({ - path: conversationPath, - fileTime: fileStat.mtimeMs, - dbTime: lastIndexed - }); - } - } + const state = indexState?.get(conversationPath) as { maxLineEnd: number | null; lastIndexed: number | null } | undefined; + const maxLineEnd = state?.maxLineEnd ?? null; + if (maxLineEnd === null) result.unindexed.push({ path: conversationPath }); - // Try parsing to detect corruption + // Parse independently of summary state and mtime. Only exchanges past + // the indexed high-water mark are a real index delta. try { - await parseConversation(conversationPath, project, conversationPath); + const exchanges = await parseConversation(conversationPath, project, conversationPath); + if (maxLineEnd !== null) { + const fileTime = fs.statSync(conversationPath).mtimeMs; + const dbTime = state?.lastIndexed ?? 0; + if (exchanges.some(exchange => exchange.lineEnd > maxLineEnd)) { + result.outdated.push({ path: conversationPath, fileTime, dbTime }); + } else if (state?.lastIndexed != null && fileTime > state.lastIndexed) { + result.archiveRefreshes.push({ path: conversationPath, fileTime, dbTime }); + } + } } catch (error) { - result.corrupted.push({ - path: conversationPath, - error: error instanceof Error ? error.message : String(error) - }); + const message = error instanceof Error ? error.message : String(error); + if (message === 'Record content rejected by screening' || + message === 'Record screening byte limit exceeded' || + message === 'Record screening nesting limit exceeded') { + result.screeningRejected.push({ path: conversationPath }); + } else if (message === 'Record screening unavailable') { + result.screeningUnavailable.push({ path: conversationPath }); + } else { + result.corrupted.push({ path: conversationPath, error: message }); + } } } } @@ -103,11 +124,11 @@ export async function verifyIndex(): Promise { console.log(`Verified ${totalChecked} conversations.`); // Check for orphaned database entries - const dbExchanges = getAllExchanges(db); - db.close(); + const dbExchanges = db ? getAllExchanges(db) : []; + db?.close(); for (const exchange of dbExchanges) { - if (!foundFiles.has(exchange.archivePath)) { + if (!foundFiles.has(exchange.archivePath) && !statIfExists(exchange.archivePath)?.isFile()) { result.orphaned.push({ uuid: exchange.id, path: exchange.archivePath @@ -119,8 +140,16 @@ export async function verifyIndex(): Promise { } export async function repairIndex(issues: VerificationResult): Promise { + if (issues.screeningRejected.length + issues.screeningUnavailable.length > 0) { + throw new Error('Resolve record screening failures before repair'); + } + readRecordExclusions(); console.log('Repairing index...'); + // Load before touching the index: strict mode fails closed here, as in sync + // and index. + const redactor = loadRedactor(); + // To avoid circular dependencies, we import the indexer functions dynamically const { initDatabase, insertExchange, deleteExchange } = await import('./db.js'); const { parseConversation } = await import('./parser.js'); @@ -137,10 +166,10 @@ export async function repairIndex(issues: VerificationResult): Promise { } // Re-index missing and outdated conversations - const toReindex = [ + const toReindex = [...new Set([ ...issues.missing.map(m => m.path), ...issues.outdated.map(o => o.path) - ]; + ])]; for (const conversationPath of toReindex) { console.log(`Re-indexing: ${conversationPath}`); @@ -160,7 +189,8 @@ export async function repairIndex(issues: VerificationResult): Promise { // Generate/update summary const summaryPath = conversationPath.replace('.jsonl', '-summary.txt'); - const summary = await summarizeConversation(exchanges); + // Under redaction, don't let the Codex fork fallback read the source rollout. + const summary = await summarizeConversation(exchanges, undefined, { allowResume: redactor === null }); fs.writeFileSync(summaryPath, summary, 'utf-8'); console.log(` Created summary: ${summary.split(/\s+/).length} words`); @@ -178,6 +208,8 @@ export async function repairIndex(issues: VerificationResult): Promise { console.log(` Indexed ${exchanges.length} exchanges`); } catch (error) { console.error(`Failed to re-index ${conversationPath}:`, error); + db.close(); + throw error; } } diff --git a/test/codex-no-fallback.test.ts b/test/codex-no-fallback.test.ts new file mode 100644 index 00000000..f4973852 --- /dev/null +++ b/test/codex-no-fallback.test.ts @@ -0,0 +1,63 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import type { ConversationExchange } from '../src/types.js'; + +vi.mock('@anthropic-ai/claude-agent-sdk', () => ({ query: vi.fn() })); +import { query } from '@anthropic-ai/claude-agent-sdk'; +import { summarizeConversation } from '../src/summarizer.js'; + +describe('Codex summary provider boundary', () => { + let directory: string; + let previous: NodeJS.ProcessEnv; + + beforeEach(() => { + previous = { ...process.env }; + directory = fs.mkdtempSync(path.join(os.tmpdir(), 'em-codex-no-fallback-')); + process.env.EPISODIC_MEMORY_CONFIG_DIR = directory; + process.env.EPISODIC_MEMORY_CODEX_BIN = path.join(directory, 'missing-codex'); + vi.mocked(query).mockReset(); + }); + + afterEach(() => { + process.env = previous; + fs.rmSync(directory, { recursive: true, force: true }); + }); + + it('does not call Claude when Codex app-server is unavailable', async () => { + vi.mocked(query).mockReturnValue({ + async *[Symbol.asyncIterator]() { + yield { type: 'result', is_error: false, result: 'Unexpected Claude fallback.' }; + }, + } as any); + const exchange: ConversationExchange = { + id: 'codex-1', project: 'fixture', timestamp: '2026-01-01T00:00:00Z', + userMessage: 'Explain the design decisions behind the memory index and the safe summary route in this project.', + assistantMessage: 'The index uses line high-water marks and summaries are generated only after provider admission.', + archivePath: path.join(directory, 'session.jsonl'), lineStart: 1, lineEnd: 2, + harness: 'codex', sessionId: '00000000-1111-4222-8333-444444444444', + }; + + await expect(summarizeConversation([exchange], exchange.sessionId)).rejects.toThrow(); + expect(query).not.toHaveBeenCalled(); + }); + + it('keeps a Codex transcript without a session ID on the Codex route', async () => { + vi.mocked(query).mockReturnValue({ + async *[Symbol.asyncIterator]() { + yield { type: 'result', is_error: false, result: 'Unexpected Claude fallback.' }; + }, + } as any); + const exchange: ConversationExchange = { + id: 'codex-no-id', project: 'fixture', timestamp: '2026-01-01T00:00:00Z', + userMessage: 'Explain the design decisions behind the memory index and the safe summary route in this project.', + assistantMessage: 'The index uses line high-water marks and summaries are generated only after provider admission.', + archivePath: path.join(directory, 'session.jsonl'), lineStart: 1, lineEnd: 2, + harness: 'codex', + }; + + await expect(summarizeConversation([exchange])).rejects.toThrow(); + expect(query).not.toHaveBeenCalled(); + }); +}); diff --git a/test/codex-summary-routing.test.ts b/test/codex-summary-routing.test.ts new file mode 100644 index 00000000..130cab3f --- /dev/null +++ b/test/codex-summary-routing.test.ts @@ -0,0 +1,212 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import type { ConversationExchange } from '../src/types.js'; + +vi.mock('@anthropic-ai/claude-agent-sdk', () => ({ query: vi.fn() })); + +import { query } from '@anthropic-ai/claude-agent-sdk'; +import { summarizeConversation } from '../src/summarizer.js'; + +let testDir: string; +let previousEnv: NodeJS.ProcessEnv; + +function exchange(sessionId?: string): ConversationExchange { + return { + id: 'codex-summary-fixture', + project: 'fixture', + timestamp: '2026-01-01T00:00:00Z', + userMessage: 'Explain the design and acceptance boundary for this synthetic Codex conversation.', + assistantMessage: 'The design keeps provider ownership and validates the route with an isolated local app-server fixture.', + archivePath: path.join(testDir, 'synthetic.jsonl'), + lineStart: 1, + lineEnd: 2, + harness: 'codex', + ...(sessionId ? { sessionId } : {}), + }; +} + +function installFakeCodex( + failMethod?: 'thread/start' | 'thread/fork', + options: { ignoreSigterm?: boolean; stalledTurn?: boolean } = {}, +): void { + const executable = path.join(testDir, 'fake-codex'); + fs.writeFileSync(executable, `#!/usr/bin/env node +const readline = require('node:readline'); +if (process.argv.includes('--version')) { + console.log('codex-cli 0.154.0'); + process.exit(0); +} +if (!process.argv.includes('app-server')) process.exit(2); +if (${options.ignoreSigterm === true}) { + const fs = require('node:fs'); + fs.writeFileSync(${JSON.stringify(path.join(testDir, 'summary-process.json'))}, JSON.stringify({ pid: process.pid, cwd: process.cwd() })); + process.on('SIGTERM', () => { + fs.writeFileSync(${JSON.stringify(path.join(testDir, 'termination-requested'))}, 'SIGTERM'); + }); +} +const emit = value => process.stdout.write(JSON.stringify(value) + '\\n'); +readline.createInterface({ input: process.stdin }).on('line', line => { + const message = JSON.parse(line); + if (message.method === 'initialize') emit({ id: message.id, result: {} }); + if (message.method === 'config/read') emit({ id: message.id, result: { config: { mcp_servers: {} } } }); + if (message.method === 'skills/list') emit({ id: message.id, result: { data: [{ skills: [], errors: [] }] } }); + if (message.method === 'thread/start') { + if (${failMethod === 'thread/start'}) { + if (${options.ignoreSigterm === true}) { + emit({ id: message.id, error: { message: 'EXCLUDED_PUBLIC_FIXTURE' } }); + return; + } + process.exit(7); + } + const params = message.params; + if (!params.ephemeral || params.threadId || params.sandbox !== 'read-only' || + params.allowProviderModelFallback !== false || !params.cwd) process.exit(8); + emit({ id: message.id, result: { thread: { id: 'ephemeral-summary' } } }); + } + if (message.method === 'thread/fork') { + if (${failMethod === 'thread/fork'}) process.exit(7); + if (message.params.threadId !== 'source-session' || !message.params.ephemeral) process.exit(9); + emit({ id: message.id, result: { thread: { id: 'forked-summary' } } }); + } + if (message.method === 'turn/start') { + if (${options.stalledTurn === true}) return; + const prompt = message.params.input[0].text; + if (message.params.threadId === 'ephemeral-summary' && + (!prompt.includes('synthetic Codex conversation') || !prompt.includes('isolated local app-server fixture'))) process.exit(10); + emit({ id: message.id, result: { turn: { id: 'summary-turn' } } }); + emit({ method: 'item/completed', params: { item: { type: 'agentMessage', text: 'Codex kept provider ownership.' } } }); + emit({ method: 'turn/completed', params: { turn: { id: 'summary-turn', status: 'completed' } } }); + } +}); +`); + fs.chmodSync(executable, 0o700); + process.env.EPISODIC_MEMORY_CODEX_BIN = executable; +} + +beforeEach(() => { + previousEnv = { ...process.env }; + testDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codex-summary-routing-')); + process.env.EPISODIC_MEMORY_CONFIG_DIR = path.join(testDir, 'config'); + vi.mocked(query).mockReset(); + vi.mocked(query).mockReturnValue({ + async *[Symbol.asyncIterator]() { + yield { type: 'result', is_error: false, result: 'Wrong provider.' }; + }, + } as any); +}); + +afterEach(() => { + process.env = previousEnv; + fs.rmSync(testDir, { recursive: true, force: true }); +}); + +describe('Codex summary provider ownership', () => { + it.skipIf(process.platform === 'win32').each(['completed', 'failed', 'timed out'] as const)( + 'terminates a SIGTERM-resistant isolated summary after the turn %s', + async status => { + installFakeCodex(status === 'failed' ? 'thread/start' : undefined, { + ignoreSigterm: true, + stalledTurn: status === 'timed out', + }); + if (status === 'timed out') process.env.EPISODIC_MEMORY_CODEX_SUMMARY_TIMEOUT_MS = '256'; + let settled = false; + const outcome = summarizeConversation([exchange()]).then( + result => { settled = true; return { status: 'resolved' as const, result }; }, + error => { settled = true; return { status: 'rejected' as const, error }; }, + ); + let deadline: NodeJS.Timeout | undefined; + const expired = new Promise<{ status: 'expired' }>(resolve => { + deadline = setTimeout(() => resolve({ status: 'expired' }), 5000); + }); + + try { + const result = await Promise.race([outcome, expired]); + expect(result.status).toBe(status === 'completed' ? 'resolved' : 'rejected'); + if (result.status === 'resolved') { + expect(result.result).toBe('Codex kept provider ownership.'); + } else if (result.status === 'rejected') { + expect(result.error.message).toBe('Isolated Codex summary failed; provider details suppressed'); + } + const child = JSON.parse(fs.readFileSync(path.join(testDir, 'summary-process.json'), 'utf8')); + expect(fs.existsSync(path.join(testDir, 'termination-requested'))).toBe(true); + expect(fs.existsSync(child.cwd)).toBe(false); + expect(() => process.kill(child.pid, 0)).toThrow(); + expect(query).not.toHaveBeenCalled(); + } finally { + if (deadline) clearTimeout(deadline); + const marker = path.join(testDir, 'summary-process.json'); + if (!settled && fs.existsSync(marker)) { + const child = JSON.parse(fs.readFileSync(marker, 'utf8')); + try { process.kill(child.pid, 'SIGKILL'); } catch {} + await outcome; + fs.rmSync(child.cwd, { recursive: true, force: true }); + } + } + }, + ); + + it('summarizes a sessionless Codex transcript in an isolated ephemeral Codex thread', async () => { + installFakeCodex(); + + expect(await summarizeConversation([exchange()])).toBe('Codex kept provider ownership.'); + expect(query).not.toHaveBeenCalled(); + }); + + it('does not borrow another harness session ID for a sessionless Codex exchange', async () => { + installFakeCodex(); + const otherHarness: ConversationExchange = { + ...exchange('other-provider-session'), + id: 'other-provider-exchange', + harness: 'claude', + }; + + expect(await summarizeConversation([otherHarness, exchange()])) + .toBe('Codex kept provider ownership.'); + expect(query).not.toHaveBeenCalled(); + }); + + it('does not fall back to Claude when a sessionless Codex summary fails', async () => { + installFakeCodex('thread/start'); + + await expect(summarizeConversation([exchange()])) + .rejects.toThrow('Isolated Codex summary failed; provider details suppressed'); + expect(query).not.toHaveBeenCalled(); + }); + + it('keeps the ephemeral fork for a Codex transcript with a session ID', async () => { + installFakeCodex(); + + expect(await summarizeConversation([exchange('source-session')])) + .toBe('Codex kept provider ownership.'); + expect(query).not.toHaveBeenCalled(); + }); + + it('uses only admitted Codex text when source-session resume is disabled', async () => { + installFakeCodex('thread/fork'); + + expect(await summarizeConversation([exchange('source-session')], 'source-session', { allowResume: false })) + .toBe('Codex kept provider ownership.'); + expect(query).not.toHaveBeenCalled(); + }); + + it('uses only transcript text when Claude source-session resume is disabled', async () => { + const transcript = { ...exchange('source-session'), harness: 'claude' as const }; + + expect(await summarizeConversation([transcript], 'source-session', { allowResume: false })) + .toBe('Wrong provider.'); + const [request] = vi.mocked(query).mock.calls[0]; + expect(request.options?.resume).toBeUndefined(); + expect(request.prompt).toContain(transcript.userMessage); + expect(request.prompt).toContain(transcript.assistantMessage); + }); + + it('does not fall back to Claude when a Codex fork fails', async () => { + installFakeCodex('thread/fork'); + + await expect(summarizeConversation([exchange('source-session')])) + .rejects.toThrow(); + expect(query).not.toHaveBeenCalled(); + }); +}); diff --git a/test/fake-secrets.ts b/test/fake-secrets.ts new file mode 100644 index 00000000..0eebca9e --- /dev/null +++ b/test/fake-secrets.ts @@ -0,0 +1,151 @@ +/** + * Fake-secret builders for redaction tests. + * + * Nothing in this file is a real credential, and no string literal here matches + * a redaction rule on its own: every secret is assembled at runtime from a + * seeded PRNG plus split prefixes. That keeps GitHub push protection and other + * scanners quiet while still producing values in the exact real-world shape. + */ + +const UPPER = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'; +const LOWER = 'abcdefghijklmnopqrstuvwxyz'; +const DIGITS = '0123456789'; +const ALNUM = UPPER + LOWER + DIGITS; +const B64 = ALNUM + '+/'; +const B64URL = ALNUM + '-_'; +const BASE32 = UPPER + '234567'; + +/** Deterministic PRNG (mulberry32) so failures reproduce. */ +export function makeRng(seed: number): () => number { + let a = seed >>> 0; + return () => { + a = (a + 0x6d2b79f5) >>> 0; + let t = a; + t = Math.imul(t ^ (t >>> 15), t | 1); + t ^= t + Math.imul(t ^ (t >>> 7), t | 61); + return ((t ^ (t >>> 14)) >>> 0) / 4294967296; + }; +} + +export class FakeSecrets { + private rng: () => number; + + constructor(seed = 1337) { + this.rng = makeRng(seed); + } + + chars(charset: string, n: number): string { + let out = ''; + for (let i = 0; i < n; i++) out += charset[Math.floor(this.rng() * charset.length)]; + return out; + } + + /** Always contains at least one digit and one letter. */ + private mixed(charset: string, n: number): string { + return this.chars(charset, n - 2) + this.chars(DIGITS, 1) + this.chars(LOWER, 1); + } + + azureClientSecret(): string { + // <3 chars>Q~<34 chars> — Entra ID client secret shape. + return this.chars(ALNUM, 3) + this.chars(DIGITS, 1) + 'Q' + '~' + this.chars(ALNUM + '_~.-', 33) + 'x'; + } + + azureStorageKey(): string { + // 64 random bytes, base64: 86 chars + '=='. + return this.chars(ALNUM, 1) + this.chars(B64, 84) + this.chars(ALNUM, 1) + '=' + '='; + } + + sasSignature(): string { + return this.chars(ALNUM, 40) + '%2B' + this.chars(ALNUM, 3) + '%3D'; + } + + privateKeyBlock(kind = 'RSA'): string { + const body: string[] = []; + for (let i = 0; i < 6; i++) body.push(this.chars(B64, 64)); + const dashes = '-'.repeat(5); + return `${dashes}BEGIN ${kind} PRIVATE` + ` KEY${dashes}\n${body.join('\n')}\n${dashes}END ${kind} PRIVATE` + ` KEY${dashes}`; + } + + jwt(): string { + const head = 'ey' + 'J' + this.chars(B64URL, 30); + const payload = 'ey' + 'J' + this.chars(B64URL, 60); + return `${head}.${payload}.${this.chars(B64URL, 43)}`; + } + + anthropicKey(): string { + return 'sk-' + 'ant-' + 'api03-' + this.chars(B64URL, 93) + 'AA'; + } + + openAiKey(): string { + return 'sk-' + 'proj-' + this.chars(ALNUM, 20) + 'T3Blbk' + 'FJ' + this.chars(ALNUM, 20); + } + + githubToken(): string { + return 'gh' + 'p_' + this.chars(ALNUM, 36); + } + + githubFineGrainedPat(): string { + return 'github' + '_pat_' + this.chars(ALNUM, 22) + '_' + this.chars(ALNUM, 59); + } + + awsAccessKeyId(): string { + return 'AK' + 'IA' + this.chars(BASE32, 16); + } + + awsSecretAccessKey(): string { + return this.mixed(ALNUM + '/+', 40); + } + + slackToken(): string { + return 'xo' + 'xb-' + this.chars(DIGITS, 12) + '-' + this.chars(DIGITS, 12) + '-' + this.chars(ALNUM, 24); + } + + googleApiKey(): string { + return 'AI' + 'za' + this.chars(ALNUM + '_-', 35); + } + + npmToken(): string { + return 'np' + 'm_' + this.chars(ALNUM, 36); + } + + /** Starts with a letter so it never looks like a `$VAR`/`%VAR%` placeholder. */ + password(): string { + return this.chars(LOWER, 1) + this.mixed(ALNUM + '!#%^*', 15); + } + + /** Letters only: a real password with no digit, which digit-gated rules miss. */ + passphrase(): string { + return this.chars(UPPER, 1) + this.chars(LOWER, 7) + this.chars(UPPER, 1) + this.chars(LOWER, 9); + } + + bearerOpaque(): string { + return this.mixed(ALNUM, 40); + } + + basicAuth(): string { + return this.chars(ALNUM, 30) + '=' + '='; + } + + /** High-entropy blob with no recognizable prefix. */ + opaqueHighEntropy(n = 48): string { + return this.mixed(ALNUM + '+/', n); + } + + gitSha(): string { + return this.chars('0123456789abcdef', 40); + } + + guid(): string { + const h = (n: number) => this.chars('0123456789abcdef', n); + return `${h(8)}-${h(4)}-${h(4)}-${h(4)}-${h(12)}`; + } + + base64Blob(n: number): string { + return this.chars(B64, n); + } +} + +/** Every `[REDACTED:...]` token in a string. */ +export function redactionTokens(text: string): string[] { + return text.match(/\[REDACTED:[a-z0-9-]+\]/g) ?? []; +} diff --git a/test/index-cli-single-instance.test.ts b/test/index-cli-single-instance.test.ts index d41203a9..c9c9f23a 100644 --- a/test/index-cli-single-instance.test.ts +++ b/test/index-cli-single-instance.test.ts @@ -1,6 +1,6 @@ import { describe, it, expect, beforeEach, afterEach } from 'vitest'; import { spawnSync } from 'child_process'; -import { mkdtempSync, mkdirSync, rmSync, existsSync } from 'fs'; +import { mkdtempSync, mkdirSync, rmSync, existsSync, writeFileSync } from 'fs'; import { fileURLToPath } from 'url'; import { dirname, join } from 'path'; import { tmpdir } from 'os'; @@ -75,18 +75,39 @@ describe('index-cli single-instance lock (#97)', () => { expect(existsSync(join(testDir, 'test.db'))).toBe(false); }); - it('skips `verify`, which can initialize and migrate the database', () => { + it('reports a lock-blocked `verify` as incomplete rather than healthy', () => { heldLock = acquireFileLock(lockPath); expect(heldLock).not.toBeNull(); const result = runIndexCli(['verify']); - expect(result.status).toBe(0); + expect(result.status).toBe(2); expect(result.stderr).toMatch(/sync already running.*skipping/); expect(result.stdout).not.toMatch(/Verifying conversation index/); expect(existsSync(join(testDir, 'test.db'))).toBe(false); }); + it('reports unindexed archives as actionable without calling them outdated', () => { + const projectDir = join(testDir, 'archive', 'project'); + mkdirSync(projectDir, { recursive: true }); + const archivePath = join(projectDir, 'session.jsonl'); + writeFileSync(archivePath, [ + JSON.stringify({ type: 'user', message: { role: 'user', content: 'Hello' }, timestamp: '2024-01-01T00:00:00Z' }), + JSON.stringify({ type: 'assistant', message: { role: 'assistant', content: 'Hi' }, timestamp: '2024-01-01T00:00:01Z' }), + ].join('\n')); + writeFileSync(archivePath.replace('.jsonl', '-summary.txt'), 'Existing summary'); + + const result = runIndexCli(['verify']); + expect(result.status).toBe(1); + expect(result.stdout).toContain('Unindexed archives: 1'); + expect(result.stdout).toContain(archivePath); + expect(result.stdout).toContain('Outdated files: 0'); + + const repair = runIndexCli(['repair']); + expect(repair.status).toBe(1); + expect(repair.stdout).toContain('unindexed archive(s) require separate review'); + }); + it('releases the lock after an uncontended run', () => { const result = runIndexCli(['verify']); diff --git a/test/mcp-server.test.ts b/test/mcp-server.test.ts index 47382c96..c70b565c 100644 --- a/test/mcp-server.test.ts +++ b/test/mcp-server.test.ts @@ -1,7 +1,7 @@ import { afterAll, beforeAll, describe, expect, it } from 'vitest'; import { Client } from '@modelcontextprotocol/sdk/client/index.js'; import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js'; -import { existsSync, mkdirSync, mkdtempSync, rmSync } from 'node:fs'; +import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync, readFileSync, unlinkSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { fileURLToPath } from 'node:url'; @@ -87,6 +87,47 @@ describe('MCP search tool', () => { } }); + it('enforces changed exclusions through the packaged MCP search entrypoint', async () => { + const { initDatabase, insertExchange } = await import('../src/db.js'); + const priorDb = process.env.EPISODIC_MEMORY_DB_PATH; + const priorTestDb = process.env.TEST_DB_PATH; + const priorConfig = process.env.EPISODIC_MEMORY_CONFIG_DIR; + const session = '00000000-1111-4222-8333-444444444444'; + const file = join(testDir!, `packaged-${session}.jsonl`); + const policy = join(testDir!, 'config', 'record-exclusions.json'); + mkdirSync(join(testDir!, 'config'), { recursive: true }); + const original = [ + { type: 'session_meta', payload: { id: session } }, + { type: 'response_item', payload: { type: 'message', role: 'user', content: [{ type: 'input_text', text: 'packaged-admission-probe' }] } }, + { type: 'response_item', payload: { type: 'message', role: 'assistant', content: [{ type: 'output_text', text: 'safe answer' }] } }, + ].map(row => JSON.stringify(row)).join('\n'); + writeFileSync(file, original); + try { + delete process.env.EPISODIC_MEMORY_DB_PATH; + process.env.TEST_DB_PATH = testDbPath; + process.env.EPISODIC_MEMORY_CONFIG_DIR = join(testDir!, 'config'); + const db = initDatabase(); + insertExchange(db, { id: 'packaged-admission', project: 'fixture', timestamp: '2026-01-01T00:00:00Z', + userMessage: 'packaged-admission-probe', assistantMessage: 'safe answer', archivePath: file, + lineStart: 2, lineEnd: 3, sessionId: session }, new Array(384).fill(0.1)); + db.close(); + const search = () => client.callTool({ name: 'search', arguments: { + query: 'packaged-admission-probe', mode: 'text', response_format: 'json', limit: 1, + } }); + expect(JSON.parse(getTextContent((await search()).content as ToolContent[])).count).toBe(1); + writeFileSync(policy, JSON.stringify({ version: 1, exchanges: [{ session_id: session, + transcript: `packaged-${session}.jsonl`, line_start: 2, line_end: 3 }], tool_calls: [] })); + const denied = await search(); expect(denied.isError).toBeFalsy(); + expect(JSON.parse(getTextContent(denied.content as ToolContent[])).count).toBe(0); + expect(readFileSync(file, 'utf8')).toBe(original); + } finally { + if (priorDb === undefined) delete process.env.EPISODIC_MEMORY_DB_PATH; else process.env.EPISODIC_MEMORY_DB_PATH = priorDb; + if (priorTestDb === undefined) delete process.env.TEST_DB_PATH; else process.env.TEST_DB_PATH = priorTestDb; + if (priorConfig === undefined) delete process.env.EPISODIC_MEMORY_CONFIG_DIR; else process.env.EPISODIC_MEMORY_CONFIG_DIR = priorConfig; + if (existsSync(policy)) unlinkSync(policy); + } + }); + it('advertises search and read tools with single and multi-concept queries', async () => { const tools = await client.listTools(); const searchTool = tools.tools.find((tool) => tool.name === 'search'); @@ -121,4 +162,34 @@ describe('MCP search tool', () => { }); expect(existsSync(testDbPath)).toBe(true); }); + + it('applies live record exclusions to direct reads and keeps physical range coordinates', async () => { + const session = '00000000-1111-4222-8333-444444444444'; + const file = join(testDir!, `rollout-${session}.jsonl`); + const config = join(testDir!, 'config', 'record-exclusions.json'); + mkdirSync(join(testDir!, 'config'), { recursive: true }); + const rows = [ + { type: 'session_meta', payload: { id: session } }, + ...['safe first', 'EXCLUDED_PUBLIC_FIXTURE', 'safe last'].map(text => ({ type: 'response_item', + payload: { type: 'message', role: 'user', content: [{ type: 'text', text }] } })), + ]; + const original = rows.map(row => JSON.stringify(row)).join('\n') + '\n'; + writeFileSync(file, original); + writeFileSync(config, JSON.stringify({ version: 1, + exchanges: [{ session_id: session, transcript: `rollout-${session}.jsonl`, line_start: 3, line_end: 3 }], tool_calls: [] })); + try { + const result = await client.callTool({ name: 'read', arguments: { path: file } }); + expect(result.isError).toBeFalsy(); + const text = getTextContent(result.content as ToolContent[]); + expect(text).not.toContain('EXCLUDED_PUBLIC_FIXTURE'); + expect(text).toContain('safe first'); + expect(text).toContain('safe last'); + const ranged = await client.callTool({ name: 'read', arguments: { path: file, startLine: 4, endLine: 4 } }); + expect(ranged.isError).toBeFalsy(); + const rangedText = getTextContent(ranged.content as ToolContent[]); + expect(rangedText).toContain('safe last'); + expect(rangedText).not.toContain('safe first'); + expect(readFileSync(file, 'utf8')).toBe(original); + } finally { unlinkSync(config); unlinkSync(file); } + }); }); diff --git a/test/record-admission.test.ts b/test/record-admission.test.ts new file mode 100644 index 00000000..761591d8 --- /dev/null +++ b/test/record-admission.test.ts @@ -0,0 +1,488 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { syncConversations } from '../src/sync.js'; +import { parseConversation } from '../src/parser.js'; +import { initDatabase, insertExchange } from '../src/db.js'; +import { indexConversations, indexSession, indexUnprocessed } from '../src/indexer.js'; +import { summarizeConversation, runCodexCommand } from '../src/summarizer.js'; +import { query } from '@anthropic-ai/claude-agent-sdk'; +import { admittedConversationLines, archiveAdmittedConversation } from '../src/record-admission.js'; + +vi.mock('../src/embeddings.js', () => ({ + initEmbeddings: vi.fn(async () => {}), + generateExchangeEmbedding: vi.fn(async () => new Array(384).fill(0)), +})); +vi.mock('@anthropic-ai/claude-agent-sdk', () => ({ query: vi.fn() })); +import { createRedactor, loadRedactionConfig } from '../src/redaction.js'; +import type { ConversationExchange } from '../src/types.js'; + +const SESSION = '00000000-1111-4222-8333-444444444444'; +const PRIVATE = 'public-fixture-excluded-content'; +let root: string; +let source: string; +let archive: string; +let config: string; +let sourceFile: string; +let archiveFile: string; +let previous: NodeJS.ProcessEnv; + +function message(role: string, text: string): object { + return { type: 'response_item', timestamp: '2026-01-01T00:00:00Z', + payload: { type: 'message', role, content: [{ type: 'text', text }] } }; +} + +function transcript(): string { + return [ + { type: 'session_meta', payload: { id: SESSION, cwd: '/synthetic/project' } }, + message('user', 'safe question one'), + message('assistant', 'safe answer one'), + message('user', PRIVATE), + message('assistant', PRIVATE), + message('user', 'safe question two'), + { type: 'response_item', payload: { type: 'function_call', call_id: 'call-denied', name: 'fixture', arguments: JSON.stringify({ value: PRIVATE }) } }, + { type: 'response_item', payload: { type: 'function_call_output', call_id: 'call-denied', output: PRIVATE } }, + message('assistant', 'safe answer two'), + ].map(value => JSON.stringify(value)).join('\n') + '\n'; +} + +function policy(value: object = { + version: 1, + exchanges: [{ session_id: SESSION, transcript: `rollout-${SESSION}.jsonl`, line_start: 4, line_end: 5 }], + tool_calls: [{ session_id: SESSION, transcript: `rollout-${SESSION}.jsonl`, call_id: 'call-denied' }], +}): void { + fs.mkdirSync(config, { recursive: true }); + fs.writeFileSync(path.join(config, 'record-exclusions.json'), JSON.stringify(value)); +} + +beforeEach(() => { + previous = { ...process.env }; + root = fs.mkdtempSync(path.join(os.tmpdir(), 'record-admission-')); + source = path.join(root, 'source'); + archive = path.join(root, 'archive'); + config = path.join(root, 'config'); + process.env.EPISODIC_MEMORY_CONFIG_DIR = config; + process.env.EPISODIC_MEMORY_DB_PATH = path.join(root, 'index.sqlite'); + delete process.env.CONVERSATION_SEARCH_EXCLUDE_PROJECTS; + process.env.TEST_PROJECTS_DIR = source; + process.env.TEST_ARCHIVE_DIR = archive; + fs.mkdirSync(path.join(source, '2026'), { recursive: true }); + sourceFile = path.join(source, '2026', `rollout-${SESSION}.jsonl`); + archiveFile = path.join(archive, '2026', `rollout-${SESSION}.jsonl`); + fs.writeFileSync(sourceFile, transcript()); +}); + +afterEach(() => { + process.env = previous; + fs.rmSync(root, { recursive: true, force: true }); + expect(fs.existsSync(root)).toBe(false); + vi.restoreAllMocks(); +}); + +describe('persistent record exclusion at native ingestion boundaries', () => { + it('redacts admitted records after exclusions without changing source coordinates', async () => { + policy(); + const original = transcript().replace('safe question one', 'safe question one fixture_secret_1234'); + fs.writeFileSync(sourceFile, original.replace(/\n/g, '\r\n')); + const before = fs.readFileSync(sourceFile); + const redactor = createRedactor(loadRedactionConfig({ includeDefaults: false, + rules: [{ id: 'fixture-secret', pattern: 'fixture_secret_[0-9]{4}' }] })); + + const result = await syncConversations(source, archive, { skipIndex: true, skipSummaries: true, redactor }); + + expect(result.errors).toEqual([]); + expect(result.redactions).toEqual([{ ruleId: 'fixture-secret', count: 1 }]); + const copied = fs.readFileSync(archiveFile, 'utf8'); + expect(copied).not.toContain(PRIVATE); + expect(copied).not.toContain('fixture_secret_1234'); + expect(copied).toContain('[REDACTED:fixture-secret]'); + expect(copied.split('\r\n')).toHaveLength(original.split('\n').length); + const exchanges = await parseConversation(archiveFile, '2026', archiveFile); + expect(exchanges.map(exchange => exchange.lineStart)).toEqual([2, 6]); + expect(fs.readFileSync(sourceFile)).toEqual(before); + }); + + it('reapplies changed redaction rules when source mtime stays unchanged', async () => { + const original = transcript().replace('safe question one', 'safe question one fixture_secret_1234'); + fs.writeFileSync(sourceFile, original); + const redactor = createRedactor(loadRedactionConfig({ includeDefaults: false, + rules: [{ id: 'fixture-secret', pattern: 'fixture_secret_[0-9]{4}' }] })); + await syncConversations(source, archive, { skipIndex: true, skipSummaries: true, redactor: null }); + const before = fs.statSync(sourceFile).mtimeMs; + + const result = await syncConversations(source, archive, { skipIndex: true, skipSummaries: true, redactor }); + + expect(result.copied).toBe(1); + expect(fs.readFileSync(archiveFile, 'utf8')).not.toContain('fixture_secret_1234'); + expect(fs.statSync(sourceFile).mtimeMs).toBe(before); + expect(fs.readFileSync(sourceFile, 'utf8')).toBe(original); + }); + + it('does not apply one transcript range to a sibling that shares its session', async () => { + policy({ version: 1, exchanges: [{ session_id: SESSION, transcript: `rollout-${SESSION}.jsonl`, line_start: 4, line_end: 5 }], tool_calls: [] }); + const sibling = path.join(source, '2026', 'agent-sibling.jsonl'); + fs.writeFileSync(sibling, transcript()); + const result = await syncConversations(source, archive, { skipIndex: true, skipSummaries: true }); + expect(result.errors).toEqual([]); + expect(fs.readFileSync(path.join(archive, '2026', 'agent-sibling.jsonl'), 'utf8')).toBe(transcript()); + const admitted = await parseConversation(archiveFile, '2026', archiveFile); + expect(admitted.map(x => x.userMessage)).toEqual(['safe question one', 'safe question two']); + }); + + it('keeps the first transcript identity when fork history contains ancestor headers', async () => { + policy({ version: 1, exchanges: [{ session_id: SESSION, transcript: `rollout-${SESSION}.jsonl`, line_start: 5, line_end: 6 }], tool_calls: [] }); + const rows = transcript().trimEnd().split('\n'); + rows.splice(1, 0, JSON.stringify({ type: 'session_meta', payload: { id: '00000000-aaaa-4bbb-8ccc-dddddddddddd' } })); + fs.writeFileSync(sourceFile, rows.join('\n') + '\n'); + const result = await syncConversations(source, archive, { skipIndex: true, skipSummaries: true }); + expect(result.errors).toEqual([]); + const exchanges = await parseConversation(archiveFile, '2026', archiveFile); + expect(exchanges.map(x => x.userMessage)).toEqual(['safe question one', 'safe question two']); + expect(exchanges.every(x => x.sessionId === SESSION)).toBe(true); + }); + + it('retains safe Claude text in a block shared with an excluded native tool call', async () => { + policy({ version: 1, exchanges: [], tool_calls: [{ session_id: SESSION, transcript: `rollout-${SESSION}.jsonl`, call_id: 'toolu-denied' }] }); + const lines = [ + { type: 'user', sessionId: SESSION, message: { role: 'user', content: 'safe Claude question' } }, + { type: 'assistant', sessionId: SESSION, message: { role: 'assistant', content: [ + { type: 'text', text: 'safe Claude reply' }, + { type: 'tool_use', id: 'toolu-denied', name: 'fixture', input: { value: PRIVATE } }, + { type: 'tool_use', id: 'toolu-allowed', name: 'fixture', input: { value: 'safe input' } }, + ] } }, + ].map(JSON.stringify).join('\n') + '\n'; + fs.writeFileSync(sourceFile, lines); + await syncConversations(source, archive, { skipIndex: true, skipSummaries: true }); + const copied = fs.readFileSync(archiveFile, 'utf8'); + expect(copied).not.toContain(PRIVATE); + expect(copied).toContain('safe Claude reply'); + expect(copied).toContain('toolu-allowed'); + const exchanges = await parseConversation(sourceFile, 'fixture', archiveFile); + expect(exchanges).toHaveLength(1); + expect(exchanges[0].toolCalls).toHaveLength(1); + expect(exchanges[0].assistantMessage).toBe('safe Claude reply'); + expect(fs.readFileSync(sourceFile, 'utf8')).toBe(lines); + }); + + it('filters only denied records before copying and preserves physical line coordinates', async () => { + policy(); + const before = fs.readFileSync(sourceFile); + const result = await syncConversations(source, archive, { skipIndex: true, skipSummaries: true }); + expect(result.errors).toEqual([]); + expect(fs.readFileSync(sourceFile)).toEqual(before); + const copied = fs.readFileSync(archiveFile, 'utf8'); + expect(copied).not.toContain(PRIVATE); + expect(copied.split('\n')).toHaveLength(transcript().split('\n').length); + const exchanges = await parseConversation(archiveFile, '2026', archiveFile); + expect(exchanges.map(x => x.userMessage)).toEqual(['safe question one', 'safe question two']); + expect(exchanges.map(x => x.lineStart)).toEqual([2, 6]); + expect(exchanges.flatMap(x => x.toolCalls ?? [])).toEqual([]); + }); + + it('applies exclusions when directly parsing original input without rewriting it', async () => { + policy(); + const before = fs.readFileSync(sourceFile); + const exchanges = await parseConversation(sourceFile, '2026', archiveFile); + expect(exchanges.map(x => x.userMessage)).toEqual(['safe question one', 'safe question two']); + expect(JSON.stringify(exchanges)).not.toContain(PRIVATE); + expect(fs.readFileSync(sourceFile)).toEqual(before); + }); + + it('reapplies a changed policy even when source mtime did not advance', async () => { + await syncConversations(source, archive, { skipIndex: true, skipSummaries: true }); + expect(fs.readFileSync(archiveFile, 'utf8')).toContain(PRIVATE); + policy(); + await syncConversations(source, archive, { skipIndex: true, skipSummaries: true }); + expect(fs.readFileSync(archiveFile, 'utf8')).not.toContain(PRIVATE); + }); + + it('fails closed on malformed policy before any destination write', async () => { + policy({ version: 1, exchanges: [{ session_id: SESSION, transcript: `rollout-${SESSION}.jsonl`, line_start: '4', line_end: 5 }], tool_calls: [] }); + await expect(syncConversations(source, archive, { skipIndex: true, skipSummaries: true })).rejects.toThrow(/record.*exclusion/i); + expect(fs.existsSync(archiveFile)).toBe(false); + }); + + it('honors whole-conversation opt-out before any archive copy or parsing', async () => { + const marker = 'DO NOT INDEX THIS CHAT'; + fs.writeFileSync(sourceFile, transcript().replace('safe question one', marker)); + const original = fs.readFileSync(sourceFile); + await syncConversations(source, archive, { skipIndex: true, skipSummaries: true }); + expect(fs.existsSync(archiveFile)).toBe(false); + expect(await parseConversation(sourceFile, '2026', archiveFile)).toEqual([]); + expect(fs.readFileSync(sourceFile)).toEqual(original); + }); + + it('rejects policy identity mismatch rather than silently selecting a different session', async () => { + policy(); + fs.writeFileSync(sourceFile, transcript().replace(`"id":"${SESSION}"`, '"id":"00000000-aaaa-4bbb-8ccc-dddddddddddd"')); + const result = await syncConversations(source, archive, { skipIndex: true, skipSummaries: true }); + expect(result.errors).toHaveLength(1); + expect(fs.existsSync(archiveFile)).toBe(false); + }); + + it.each(['all', 'session', 'incremental'] as const)('does not report %s indexing success after an admission rejection', async mode => { + policy(); + fs.writeFileSync(sourceFile, transcript().replace(`"id":"${SESSION}"`, '"id":"00000000-aaaa-4bbb-8ccc-dddddddddddd"')); + const run = mode === 'all' ? indexConversations(undefined, undefined, 1, true) + : mode === 'session' ? indexSession(SESSION, 1, true) : indexUnprocessed(1, true); + await expect(run).rejects.toThrow(/session identity mismatch/); + expect(fs.existsSync(archiveFile)).toBe(false); + }); + + it('fails closed when the policy itself is a symlink', async () => { + policy(); + const file = path.join(config, 'record-exclusions.json'); + fs.renameSync(file, path.join(root, 'policy.json')); + fs.symlinkSync(path.join(root, 'policy.json'), file); + await expect(syncConversations(source, archive, { skipIndex: true, skipSummaries: true })).rejects.toThrow(/record.*exclusion/i); + expect(fs.existsSync(archiveFile)).toBe(false); + }); + + it('rejects original/archive hardlink aliasing without modifying either input', async () => { + policy(); + fs.mkdirSync(path.dirname(archiveFile), { recursive: true }); + fs.linkSync(sourceFile, archiveFile); + const before = fs.readFileSync(sourceFile); + const result = await syncConversations(source, archive, { skipIndex: true, skipSummaries: true }); + expect(result.errors).toHaveLength(1); + expect(fs.readFileSync(sourceFile)).toEqual(before); + expect(fs.statSync(sourceFile).ino).toBe(fs.statSync(archiveFile).ino); + }); + + it('rejects a symlinked destination parent without creating an external archive', async () => { + policy(); + const outside = path.join(root, 'outside'); + fs.mkdirSync(outside); + fs.mkdirSync(archive); + fs.symlinkSync(outside, path.dirname(archiveFile)); + await expect(archiveAdmittedConversation(sourceFile, archiveFile)).rejects.toThrow(/archive parent.*symlink/i); + expect(fs.readdirSync(outside)).toEqual([]); + }); + + it('preserves nonmatching final newline bytes with an active empty policy', async () => { + policy({ version: 1, exchanges: [], tool_calls: [] }); + const bytes = transcript().replace(/\n$/, ''); + fs.writeFileSync(sourceFile, bytes); + await syncConversations(source, archive, { skipIndex: true, skipSummaries: true }); + expect(fs.readFileSync(archiveFile, 'utf8')).toBe(bytes); + }); + + it('refreshes an archive when the source grows but its mtime ties the archive', async () => { + expect(await archiveAdmittedConversation(sourceFile, archiveFile)).toBe(true); + const archiveTime = fs.statSync(archiveFile).mtime; + fs.appendFileSync(sourceFile, JSON.stringify(message('user', 'An appended safe exchange')) + '\n'); + fs.utimesSync(sourceFile, archiveTime, archiveTime); + + expect(await archiveAdmittedConversation(sourceFile, archiveFile)).toBe(true); + expect(fs.readFileSync(archiveFile)).toEqual(fs.readFileSync(sourceFile)); + }); + + it('rejects source mutation during streaming without publishing a derived file', async () => { + policy(); + const original = fs.createReadStream; + let changed = false; + vi.spyOn(fs, 'createReadStream').mockImplementation(((file: any, options: any) => { + if (file === sourceFile && !changed) { + changed = true; + fs.appendFileSync(sourceFile, JSON.stringify(message('user', 'later public content')) + '\n'); + } + return original(file, options); + }) as typeof fs.createReadStream); + await expect(archiveAdmittedConversation(sourceFile, archiveFile)).rejects.toThrow(/changed during record admission/); + expect(fs.existsSync(archiveFile)).toBe(false); + expect(fs.readdirSync(path.dirname(archiveFile))).toEqual([]); + }); + + it('does not copy unscanned bytes when an opt-out is appended without a record policy', async () => { + const original = fs.createReadStream; + const copy = fs.copyFileSync; + let changed = false; + const change = () => { + if (!changed) { + changed = true; + fs.appendFileSync(sourceFile, JSON.stringify(message('user', 'DO NOT INDEX THIS CHAT')) + '\n'); + } + }; + vi.spyOn(fs, 'createReadStream').mockImplementation(((file: any, options: any) => { + if (file === sourceFile) change(); + return original(file, options); + }) as typeof fs.createReadStream); + vi.spyOn(fs, 'copyFileSync').mockImplementation(((src: any, dest: any, flags: any) => { + if (src === sourceFile) change(); + return copy(src, dest, flags); + }) as typeof fs.copyFileSync); + await expect(archiveAdmittedConversation(sourceFile, archiveFile)).rejects.toThrow(/changed during record admission/); + expect(fs.existsSync(archiveFile)).toBe(false); + }); + + it('rejects a policy change during a derived copy rather than publishing stale admission', async () => { + policy({ version: 1, exchanges: [], tool_calls: [] }); + const original = fs.createReadStream; + let changed = false; + vi.spyOn(fs, 'createReadStream').mockImplementation(((file: any, options: any) => { + if (file === sourceFile && !changed) { changed = true; policy(); } + return original(file, options); + }) as typeof fs.createReadStream); + await expect(archiveAdmittedConversation(sourceFile, archiveFile)).rejects.toThrow(/policy changed during record admission/); + expect(fs.existsSync(archiveFile)).toBe(false); + expect(fs.readdirSync(path.dirname(archiveFile))).toEqual([]); + }); + + it('rejects a parser input replaced between marker scan and stream consumption', async () => { + policy(); + const original = fs.createReadStream; + let changed = false; + vi.spyOn(fs, 'createReadStream').mockImplementation(((file: any, options: any) => { + if (file === sourceFile && !changed) { + changed = true; + fs.renameSync(sourceFile, sourceFile + '.previous'); + fs.writeFileSync(sourceFile, transcript()); + } + return original(file, options); + }) as typeof fs.createReadStream); + await expect((async () => { + for await (const _line of admittedConversationLines(sourceFile)) { /* Consume to the verification boundary. */ } + })()).rejects.toThrow(/changed during record admission/); + }); + + it('does not let project-exclusion environment override record policy', async () => { + policy(); + process.env.CONVERSATION_SEARCH_EXCLUDE_PROJECTS = 'unrelated-project'; + await syncConversations(source, archive, { skipIndex: true, skipSummaries: true }); + expect(fs.readFileSync(archiveFile, 'utf8')).not.toContain(PRIVATE); + }); + + it.each(['all', 'session', 'incremental'] as const)('filters before the %s index archive copy', async mode => { + policy(); + if (mode === 'all') await indexConversations(undefined, undefined, 1, true); + if (mode === 'session') await indexSession(SESSION, 1, true); + if (mode === 'incremental') await indexUnprocessed(1, true); + expect(fs.readFileSync(archiveFile, 'utf8')).not.toContain(PRIVATE); + const db = initDatabase(); + try { + expect(db.prepare('SELECT count(*) AS n FROM exchanges').get()).toEqual({ n: 2 }); + expect(db.prepare('SELECT count(*) AS n FROM tool_calls').get()).toEqual({ n: 0 }); + } finally { db.close(); } + }); + + it('starts an ephemeral Codex text-only context rather than forking hidden history', async () => { + const fake = ` + const readline = require('readline'); + const lines = readline.createInterface({ input: process.stdin }); + const emit = value => process.stdout.write(JSON.stringify(value) + '\\n'); + lines.on('line', line => { + const m = JSON.parse(line); + if (m.method === 'initialize') emit({ id: m.id, result: {} }); + if (m.method === 'thread/fork') process.exit(9); + if (m.method === 'thread/start') { + if (!m.params.ephemeral || m.params.threadId) process.exit(10); + emit({ id: m.id, result: { thread: { id: 'synthetic-ephemeral' } } }); + } + if (m.method === 'turn/start') { + emit({ id: m.id, result: { turn: { id: 'synthetic-turn' } } }); + emit({ method: 'item/completed', params: { item: { type: 'agentMessage', text: 'Safe text only.' } } }); + emit({ method: 'turn/completed', params: { turn: { id: 'synthetic-turn', status: 'completed' } } }); + } + }); + `; + expect(await runCodexCommand({ command: process.execPath, args: ['-e', fake], + prompt: 'Only synthetic admitted content.', skipVersionCheck: true })).toContain('Safe text only.'); + }); + + it('does not resume a source session when a native record policy is active', async () => { + policy(); + vi.mocked(query).mockReturnValue({ + async *[Symbol.asyncIterator]() { + yield { type: 'result', is_error: false, result: 'Safe summary.' }; + }, + } as any); + const exchange: ConversationExchange = { id: 'safe-summary', project: 'fixture', + timestamp: '2026-01-01T00:00:00Z', userMessage: 'Explain the selected project architecture and its acceptance boundaries.', + assistantMessage: 'This is an ordinary safe answer that must remain available for summarization without any original hidden session context.', + archivePath: archiveFile, lineStart: 2, lineEnd: 3, sessionId: SESSION, harness: 'claude' }; + expect(await summarizeConversation([exchange], SESSION)).toBe('Safe summary.'); + const request = vi.mocked(query).mock.calls.at(-1)![0]; + expect(request.options.resume).toBeUndefined(); + expect(request.prompt).toContain(exchange.userMessage); + expect(request.prompt).not.toContain(PRIVATE); + }); + + it('does not route a policy-admitted Codex transcript without a session ID to Claude', async () => { + policy(); + process.env.EPISODIC_MEMORY_CODEX_BIN = path.join(root, 'missing-codex'); + vi.mocked(query).mockReset(); + vi.mocked(query).mockReturnValue({ + async *[Symbol.asyncIterator]() { + yield { type: 'result', is_error: false, result: 'Wrong provider.' }; + }, + } as any); + const exchange: ConversationExchange = { id: 'codex-no-id', project: 'fixture', + timestamp: '2026-01-01T00:00:00Z', userMessage: 'Explain the selected project architecture and its acceptance boundaries.', + assistantMessage: 'This admitted answer is long enough to require a summary but must never reach a different provider.', + archivePath: archiveFile, lineStart: 2, lineEnd: 3, harness: 'codex' }; + + await expect(summarizeConversation([exchange])).rejects.toThrow(); + expect(query).not.toHaveBeenCalled(); + }); + + it('refuses direct database reinsertion after native deletion across reopen', () => { + policy(); + const exchange: ConversationExchange = { id: 'denied-body', project: 'fixture', + timestamp: '2026-01-01T00:00:00Z', userMessage: PRIVATE, assistantMessage: PRIVATE, + archivePath: archiveFile, lineStart: 4, lineEnd: 5, sessionId: SESSION, harness: 'codex' }; + for (let n = 0; n < 2; n++) { + const db = initDatabase(); + try { + insertExchange(db, exchange, new Array(384).fill(0)); + expect(db.prepare('SELECT count(*) AS n FROM exchanges').get()).toEqual({ n: 0 }); + expect(db.prepare('SELECT count(*) AS n FROM vec_exchanges').get()).toEqual({ n: 0 }); + } finally { db.close(); } + } + }); + + it('does not trust an inherited ancestor ID over the transcript identity at direct insertion', () => { + policy(); + const db = initDatabase(); + try { + const exchange: ConversationExchange = { id: 'ancestor-metadata', project: 'fixture', + timestamp: '2026-01-01T00:00:00Z', userMessage: PRIVATE, assistantMessage: PRIVATE, + archivePath: archiveFile, lineStart: 4, lineEnd: 5, + sessionId: '00000000-aaaa-4bbb-8ccc-dddddddddddd', harness: 'codex' }; + insertExchange(db, exchange, new Array(384).fill(0)); + expect(db.prepare('SELECT count(*) AS n FROM exchanges').get()).toEqual({ n: 0 }); + expect(db.prepare('SELECT count(*) AS n FROM vec_exchanges').get()).toEqual({ n: 0 }); + } finally { db.close(); } + }); + + it('filters denied native tool IDs even when direct callers retain ancestor metadata', () => { + policy(); + const db = initDatabase(); + try { + const exchange: ConversationExchange = { id: 'ancestor-safe-body', project: 'fixture', + timestamp: '2026-01-01T00:00:00Z', userMessage: 'safe question', assistantMessage: 'safe answer', + archivePath: archiveFile, lineStart: 6, lineEnd: 9, + sessionId: '00000000-aaaa-4bbb-8ccc-dddddddddddd', harness: 'codex', + toolCalls: [{ id: 'call-denied', exchangeId: 'ancestor-safe-body', toolName: 'fixture', + toolInput: { value: PRIVATE }, toolResult: PRIVATE, isError: false, timestamp: '2026-01-01T00:00:00Z' }] }; + insertExchange(db, exchange, new Array(384).fill(0)); + expect(db.prepare('SELECT count(*) AS n FROM exchanges').get()).toEqual({ n: 1 }); + expect(db.prepare('SELECT count(*) AS n FROM tool_calls').get()).toEqual({ n: 0 }); + } finally { db.close(); } + }); + + it('retains a safe body but removes denied tool calls at database insertion', () => { + policy(); + const db = initDatabase(); + try { + const exchange: ConversationExchange = { id: 'safe-body', project: 'fixture', + timestamp: '2026-01-01T00:00:00Z', userMessage: 'safe question', assistantMessage: 'safe answer', + archivePath: archiveFile, lineStart: 6, lineEnd: 9, sessionId: SESSION, harness: 'codex', + toolCalls: [{ id: 'call-denied', exchangeId: 'safe-body', toolName: 'fixture', toolInput: { value: PRIVATE }, toolResult: PRIVATE, isError: false, timestamp: '2026-01-01T00:00:00Z' }] }; + insertExchange(db, exchange, new Array(384).fill(0)); + expect(db.prepare('SELECT count(*) AS n FROM exchanges').get()).toEqual({ n: 1 }); + expect(db.prepare('SELECT count(*) AS n FROM tool_calls').get()).toEqual({ n: 0 }); + expect(exchange.toolCalls).toHaveLength(1); + } finally { db.close(); } + }); +}); diff --git a/test/record-screening.test.ts b/test/record-screening.test.ts new file mode 100644 index 00000000..0640d95f --- /dev/null +++ b/test/record-screening.test.ts @@ -0,0 +1,311 @@ +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import crypto from 'node:crypto'; +import { spawnSync } from 'node:child_process'; +import { + admitExchange, + archiveAdmittedConversation, + readAdmittedConversation, + readRecordExclusions, +} from '../src/record-admission.js'; +import { initDatabase, insertExchange } from '../src/db.js'; +import { searchConversations } from '../src/search.js'; +import { verifyIndex } from '../src/verify.js'; +import type { ConversationExchange } from '../src/types.js'; + +const session = '00000000-1111-4222-8333-444444444444'; +// Deliberately public nonfunctional fixture, never a production credential. +const token = 'ghp_' + 'A1b2C3d4E5f6G7h8I9j0K1l2M3n4O5p6Q7r8'; +const scannerName = process.platform === 'win32' ? 'gitleaks.exe' : 'gitleaks'; +const realScannerPath = process.env.TEST_GITLEAKS_PATH || (process.env.PATH || '').split(path.delimiter) + .map(directory => path.join(directory, scannerName)).find(file => fs.existsSync(file)); +let scanner: string; +let digest: string; +let root: string; +let prior: NodeJS.ProcessEnv; +let source: string; +let destination: string; + +function policy(overrides: Record = {}): void { + fs.writeFileSync(path.join(root, 'record-exclusions.json'), JSON.stringify({ + version: 1, + exchanges: [], + tool_calls: [], + screening: { + engine: 'gitleaks', executable: scanner, sha256: digest, + timeout_ms: 10_000, max_record_bytes: 1_048_576, ...overrides, + }, + }), { mode: 0o600 }); +} + +function exchange(text = 'ordinary safe explanation'): ConversationExchange { + return { + id: 'fixture-exchange', project: 'fixture', timestamp: '2026-01-01T00:00:00Z', + userMessage: text, assistantMessage: 'safe answer', archivePath: source, + lineStart: 2, lineEnd: 3, sessionId: session, harness: 'codex', + }; +} + +function transcript(text: string): string { + return [ + { type: 'session_meta', payload: { id: session, cwd: '/synthetic' } }, + { type: 'response_item', payload: { type: 'message', role: 'user', + content: [{ type: 'input_text', text }] } }, + { type: 'response_item', payload: { type: 'message', role: 'assistant', + content: [{ type: 'output_text', text: 'safe answer' }] } }, + ].map(JSON.stringify).join('\n') + '\n'; +} + +function archivedTranscript(text: string, withSummary = false): string { + const archiveDir = path.join(root, 'archive'); + const projectDir = path.join(archiveDir, 'project'); + fs.mkdirSync(projectDir, { recursive: true }); + process.env.TEST_ARCHIVE_DIR = archiveDir; + const file = path.join(projectDir, path.basename(source)); + fs.writeFileSync(file, transcript(text)); + if (withSummary) fs.writeFileSync(file.replace('.jsonl', '-summary.txt'), 'Existing summary'); + return file; +} + +// Contract probes are executable and digest-pinned without requiring Gitleaks +// on every CI host. A separate optional integration test exercises Gitleaks. +function scannerProbe(body: string): string { + const file = path.join(root, 'scanner-probe'); + fs.writeFileSync(file, `#!${process.execPath}\n${body}\n`, { mode: 0o700 }); + policy({ executable: fs.realpathSync(file), + sha256: crypto.createHash('sha256').update(fs.readFileSync(file)).digest('hex') }); + return file; +} + +beforeEach(() => { + prior = { ...process.env }; + root = fs.mkdtempSync(path.join(os.tmpdir(), 'record-screening-test-')); + process.env.EPISODIC_MEMORY_CONFIG_DIR = root; + process.env.EPISODIC_MEMORY_DB_PATH = path.join(root, 'test.sqlite'); + scanner = path.join(root, 'scanner-fixture'); + fs.writeFileSync(scanner, `#!${process.execPath} +const fs = require('node:fs'); +const input = fs.readFileSync(0, 'utf8'); +const found = input.includes('ghp_') || /api_key\\s*=\\s*\\S{20}/i.test(input) || + (input.includes('Sourcegraph') && /[a-f0-9]{40}/i.test(input)); +process.stdout.write(found ? '[{"RuleID":"fixture"}]' : '[]'); +process.exit(found ? 10 : 0); +`, { mode: 0o700 }); + scanner = fs.realpathSync(scanner); + digest = crypto.createHash('sha256').update(fs.readFileSync(scanner)).digest('hex'); + source = path.join(root, `rollout-${session}.jsonl`); + destination = path.join(root, 'archive', path.basename(source)); + fs.writeFileSync(source, transcript('safe question')); + policy(); +}); + +afterEach(() => { + process.env = prior; + fs.rmSync(root, { recursive: true, force: true }); + expect(fs.existsSync(root)).toBe(false); +}); + +describe('pinned record-local screening at native admission', () => { + it('fails verification on an invalid record policy even with no archived files', async () => { + fs.writeFileSync(path.join(root, 'record-exclusions.json'), '{invalid'); + process.env.TEST_ARCHIVE_DIR = path.join(root, 'empty-archive'); + await expect(verifyIndex()).rejects.toThrow(/record exclusion policy/i); + }); + + it.each(['rejected', 'unavailable'] as const)('classifies %s screening independently from corruption and missing summary', async outcome => { + const file = archivedTranscript(token); + if (outcome === 'unavailable') policy({ sha256: '0'.repeat(64) }); + + const result = await verifyIndex(); + expect(result.missing.map(item => item.path)).toContain(file); + expect(result.corrupted).toEqual([]); + expect(result.screeningRejected.map(item => item.path)).toEqual(outcome === 'rejected' ? [file] : []); + expect(result.screeningUnavailable.map(item => item.path)).toEqual(outcome === 'unavailable' ? [file] : []); + }); + + it('does not report a green repair when configured screening rejects an archive', () => { + archivedTranscript(token, true); + const cli = path.join(import.meta.dirname, '..', 'dist', 'index-cli.js'); + const result = spawnSync(process.execPath, [cli, 'repair'], { + env: process.env, encoding: 'utf8', timeout: 30_000, + }); + expect(result.status).toBe(1); + expect(result.stdout).not.toContain('No issues to repair!'); + expect(result.stdout).not.toContain('Repair complete.'); + }); + + it.skipIf(!realScannerPath)('rejects a synthetic token with the actual Gitleaks executable', () => { + const executable = fs.realpathSync(realScannerPath!); + policy({ executable, sha256: crypto.createHash('sha256').update(fs.readFileSync(executable)).digest('hex') }); + expect(() => admitExchange(exchange(token))).toThrow(/content rejected by screening/i); + }); + + it('screens retained search content and summaries against the current pinned scanner', async () => { + const db = initDatabase(); + insertExchange(db, exchange(), new Array(384).fill(0)); + db.prepare('UPDATE exchanges SET assistant_message = ?').run(token); + db.close(); + await expect(searchConversations('ordinary', { mode: 'text' })).rejects.toThrow(/content rejected by screening/i); + const repaired = initDatabase(); + repaired.prepare('UPDATE exchanges SET assistant_message = ?').run('safe answer'); + repaired.close(); + fs.writeFileSync(source.replace('.jsonl', '-summary.txt'), token); + await expect(searchConversations('ordinary', { mode: 'text' })).rejects.toThrow(/content rejected by screening/i); + fs.unlinkSync(source.replace('.jsonl', '-summary.txt')); + policy({ sha256: '0'.repeat(64) }); + await expect(searchConversations('ordinary', { mode: 'text' })).rejects.toThrow(/screening unavailable/i); + }); + + it('accepts an explicitly configured scanner without changing exclusion identity', () => { + expect(readRecordExclusions()).toMatchObject({ version: 1, screening: { engine: 'gitleaks' } }); + expect(admitExchange(exchange())).toEqual(exchange()); + }); + + it('rejects a new synthetic token at direct database ingress without inserting rows', () => { + const db = initDatabase(); + try { + expect(() => insertExchange(db, exchange(token), new Array(384).fill(0))) + .toThrow(/content rejected by screening/i); + expect(db.prepare('SELECT count(*) AS n FROM exchanges').get()).toEqual({ n: 0 }); + expect(db.prepare('SELECT count(*) AS n FROM vec_exchanges').get()).toEqual({ n: 0 }); + } finally { db.close(); } + }); + + it('preserves original and previous archive when new content fails screening', async () => { + const original = transcript(token); + fs.writeFileSync(source, original); + fs.mkdirSync(path.dirname(destination)); + fs.writeFileSync(destination, 'previous admitted bytes\n'); + await expect(archiveAdmittedConversation(source, destination)).rejects.toThrow(/content rejected by screening/i); + expect(fs.readFileSync(source, 'utf8')).toBe(original); + expect(fs.readFileSync(destination, 'utf8')).toBe('previous admitted bytes\n'); + expect(fs.readdirSync(path.dirname(destination))).toEqual([path.basename(destination)]); + }); + + it('does not expose newly detected content through native read', async () => { + fs.writeFileSync(source, transcript(token)); + await expect(readAdmittedConversation(source)).rejects.toThrow(/content rejected by screening/i); + }); + + it('screens decoded JSON strings rather than trusting escaped token spelling', async () => { + fs.writeFileSync(source, transcript(token).replace('ghp_', '\\u0067hp_')); + await expect(archiveAdmittedConversation(source, destination)).rejects.toThrow(/content rejected by screening/i); + expect(fs.existsSync(destination)).toBe(false); + }); + + it('keeps separate user and assistant records from contaminating keyword screening', () => { + const x = exchange('Sourcegraph'); + x.assistantMessage = ['01234567', '89abcdef', '01234567', '89abcdef', '01234567'].join(''); + expect(admitExchange(x)).toEqual(x); + }); + + it('rejects newly detected tool payloads without exposing their values in errors', () => { + const x = exchange(); + x.toolCalls = [{ id: 'tool', exchangeId: x.id, toolName: 'fixture', + toolInput: { value: token }, toolResult: 'safe result', timestamp: x.timestamp, isError: false }]; + let message = ''; + try { admitExchange(x); } catch (error) { message = String(error); } + expect(message).toMatch(/content rejected by screening/i); + expect(message).not.toContain(token); + }); + + it('applies exact known exclusions before scanning so safe neighboring records survive', async () => { + const file = path.join(root, 'record-exclusions.json'); + const configured = JSON.parse(fs.readFileSync(file, 'utf8')); + configured.exchanges = [{ session_id: session, transcript: path.basename(source), line_start: 2, line_end: 2 }]; + fs.writeFileSync(file, JSON.stringify(configured)); + fs.writeFileSync(source, transcript(token)); + expect(await archiveAdmittedConversation(source, destination)).toBe(true); + const actual = fs.readFileSync(destination, 'utf8'); + expect(actual).not.toContain(token); + expect(actual.split('\n')[1]).toBe(''); + expect(actual).toContain('safe answer'); + }); + + it('fails closed when the scanner is unavailable or has the wrong digest', () => { + policy({ executable: path.join(root, 'missing') }); + expect(() => admitExchange(exchange())).toThrow(/screening unavailable/i); + policy({ sha256: '0'.repeat(64) }); + expect(() => admitExchange(exchange())).toThrow(/screening unavailable/i); + }); + + it('rejects oversized input before execution and never logs it', () => { + policy({ max_record_bytes: 32 }); + expect(() => admitExchange(exchange('x'.repeat(33)))).toThrow(/screening byte limit/i); + }); + + it('cannot disable configured screening through ambient scanner config or allow comments', () => { + process.env.GITLEAKS_CONFIG = '/nonexistent/ambient.toml'; + expect(() => admitExchange(exchange(token + ' # gitleaks:allow'))) + .toThrow(/content rejected by screening/i); + }); + + it('retains byte-identical safe archives and idempotent retry behavior', async () => { + const raw = fs.readFileSync(source); + expect(await archiveAdmittedConversation(source, destination)).toBe(true); + expect(fs.readFileSync(destination)).toEqual(raw); + expect(await archiveAdmittedConversation(source, destination)).toBe(false); + expect(fs.readFileSync(source)).toEqual(raw); + }); + + it('retains key/value assignment syntax required by generic credential rules', () => { + const text = JSON.stringify({ api_key: ['rP8v2Ls7', 'Qm4Zx9Nc', '6Tb3Wy5H', 'd1Kf0Aj8'].join('') }); + expect(() => admitExchange(exchange(text))).toThrow(/content rejected by screening/i); + }); + + it('does not bypass screening when session identity is absent', () => { + const x = exchange(token); + delete x.sessionId; + x.archivePath = path.join(root, 'anonymous.jsonl'); + expect(() => admitExchange(x)).toThrow(/content rejected by screening/i); + }); + + it.each([ + ['empty report', '', 0], + ['malformed report', '{', 0], + ['non-array report', '{}', 0], + ['contradictory clean exit', '[{"RuleID":"fixture"}]', 0], + ['empty detection', '[]', 10], + ['invalid detection', '[{}]', 10], + ['scanner failure', '[]', 2], + ])('rejects %s without exposing subprocess output', (_name, stdout, status) => { + scannerProbe(`process.stderr.write('private-probe-diagnostic'); + process.stdout.write(${JSON.stringify(stdout)}); process.exit(${status});`); + let message = ''; + try { admitExchange(exchange()); } catch (error) { message = String(error); } + expect(message).toMatch(/screening unavailable/i); + expect(message).not.toContain('private-probe-diagnostic'); + }); + + it('terminates a hung scanner within its bounded timeout', () => { + scannerProbe('setInterval(() => {}, 1000);'); + const file = path.join(root, 'record-exclusions.json'); + const configured = JSON.parse(fs.readFileSync(file, 'utf8')); + configured.screening.timeout_ms = 100; + fs.writeFileSync(file, JSON.stringify(configured)); + const start = performance.now(); + expect(() => admitExchange(exchange())).toThrow(/screening unavailable/i); + expect(performance.now() - start).toBeLessThan(3000); + }); + + it('rejects non-executable and symlink scanner identities', () => { + const file = scannerProbe("process.stdout.write('[]');"); + fs.chmodSync(file, 0o600); + expect(() => admitExchange(exchange())).toThrow(/screening unavailable/i); + fs.chmodSync(file, 0o700); + const alias = path.join(root, 'scanner-link'); + fs.symlinkSync(file, alias); + const configured = JSON.parse(fs.readFileSync(path.join(root, 'record-exclusions.json'), 'utf8')); + policy({ ...configured.screening, executable: alias }); + expect(() => admitExchange(exchange())).toThrow(/screening unavailable/i); + }); + + it('rejects scanner mutation during execution and cleans private work directories', () => { + const before = fs.readdirSync(os.tmpdir()).filter(name => name.startsWith('episodic-screen-')).sort(); + scannerProbe("require('node:fs').appendFileSync(__filename, '\\n// changed'); process.stdout.write('[]');"); + expect(() => admitExchange(exchange())).toThrow(/screening unavailable/i); + expect(fs.readdirSync(os.tmpdir()).filter(name => name.startsWith('episodic-screen-')).sort()).toEqual(before); + }); +}); diff --git a/test/redact-rewrite.test.ts b/test/redact-rewrite.test.ts new file mode 100644 index 00000000..b794a2fb --- /dev/null +++ b/test/redact-rewrite.test.ts @@ -0,0 +1,260 @@ +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, rmSync, existsSync, statSync, utimesSync } from 'fs'; +import { join, sep } from 'path'; +import { tmpdir } from 'os'; +import Database from 'better-sqlite3'; +import * as sqliteVec from 'sqlite-vec'; + +import { initDatabase, insertExchange } from '../src/db.js'; +import { parseConversation } from '../src/parser.js'; +import { rewriteArchive } from '../src/redact-rewrite.js'; +import { createRedactor, DEFAULT_REDACTION_CONFIG } from '../src/redaction.js'; +import { EMBEDDING_VERSION } from '../src/embedding-migration.js'; +import { FakeSecrets } from './fake-secrets.js'; + +const SESSION = '9a8b7c6d-1111-4222-8333-944455556666'; + +function transcript(secret: string, password: string, clean: string): string { + const base = { sessionId: SESSION, isSidechain: false, cwd: '/work/legacy' }; + return [ + { ...base, type: 'user', uuid: 'u1', parentUuid: null, timestamp: '2025-06-01T00:00:00.000Z', message: { role: 'user', content: `use AccountKey=${secret} please` } }, + { + ...base, type: 'assistant', uuid: 'a1', parentUuid: 'u1', timestamp: '2025-06-01T00:00:01.000Z', + message: { role: 'assistant', content: [ + { type: 'text', text: 'Running it.' }, + { type: 'tool_use', id: 'toolu_1', name: 'Bash', input: { command: `mysql --password=${password}` } }, + ] }, + }, + { ...base, type: 'user', uuid: 'u2', parentUuid: 'a1', timestamp: '2025-06-01T00:00:02.000Z', message: { role: 'user', content: [{ type: 'tool_result', tool_use_id: 'toolu_1', content: `ok password: ${password}` }] } }, + { ...base, type: 'assistant', uuid: 'a2', parentUuid: 'u2', timestamp: '2025-06-01T00:00:03.000Z', message: { role: 'assistant', content: [{ type: 'text', text: clean }] } }, + ].map(l => JSON.stringify(l)).join('\n') + '\n'; +} + +describe('redact --rewrite backfill', () => { + let root: string; + let archiveDir: string; + let dbPath: string; + const savedEnv = { ...process.env }; + const embed = vi.fn(async (..._args: unknown[]) => new Array(384).fill(0.5)); + + beforeEach(() => { + root = mkdtempSync(join(tmpdir(), 'episodic-memory-redact-rewrite-')); + archiveDir = join(root, 'archive'); + dbPath = join(root, 'db.sqlite'); + process.env.TEST_DB_PATH = dbPath; + embed.mockClear(); + }); + + afterEach(() => { + process.env = { ...savedEnv }; + rmSync(root, { recursive: true, force: true }); + }); + + /** Simulate a pre-redaction install: unredacted archive, rows, and summary. */ + async function seedLegacy(fake: FakeSecrets) { + const secret = fake.azureStorageKey(); + const password = fake.password(); + const projectDir = join(archiveDir, '-work-legacy'); + mkdirSync(join(projectDir, SESSION, 'subagents'), { recursive: true }); + const dirty = join(projectDir, `${SESSION}.jsonl`); + const clean = join(projectDir, 'clean-session.jsonl'); + const nested = join(projectDir, SESSION, 'subagents', 'agent-abc.jsonl'); + writeFileSync(dirty, transcript(secret, password, 'All done.')); + writeFileSync(nested, transcript(secret, password, 'Sub-agent done.')); + writeFileSync(clean, transcript('[REDACTED:connection-string-secret]', '[REDACTED:secret-assignment]', 'Nothing here.')); + writeFileSync(dirty.replace('.jsonl', '-summary.txt'), 'Configured storage access.'); + writeFileSync(clean.replace('.jsonl', '-summary.txt'), 'A clean summary.'); + const old = new Date('2025-06-02T00:00:00Z'); + for (const f of [dirty, clean, nested]) utimesSync(f, old, old); + + const db = initDatabase(); + for (const file of [dirty, clean, nested]) { + for (const ex of await parseConversation(file, '-work-legacy', file)) { + insertExchange(db, ex, new Array(384).fill(0), ex.toolCalls?.map(t => t.toolName)); + } + } + // Pretend these rows predate the current encoder bookkeeping. + db.prepare('UPDATE exchanges SET embedding_version = 0').run(); + db.close(); + return { secret, password, dirty, clean, nested }; + } + + function dbDump(): string { + const db = new Database(dbPath, { readonly: true }); + sqliteVec.load(db); + try { + return JSON.stringify([ + db.prepare('SELECT user_message, assistant_message FROM exchanges').all(), + db.prepare('SELECT tool_input, tool_result FROM tool_calls').all(), + ]); + } finally { + db.close(); + } + } + + it('rewrites the archive and index in place, then is a no-op on a second run', async () => { + const fake = new FakeSecrets(808); + const { secret, password, dirty, clean, nested } = await seedLegacy(fake); + const lineCount = readFileSync(dirty, 'utf-8').split('\n').length; + const mtimeBefore = statSync(dirty).mtimeMs; + const cleanBytes = readFileSync(clean, 'utf-8'); + expect(dbDump()).toContain(secret); + + const redactor = createRedactor(DEFAULT_REDACTION_CONFIG); + const result = await rewriteArchive({ archiveDir, redactor, embed }); + + // Archive: secrets gone, structure and mtime preserved, nested files covered. + for (const f of [dirty, nested]) { + const text = readFileSync(f, 'utf-8'); + expect(text).not.toContain(secret); + expect(text).not.toContain(password); + text.split('\n').filter(Boolean).forEach(l => expect(() => JSON.parse(l)).not.toThrow()); + } + expect(readFileSync(dirty, 'utf-8').split('\n').length).toBe(lineCount); + expect(Math.abs(statSync(dirty).mtimeMs - mtimeBefore)).toBeLessThan(2); + expect(readFileSync(clean, 'utf-8')).toBe(cleanBytes); + + // Summaries generated from unredacted text are removed; clean ones stay. + expect(existsSync(dirty.replace('.jsonl', '-summary.txt'))).toBe(false); + expect(existsSync(clean.replace('.jsonl', '-summary.txt'))).toBe(true); + + // Index: text columns clean, affected rows re-embedded with the current version. + const dump = dbDump(); + expect(dump).not.toContain(secret); + expect(dump).not.toContain(password); + expect(dump).toContain('[REDACTED:'); + expect(embed).toHaveBeenCalled(); + expectNoSecretInCalls(embed.mock.calls, [secret, password]); + + const db = new Database(dbPath, { readonly: true }); + // Only rows that had secrets are touched; the clean file's rows keep their old version. + const versions = db.prepare( + `SELECT embedding_version v, COUNT(*) n FROM exchanges WHERE archive_path IN (?, ?) AND (user_message LIKE '%REDACTED%' OR assistant_message LIKE '%REDACTED%') GROUP BY v` + ).all(dirty, nested) as Array<{ v: number }>; + const untouched = db.prepare('SELECT DISTINCT embedding_version v FROM exchanges WHERE archive_path = ?').all(clean) as Array<{ v: number }>; + db.close(); + expect(versions.map(r => r.v)).toEqual([EMBEDDING_VERSION]); + expect(untouched.map(r => r.v)).toEqual([0]); + + expect(result.filesRewritten).toBe(2); + expect(result.summariesRemoved).toBe(1); + expect(result.rowsUpdated).toBeGreaterThan(0); + expect(JSON.stringify(result)).not.toContain(secret); + + // Second run: nothing left to do. + embed.mockClear(); + const second = await rewriteArchive({ archiveDir, redactor, embed }); + expect(second.filesRewritten).toBe(0); + expect(second.rowsUpdated).toBe(0); + expect(second.summariesRemoved).toBe(0); + expect(embed).not.toHaveBeenCalled(); + }); + + it('--dry-run reports counts without touching anything', async () => { + const fake = new FakeSecrets(909); + const { secret, dirty } = await seedLegacy(fake); + const before = readFileSync(dirty, 'utf-8'); + + const result = await rewriteArchive({ archiveDir, redactor: createRedactor(DEFAULT_REDACTION_CONFIG), embed, dryRun: true }); + + expect(result.filesRewritten).toBe(2); + expect(result.rowsUpdated).toBeGreaterThan(0); + expect(readFileSync(dirty, 'utf-8')).toBe(before); + expect(existsSync(dirty.replace('.jsonl', '-summary.txt'))).toBe(true); + expect(dbDump()).toContain(secret); + expect(embed).not.toHaveBeenCalled(); + }); + + it('--dry-run leaves an older-schema index byte-identical instead of migrating it', async () => { + await seedLegacy(new FakeSecrets(910)); + const legacy = new Database(dbPath); + legacy.exec('ALTER TABLE exchanges DROP COLUMN embedding_version'); + legacy.close(); + const before = readFileSync(dbPath); + + const result = await rewriteArchive({ archiveDir, redactor: createRedactor(DEFAULT_REDACTION_CONFIG), embed, dryRun: true }); + + expect(result.rowsUpdated).toBeGreaterThan(0); + expect(readFileSync(dbPath).equals(before)).toBe(true); + const db = new Database(dbPath, { readonly: true }); + const columns = (db.prepare('PRAGMA table_info(exchanges)').all() as Array<{ name: string }>).map(c => c.name); + db.close(); + expect(columns).not.toContain('embedding_version'); + }); + + it('--dry-run does not create an index that does not exist', async () => { + mkdirSync(join(archiveDir, '-work-legacy'), { recursive: true }); + const result = await rewriteArchive({ archiveDir, redactor: createRedactor(DEFAULT_REDACTION_CONFIG), embed, dryRun: true }); + expect(result.rowsUpdated).toBe(0); + expect(existsSync(dbPath)).toBe(false); + }); + + it('report lists one hit per redacted value, with location, rule and shape but never the value', async () => { + const { secret, password } = await seedLegacy(new FakeSecrets(911)); + const hits: Array<{ location: string; ruleId: string; shape: string; context: string }> = []; + + const result = await rewriteArchive({ + archiveDir, redactor: createRedactor(DEFAULT_REDACTION_CONFIG), embed, dryRun: true, report: hit => hits.push(hit), + }); + + const total = result.findings.reduce((n, f) => n + f.count, 0); + expect(hits.length).toBe(total); + const text = JSON.stringify(hits); + expect(text).not.toContain(secret); + expect(text).not.toContain(password); + + const archiveHit = hits.find(h => h.location === `-work-legacy${sep}${SESSION}.jsonl:1`); + expect(archiveHit?.ruleId).toBe('connection-string-secret'); + expect(archiveHit!.shape).toMatch(new RegExp(`^len=${secret.length} \\S+ H=\\d+\\.\\d$`)); + expect(archiveHit!.context).toContain('AccountKey=[REDACTED:connection-string-secret]'); + expect(hits.some(h => h.location.startsWith('index:') && h.location.includes(' user'))).toBe(true); + }); + + it('report refuses to run without dryRun, since files would already be rewritten', async () => { + mkdirSync(archiveDir, { recursive: true }); + await expect(rewriteArchive({ + archiveDir, redactor: createRedactor(DEFAULT_REDACTION_CONFIG), embed, report: () => {}, + })).rejects.toThrow(/dryRun/); + }); + + /** Dry-run report over one archive file holding `line`. */ + async function reportLine(line: string) { + mkdirSync(join(archiveDir, '-work-x'), { recursive: true }); + writeFileSync(join(archiveDir, '-work-x', 's.jsonl'), line + '\n'); + const hits: Array<{ location: string; ruleId: string; shape: string; context: string }> = []; + await rewriteArchive({ archiveDir, redactor: createRedactor(DEFAULT_REDACTION_CONFIG), embed, dryRun: true, report: h => hits.push(h) }); + return hits; + } + + it('report pairs a new hit with its own token, not an earlier token of the same rule', async () => { + const pw = new FakeSecrets(912).chars('abcdefghijklmnopqrstuvwxyz', 14); + const hits = await reportLine( + JSON.stringify({ content: `first: Server=a;password=${pw}; later: Server=b;Password=[REDACTED:connection-string-secret];` }) + ); + expect(hits).toHaveLength(1); + expect(hits[0].shape).toMatch(/^len=14 a /); + expect(hits[0].context).toContain('first: Server=a;password=[REDACTED:connection-string-secret]'); + expect(hits[0].context).not.toContain('--hit'); + }); + + it('report keeps a hit swallowed by a whole secret field, and shapes the field from its original value', async () => { + const fake = new FakeSecrets(913); + const word = fake.chars('abcdefghijklmnopqrstuvwxyz', 9); + const jwt = fake.jwt(); + const hits = await reportLine(JSON.stringify({ password: `${word} ${jwt}` })); + + expect(hits.map(h => h.ruleId).sort()).toEqual(['jwt', 'secret-field']); + const field = hits.find(h => h.ruleId === 'secret-field')!; + expect(field.shape).toMatch(new RegExp(`^len=${word.length + 1 + jwt.length} `)); + const swallowed = hits.find(h => h.ruleId === 'jwt')!; + expect(swallowed.shape).toMatch(new RegExp(`^len=${jwt.length} `)); + expect(swallowed.context).toBe(field.context); + expect(field.context).toContain('"password":"[REDACTED:secret-field]"'); + expect(JSON.stringify(hits)).not.toContain(word); + }); +}); + +function expectNoSecretInCalls(calls: unknown[][], secrets: string[]) { + const text = JSON.stringify(calls); + for (const s of secrets) expect(text.includes(s)).toBe(false); +} diff --git a/test/redaction-pipeline.test.ts b/test/redaction-pipeline.test.ts new file mode 100644 index 00000000..a1a19c06 --- /dev/null +++ b/test/redaction-pipeline.test.ts @@ -0,0 +1,421 @@ +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import { + mkdtempSync, mkdirSync, writeFileSync, readFileSync, rmSync, existsSync, readdirSync, statSync, + appendFileSync, utimesSync, +} from 'fs'; +import { join } from 'path'; +import { tmpdir } from 'os'; +import Database from 'better-sqlite3'; +import * as sqliteVec from 'sqlite-vec'; + +// Capture every embedding input and every summarizer call. sync.ts loads both +// via dynamic import and indexer.ts statically; vi.mock intercepts both. +const { embedSpy, summarizeSpy } = vi.hoisted(() => ({ + embedSpy: vi.fn(async (..._args: unknown[]) => new Array(384).fill(0)), + summarizeSpy: vi.fn(async (..._args: unknown[]) => 'A summary.'), +})); +vi.mock('../src/embeddings.js', () => ({ + initEmbeddings: vi.fn(async () => {}), + generateExchangeEmbedding: embedSpy, + generateQueryEmbedding: vi.fn(async () => new Array(384).fill(0)), + generateEmbedding: vi.fn(), + initEmbeddingsFailed: false, +})); +vi.mock('../src/summarizer.js', async () => { + const actual = await vi.importActual('../src/summarizer.js'); + return { ...actual, summarizeConversation: summarizeSpy }; +}); + +import { syncConversations } from '../src/sync.js'; +import { indexUnprocessed } from '../src/indexer.js'; +import { searchConversations } from '../src/search.js'; +import { formatConversationAsMarkdown } from '../src/show.js'; +import { exportOpencodeSessions } from '../src/opencode-sync.js'; +import { createRedactor, DEFAULT_REDACTION_CONFIG } from '../src/redaction.js'; +import { FakeSecrets } from './fake-secrets.js'; + +const SESSION = '4f1c2b3a-1111-4222-8333-944455556666'; + +interface Seed { + secrets: string[]; + sha: string; + guid: string; +} + +function seedSecrets(seed: number): Seed { + const fake = new FakeSecrets(seed); + return { + secrets: [ + fake.azureClientSecret(), + fake.azureStorageKey(), + fake.sasSignature(), + fake.githubToken(), + fake.anthropicKey(), + fake.password(), + fake.jwt(), + fake.privateKeyBlock().split('\n')[2], + ], + sha: fake.gitSha(), + guid: fake.guid(), + }; +} + +function claudeTranscript(s: Seed): string { + const [clientSecret, storageKey, sasSig, ghToken, antKey, password, jwt] = s.secrets; + const keyBlock = (() => { + // Rebuild the block whose body line is s.secrets[7]. + const dashes = '-'.repeat(5); + return `${dashes}BEGIN RSA PRIVATE` + ` KEY${dashes}\n${s.secrets[7]}\n${dashes}END RSA PRIVATE` + ` KEY${dashes}`; + })(); + const ts = (n: number) => new Date(Date.UTC(2026, 0, 1, 0, n)).toISOString(); + const base = { sessionId: SESSION, isSidechain: false, cwd: '/work/contoso', gitBranch: 'main', version: '2.0.0' }; + const lines = [ + { + ...base, type: 'user', uuid: 'u1', parentUuid: null, timestamp: ts(1), + message: { + role: 'user', + content: `Deploy with client secret ${clientSecret} for tenant ${s.guid}; last good commit ${s.sha}`, + }, + }, + { + ...base, type: 'assistant', uuid: 'a1', parentUuid: 'u1', timestamp: ts(2), + message: { + role: 'assistant', + content: [ + { type: 'text', text: 'Checking the storage account keys.' }, + { + type: 'tool_use', id: 'toolu_1', name: 'Bash', + input: { command: `az storage blob upload --connection-string "AccountName=contosodata;AccountKey=${storageKey}" --sas-token "sv=2022-11-02&sig=${sasSig}"` }, + }, + ], + }, + }, + { + ...base, type: 'user', uuid: 'u2', parentUuid: 'a1', timestamp: ts(3), + message: { + role: 'user', + content: [{ + type: 'tool_result', tool_use_id: 'toolu_1', + content: `GITHUB_TOKEN=${ghToken}\nANTHROPIC_API_KEY=${antKey}\npassword: ${password}\n${keyBlock}`, + }], + }, + }, + { + ...base, type: 'assistant', uuid: 'a2', parentUuid: 'u2', timestamp: ts(4), + message: { role: 'assistant', content: [{ type: 'text', text: `Uploaded. The bearer was Bearer ${jwt}.` }] }, + }, + ]; + return lines.map(l => JSON.stringify(l)).join('\n') + '\n'; +} + +function codexRollout(s: Seed): string { + const [clientSecret, storageKey] = s.secrets; + const lines = [ + { timestamp: '2026-05-12T18:00:00.000Z', type: 'session_meta', payload: { id: '019e4c75-d5bf-7c71-9df7-77f5fb86b711', cwd: '/work/contoso', cli_version: '0.130.0', model_provider: 'openai' } }, + { timestamp: '2026-05-12T18:00:02.000Z', type: 'response_item', payload: { type: 'message', role: 'user', content: [{ type: 'input_text', text: `Rotate ${clientSecret} please` }] } }, + { timestamp: '2026-05-12T18:00:04.000Z', type: 'response_item', payload: { type: 'function_call', name: 'exec_command', arguments: JSON.stringify({ cmd: `echo AccountKey=${storageKey}` }), call_id: 'c1' } }, + { timestamp: '2026-05-12T18:00:05.000Z', type: 'response_item', payload: { type: 'function_call_output', call_id: 'c1', output: `AccountKey=${storageKey}` } }, + { timestamp: '2026-05-12T18:00:06.000Z', type: 'response_item', payload: { type: 'message', role: 'assistant', content: [{ type: 'output_text', text: 'Rotated.' }] } }, + ]; + return lines.map(l => JSON.stringify(l)).join('\n') + '\n'; +} + +function walkFiles(dir: string): string[] { + if (!existsSync(dir)) return []; + const out: string[] = []; + for (const entry of readdirSync(dir, { withFileTypes: true })) { + const p = join(dir, entry.name); + if (entry.isDirectory()) out.push(...walkFiles(p)); + else out.push(p); + } + return out; +} + +function dbText(dbPath: string): string { + if (!existsSync(dbPath)) return ''; + const db = new Database(dbPath, { readonly: true }); + sqliteVec.load(db); + try { + const ex = db.prepare('SELECT user_message, assistant_message FROM exchanges').all(); + const tc = db.prepare('SELECT tool_input, tool_result FROM tool_calls').all(); + return JSON.stringify([ex, tc]); + } finally { + db.close(); + } +} + +function expectNoSecrets(haystack: string, s: Seed, where: string) { + for (const secret of s.secrets) { + expect(haystack.includes(secret), `${where} leaked a seeded secret`).toBe(false); + } +} + +describe('redaction pipeline', () => { + let root: string; + let sourceDir: string; + let archiveDir: string; + let dbPath: string; + let logged: string[]; + const savedEnv = { ...process.env }; + + beforeEach(() => { + root = mkdtempSync(join(tmpdir(), 'episodic-memory-redaction-pipeline-')); + sourceDir = join(root, 'source'); + archiveDir = join(root, 'archive'); + dbPath = join(root, 'db.sqlite'); + mkdirSync(join(sourceDir, '-work-contoso'), { recursive: true }); + process.env.TEST_DB_PATH = dbPath; + process.env.TEST_PROJECTS_DIR = sourceDir; + process.env.TEST_ARCHIVE_DIR = archiveDir; + process.env.EPISODIC_MEMORY_CONFIG_DIR = join(root, 'config'); + delete process.env.EPISODIC_MEMORY_REDACTION; + delete process.env.EPISODIC_MEMORY_REDACTION_RULES; + delete process.env.EPISODIC_MEMORY_REDACTION_STRICT; + delete process.env.EPISODIC_MEMORY_SKIP_SUMMARIES; + + embedSpy.mockClear(); + summarizeSpy.mockClear(); + logged = []; + for (const method of ['log', 'error', 'warn', 'info'] as const) { + vi.spyOn(console, method).mockImplementation((...args: unknown[]) => { + logged.push(args.map(a => (typeof a === 'string' ? a : JSON.stringify(a))).join(' ')); + }); + } + }); + + afterEach(() => { + vi.restoreAllMocks(); + process.env = { ...savedEnv }; + rmSync(root, { recursive: true, force: true }); + }); + + it('a seeded secret appears in zero of: archive, SQLite, embedding input, summarizer input, logs', async () => { + const s = seedSecrets(2024); + const claudeSrc = join(sourceDir, '-work-contoso', `${SESSION}.jsonl`); + writeFileSync(claudeSrc, claudeTranscript(s)); + writeFileSync(join(sourceDir, '-work-contoso', 'rollout-2026-05-12T18-00-00-019e4c75-d5bf-7c71-9df7-77f5fb86b711.jsonl'), codexRollout(s)); + + const result = await syncConversations(sourceDir, archiveDir); + expect(result.errors).toEqual([]); + expect(result.copied).toBe(2); + expect(result.indexed).toBe(2); + expect(result.summarized).toBe(2); + expect(result.redactions?.length).toBeGreaterThan(0); + expect(JSON.stringify(result.redactions)).not.toMatch(/[A-Za-z0-9+/]{30,}/); + + // (1) archive + const archived = walkFiles(archiveDir).filter(f => f.endsWith('.jsonl')); + expect(archived.length).toBe(2); + for (const file of archived) { + const text = readFileSync(file, 'utf-8'); + expectNoSecrets(text, s, `archive ${file}`); + expect(text).toContain('[REDACTED:'); + } + + // (2) SQLite text columns (exchanges + tool_calls) + const stored = dbText(dbPath); + expect(stored.length).toBeGreaterThan(0); + expectNoSecrets(stored, s, 'SQLite'); + expect(stored).toContain('[REDACTED:'); + + // (3) embedding input + expect(embedSpy).toHaveBeenCalled(); + expectNoSecrets(JSON.stringify(embedSpy.mock.calls), s, 'embedding input'); + + // (4) summarizer input — and resume/fork must be off, since those paths + // would hand the model the unredacted source transcript. + expect(summarizeSpy).toHaveBeenCalledTimes(2); + expectNoSecrets(JSON.stringify(summarizeSpy.mock.calls), s, 'summarizer input'); + for (const call of summarizeSpy.mock.calls) { + expect(call[2]).toMatchObject({ allowResume: false }); + } + + // (5) logs + expectNoSecrets(logged.join('\n'), s, 'console output'); + }); + + it('archive stays valid JSONL with the same line count, and show/read still render it', async () => { + const s = seedSecrets(7); + const src = join(sourceDir, '-work-contoso', `${SESSION}.jsonl`); + writeFileSync(src, claudeTranscript(s)); + await syncConversations(sourceDir, archiveDir, { skipSummaries: true }); + + const archived = join(archiveDir, '-work-contoso', `${SESSION}.jsonl`); + const srcLines = readFileSync(src, 'utf-8').split('\n'); + const outLines = readFileSync(archived, 'utf-8').split('\n'); + expect(outLines.length).toBe(srcLines.length); + outLines.filter(Boolean).forEach(l => expect(() => JSON.parse(l)).not.toThrow()); + + // The archive keeps the source mtime so the next sync treats it as current. + expect(Math.abs(statSync(archived).mtimeMs - statSync(src).mtimeMs)).toBeLessThan(2); + const again = await syncConversations(sourceDir, archiveDir, { skipSummaries: true }); + expect(again.copied).toBe(0); + + const md = formatConversationAsMarkdown(readFileSync(archived, 'utf-8')); + expect(md).toContain('[REDACTED:azure-client-secret]'); + expectNoSecrets(md, s, 'show output'); + }); + + it('git SHAs and GUIDs remain findable via text search', async () => { + const s = seedSecrets(31); + writeFileSync(join(sourceDir, '-work-contoso', `${SESSION}.jsonl`), claudeTranscript(s)); + await syncConversations(sourceDir, archiveDir, { skipSummaries: true }); + + const bySha = await searchConversations(s.sha, { mode: 'text' }); + expect(bySha.length).toBe(1); + const byGuid = await searchConversations(s.guid, { mode: 'text' }); + expect(byGuid.length).toBe(1); + const byToken = await searchConversations('[REDACTED:azure-client-secret]', { mode: 'text' }); + expect(byToken.length).toBe(1); + }); + + it('fails closed in strict mode: a corrupt rules file leaves no new archive or index content', async () => { + const s = seedSecrets(5); + writeFileSync(join(sourceDir, '-work-contoso', `${SESSION}.jsonl`), claudeTranscript(s)); + const rules = join(root, 'rules.json'); + writeFileSync(rules, '{ "rules": [ { "id": "x", "pattern": "(" } ] }'); + process.env.EPISODIC_MEMORY_REDACTION_RULES = rules; + + await expect(syncConversations(sourceDir, archiveDir)).rejects.toThrow(/redaction/i); + expect(walkFiles(archiveDir)).toEqual([]); + expect(dbText(dbPath)).toBe(''); + expect(embedSpy).not.toHaveBeenCalled(); + expect(summarizeSpy).not.toHaveBeenCalled(); + }); + + it('EPISODIC_MEMORY_REDACTION=off restores the byte-for-byte copy and resume-capable summaries', async () => { + process.env.EPISODIC_MEMORY_REDACTION = 'off'; + const s = seedSecrets(8); + const src = join(sourceDir, '-work-contoso', `${SESSION}.jsonl`); + writeFileSync(src, claudeTranscript(s)); + await syncConversations(sourceDir, archiveDir); + expect(readFileSync(join(archiveDir, '-work-contoso', `${SESSION}.jsonl`), 'utf-8')).toBe(readFileSync(src, 'utf-8')); + expect(summarizeSpy.mock.calls[0][2]).toMatchObject({ allowResume: true }); + }); + + it('.NET/Azure key-context secrets (no recognizable shape) never reach archive or SQLite', async () => { + const fake = new FakeSecrets(1999); + const appsettingsSecret = fake.passphrase(); + const appSettingValue = fake.passphrase(); + const vaultValue = fake.passphrase(); + const mcpSecret = fake.passphrase(); + const seed: Seed = { secrets: [appsettingsSecret, appSettingValue, vaultValue, mcpSecret], sha: fake.gitSha(), guid: fake.guid() }; + const base = { sessionId: SESSION, isSidechain: false, cwd: '/work/contoso' }; + const lines = [ + { ...base, type: 'user', uuid: 'u1', parentUuid: null, timestamp: '2026-01-01T00:00:00Z', message: { role: 'user', content: 'Why does the API fail to get a token?' } }, + { ...base, type: 'assistant', uuid: 'a1', parentUuid: 'u1', timestamp: '2026-01-01T00:00:01Z', message: { role: 'assistant', content: [ + { type: 'tool_use', id: 't1', name: 'Read', input: { file_path: '/work/contoso/appsettings.Development.json' } }, + ] } }, + { ...base, type: 'user', uuid: 'u2', parentUuid: 'a1', timestamp: '2026-01-01T00:00:02Z', + message: { role: 'user', content: [{ type: 'tool_result', tool_use_id: 't1', content: JSON.stringify({ AzureAd: { TenantId: seed.guid, ClientSecret: appsettingsSecret } }, null, 2) }] } }, + { ...base, type: 'assistant', uuid: 'a2', parentUuid: 'u2', timestamp: '2026-01-01T00:00:03Z', message: { role: 'assistant', content: [ + { type: 'tool_use', id: 't2', name: 'Bash', input: { command: 'az webapp config appsettings list -g rg -n app && az keyvault secret show --vault-name kv -n Db' } }, + ] } }, + { ...base, type: 'user', uuid: 'u3', parentUuid: 'a2', timestamp: '2026-01-01T00:00:04Z', + message: { role: 'user', content: [{ type: 'tool_result', tool_use_id: 't2', content: + JSON.stringify([{ name: 'Stripe__ApiKey', slotSetting: false, value: appSettingValue }], null, 2) + '\n' + + JSON.stringify({ id: 'https://kv.vault.azure.net/secrets/Db/0123', value: vaultValue }, null, 2) }] }, + toolUseResult: { structuredContent: { name: 'GraphClientSecret', value: mcpSecret } } }, + { ...base, type: 'assistant', uuid: 'a3', parentUuid: 'u3', timestamp: '2026-01-01T00:00:05Z', message: { role: 'assistant', content: [{ type: 'text', text: 'The client secret has expired.' }] } }, + ]; + writeFileSync(join(sourceDir, '-work-contoso', `${SESSION}.jsonl`), lines.map(l => JSON.stringify(l)).join('\n') + '\n'); + + const result = await syncConversations(sourceDir, archiveDir, { skipSummaries: true }); + expect(result.errors).toEqual([]); + const archived = readFileSync(join(archiveDir, '-work-contoso', `${SESSION}.jsonl`), 'utf-8'); + expectNoSecrets(archived, seed, 'archive'); + expectNoSecrets(dbText(dbPath), seed, 'SQLite'); + expectNoSecrets(JSON.stringify(embedSpy.mock.calls), seed, 'embedding input'); + expect(archived).toContain(seed.guid); + archived.split('\n').filter(Boolean).forEach(l => expect(() => JSON.parse(l)).not.toThrow()); + }); + + it('the `index` path (indexUnprocessed) parses the redacted archive, not the source', async () => { + const s = seedSecrets(77); + writeFileSync(join(sourceDir, '-work-contoso', `${SESSION}.jsonl`), claudeTranscript(s)); + + await indexUnprocessed(1, false); + + const archived = walkFiles(archiveDir).filter(f => f.endsWith('.jsonl')); + expect(archived.length).toBe(1); + expectNoSecrets(readFileSync(archived[0], 'utf-8'), s, 'archive (index path)'); + expectNoSecrets(dbText(dbPath), s, 'SQLite (index path)'); + expectNoSecrets(JSON.stringify(embedSpy.mock.calls), s, 'embedding input (index path)'); + expect(summarizeSpy).toHaveBeenCalledTimes(1); + expectNoSecrets(JSON.stringify(summarizeSpy.mock.calls), s, 'summarizer input (index path)'); + expect(summarizeSpy.mock.calls[0][2]).toMatchObject({ allowResume: false }); + }); + + it('the `index` path refreshes an indexed archive even when the append left the source mtime no newer', async () => { + const fake = new FakeSecrets(78); + const secret = fake.azureClientSecret(); + const base = { sessionId: SESSION, isSidechain: false, cwd: '/work/contoso', gitBranch: 'main', version: '2.0.0' }; + const exchange = (n: number, text: string) => [ + { ...base, type: 'user', uuid: `u${n}`, parentUuid: n > 1 ? `a${n - 1}` : null, + timestamp: new Date(Date.UTC(2026, 0, 1, 0, n)).toISOString(), + message: { role: 'user', content: text } }, + { ...base, type: 'assistant', uuid: `a${n}`, parentUuid: `u${n}`, + timestamp: new Date(Date.UTC(2026, 0, 1, 0, n, 30)).toISOString(), + message: { role: 'assistant', content: [{ type: 'text', text: `Reply ${n}.` }] } }, + ].map(l => JSON.stringify(l) + '\n').join(''); + const src = join(sourceDir, '-work-contoso', `${SESSION}.jsonl`); + + writeFileSync(src, exchange(1, 'First question.')); + await indexUnprocessed(1, true); + const archived = walkFiles(archiveDir).filter(f => f.endsWith('.jsonl'))[0]; + const archivedMtime = statSync(archived).mtime; + + // Append, then pin the source mtime to the archive's: the archive writer's mtime + // check alone would treat the archive as current. + appendFileSync(src, exchange(2, `Second question with ${secret} as the secret`)); + utimesSync(src, archivedMtime, archivedMtime); + await indexUnprocessed(1, true); + + const text = dbText(dbPath); + expect(text).toContain('Second question with [REDACTED:'); + expect(text).not.toContain(secret); + }); +}); + +describe('redaction: opencode staging export', () => { + let root: string; + const savedEnv = { ...process.env }; + + beforeEach(() => { + root = mkdtempSync(join(tmpdir(), 'episodic-memory-redaction-opencode-')); + process.env.EPISODIC_MEMORY_OPENCODE_DB_PATH = join(root, 'opencode.db'); + process.env.EPISODIC_MEMORY_OPENCODE_TRANSCRIPT_DIR = join(root, 'transcripts'); + }); + + afterEach(() => { + process.env = { ...savedEnv }; + rmSync(root, { recursive: true, force: true }); + }); + + it('redacts secrets before the staging JSONL is written', () => { + const fake = new FakeSecrets(55); + const secret = fake.azureStorageKey(); + const db = new Database(join(root, 'opencode.db')); + db.exec(` + CREATE TABLE project (id TEXT PRIMARY KEY, worktree TEXT NOT NULL, name TEXT, time_created INTEGER NOT NULL, time_updated INTEGER NOT NULL, sandboxes TEXT NOT NULL); + CREATE TABLE session (id TEXT PRIMARY KEY, project_id TEXT NOT NULL, slug TEXT NOT NULL, directory TEXT NOT NULL, title TEXT NOT NULL, version TEXT NOT NULL, time_created INTEGER NOT NULL, time_updated INTEGER NOT NULL, agent TEXT, model TEXT); + CREATE TABLE message (id TEXT PRIMARY KEY, session_id TEXT NOT NULL, time_created INTEGER NOT NULL, time_updated INTEGER NOT NULL, data TEXT NOT NULL); + CREATE TABLE part (id TEXT PRIMARY KEY, message_id TEXT NOT NULL, session_id TEXT NOT NULL, time_created INTEGER NOT NULL, time_updated INTEGER NOT NULL, data TEXT NOT NULL); + `); + db.prepare(`INSERT INTO project VALUES ('p1', '/work/x', 'X', 1700000000000, 1700000000000, '[]')`).run(); + db.prepare(`INSERT INTO session VALUES ('ses_1', 'p1', 's', '/work/x', 'T', '1.0.0', 1700000000000, 1700000004000, 'build', NULL)`).run(); + db.prepare(`INSERT INTO message VALUES ('m1', 'ses_1', 1700000001000, 1700000001000, ?)`).run(JSON.stringify({ role: 'user' })); + db.prepare(`INSERT INTO part VALUES ('pt1', 'm1', 'ses_1', 1700000001000, 1700000001000, ?)`).run( + JSON.stringify({ type: 'text', text: `AccountKey=${secret}` }) + ); + db.close(); + + const result = exportOpencodeSessions({ redactor: createRedactor(DEFAULT_REDACTION_CONFIG) }); + expect(result.exported).toBe(1); + const files = walkFiles(join(root, 'transcripts')); + expect(files.length).toBe(1); + const text = readFileSync(files[0], 'utf-8'); + expect(text).not.toContain(secret); + expect(text).toContain('[REDACTED:connection-string-secret]'); + }); +}); diff --git a/test/redaction.test.ts b/test/redaction.test.ts new file mode 100644 index 00000000..3ac42229 --- /dev/null +++ b/test/redaction.test.ts @@ -0,0 +1,736 @@ +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import { mkdtempSync, writeFileSync, readFileSync, rmSync, mkdirSync, chmodSync, statSync } from 'fs'; +import { join } from 'path'; +import { tmpdir } from 'os'; + +import { + createRedactor, + loadRedactor, + loadRedactionConfig, + getRedactionSettings, + redactJsonlLine, + copyFileRedacted, + FindingsTally, + RedactionConfigError, + DEFAULT_REDACTION_CONFIG, + type Redactor, +} from '../src/redaction.js'; +import { FakeSecrets, redactionTokens } from './fake-secrets.js'; + +const ctx = { source: 'test', path: '/tmp/x.jsonl' }; + +function defaults(): Redactor { + return createRedactor(DEFAULT_REDACTION_CONFIG); +} + +/** Assert `secret` is gone, the expected token is present, and findings name the rule. */ +function expectRedacted(redactor: Redactor, text: string, secret: string, ruleId: string) { + const result = redactor.redact(text, ctx); + expect(result.text).not.toContain(secret); + expect(result.text).toContain(`[REDACTED:${ruleId}]`); + expect(result.findings.find(f => f.ruleId === ruleId)?.count ?? 0).toBeGreaterThan(0); + // Findings carry rule IDs and counts, never the matched value. + expect(JSON.stringify(result.findings)).not.toContain(secret); + return result; +} + +describe('redaction: default rules (positive)', () => { + let fake: FakeSecrets; + let redactor: Redactor; + + beforeEach(() => { + fake = new FakeSecrets(42); + redactor = defaults(); + }); + + it('azure-client-secret in pasted `az ad sp credential reset` output', () => { + const secret = fake.azureClientSecret(); + const text = `{\n "appId": "${fake.guid()}",\n "password": "${secret}",\n "tenant": "${fake.guid()}"\n}`; + expectRedacted(redactor, text, secret, 'azure-client-secret'); + }); + + it('azure-client-secret bare in prose', () => { + const secret = fake.azureClientSecret(); + expectRedacted(redactor, `use ${secret} as the client secret`, secret, 'azure-client-secret'); + }); + + it('azure-client-secret at the end of a sentence', () => { + const secret = fake.azureClientSecret(); + const out = expectRedacted(redactor, `The client secret is ${secret}.`, secret, 'azure-client-secret'); + expect(out.text).toBe('The client secret is [REDACTED:azure-client-secret].'); + }); + + it('azure-storage-key in `az storage account keys list` output', () => { + const secret = fake.azureStorageKey(); + const text = `[\n {\n "keyName": "key1",\n "permissions": "FULL",\n "value": "${secret}"\n }\n]`; + expectRedacted(redactor, text, secret, 'azure-storage-key'); + }); + + it('connection-string-secret redacts only the value, keeping account and endpoint searchable', () => { + const key = fake.azureStorageKey(); + const text = `DefaultEndpointsProtocol=https;AccountName=contosodata;AccountKey=${key};EndpointSuffix=core.windows.net`; + const result = expectRedacted(redactor, text, key, 'connection-string-secret'); + expect(result.text).toContain('AccountName=contosodata'); + expect(result.text).toContain('EndpointSuffix=core.windows.net'); + }); + + it('connection-string-secret in an appsettings.json blob (SQL Server password field)', () => { + const pw = fake.password(); + const text = JSON.stringify({ + ConnectionStrings: { + Default: `Server=tcp:contoso-sql.database.windows.net,1433;Database=orders;User ID=app;Password=${pw};Encrypt=True;`, + }, + }, null, 2); + const result = expectRedacted(redactor, text, pw, 'connection-string-secret'); + expect(result.text).toContain('contoso-sql.database.windows.net'); + expect(result.text).toContain('Database=orders'); + }); + + it('connection-string-secret for Service Bus SharedAccessKey', () => { + const key = fake.chars('ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/', 43) + '='; + const text = `Endpoint=sb://contoso.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=${key}`; + const result = expectRedacted(redactor, text, key, 'connection-string-secret'); + expect(result.text).toContain('SharedAccessKeyName=RootManageSharedAccessKey'); + }); + + it('connection-string-secret keywords are case-insensitive, as in ADO.NET', () => { + // Letters only: secret-assignment needs a digit, so this rule is the only catch. + const pw = fake.chars('abcdefghijklmnopqrstuvwxyz', 19); + for (const keyword of ['password', 'PASSWORD', 'pwd', 'accountkey']) { + const result = expectRedacted(redactor, `Server=db;Database=app;${keyword}=${pw};`, pw, 'connection-string-secret'); + expect(result.text).toContain(`${keyword}=[REDACTED:connection-string-secret];`); + } + }); + + it('azure-sas-token redacts the sig while keeping the blob URL', () => { + const sig = fake.sasSignature(); + const text = `https://contoso.blob.core.windows.net/backups/db.bak?sv=2022-11-02&ss=b&srt=co&sp=rl&se=2026-12-31T00:00:00Z&sig=${sig}`; + const result = expectRedacted(redactor, text, sig, 'azure-sas-token'); + expect(result.text).toContain('https://contoso.blob.core.windows.net/backups/db.bak?sv=2022-11-02'); + }); + + it('private-key-block (multi-line, PEM)', () => { + const block = fake.privateKeyBlock('RSA'); + const body = block.split('\n')[2]; + const result = expectRedacted(redactor, `cat key.pem\n${block}\n$ `, body, 'private-key-block'); + expect(result.text).not.toContain('PRIVATE KEY'); + }); + + it('private-key-block (OPENSSH, escaped newlines as in nested JSON)', () => { + const block = fake.privateKeyBlock('OPENSSH').replace(/\n/g, '\\n'); + const body = block.split('\\n')[3]; + expectRedacted(redactor, `{"content":"${block}"}`, body, 'private-key-block'); + }); + + it('private-key-block truncated (no END marker) is still redacted to end of text', () => { + const block = fake.privateKeyBlock('EC'); + const truncated = block.split('\n').slice(0, 4).join('\n'); + const body = truncated.split('\n')[2]; + expectRedacted(redactor, truncated, body, 'private-key-block'); + }); + + it('jwt (Azure access token from `az account get-access-token`)', () => { + const token = fake.jwt(); + const text = `{\n "accessToken": "${token}",\n "expiresOn": "2026-10-04 12:00:00.000000",\n "tokenType": "Bearer"\n}`; + const result = redactor.redact(text, ctx); + expect(result.text).not.toContain(token); + expect(redactionTokens(result.text).length).toBeGreaterThan(0); + }); + + it('bearer-token in an Authorization header', () => { + const token = fake.bearerOpaque(); + expectRedacted(redactor, `curl -H "Authorization: Bearer ${token}" https://api.example.com`, token, 'bearer-token'); + }); + + it('basic-auth in an Authorization header', () => { + const token = fake.basicAuth(); + expectRedacted(redactor, `Authorization: Basic ${token}`, token, 'basic-auth'); + }); + + it('anthropic-api-key', () => { + const key = fake.anthropicKey(); + expectRedacted(redactor, `export ANTHROPIC_API_KEY=${key}`, key, 'anthropic-api-key'); + }); + + it('openai-api-key', () => { + const key = fake.openAiKey(); + expectRedacted(redactor, `OPENAI_API_KEY="${key}"`, key, 'openai-api-key'); + }); + + it('github-token (classic) and fine-grained PAT', () => { + const classic = fake.githubToken(); + expectRedacted(redactor, `git remote set-url origin https://${classic}@github.com/o/r.git`, classic, 'github-token'); + const fine = fake.githubFineGrainedPat(); + expectRedacted(redactor, `GH_TOKEN=${fine}`, fine, 'github-token'); + }); + + it('aws-access-key-id and aws-secret-access-key in ~/.aws/credentials', () => { + const id = fake.awsAccessKeyId(); + const secret = fake.awsSecretAccessKey(); + const text = `[default]\naws_access_key_id = ${id}\naws_secret_access_key = ${secret}\n`; + expectRedacted(redactor, text, id, 'aws-access-key-id'); + expectRedacted(redactor, text, secret, 'aws-secret-access-key'); + }); + + it('slack-token, google-api-key, npm-token', () => { + const slack = fake.slackToken(); + expectRedacted(redactor, `SLACK_BOT_TOKEN=${slack}`, slack, 'slack-token'); + const google = fake.googleApiKey(); + expectRedacted(redactor, `key=${google}&q=coffee`, google, 'google-api-key'); + const npm = fake.npmToken(); + expectRedacted(redactor, `//registry.npmjs.org/:_authToken=${npm}`, npm, 'npm-token'); + }); + + it('url-credentials redacts only the password', () => { + const pw = fake.password().replace(/[#%^*!]/g, 'x'); + const text = `DATABASE_URL=postgres://app_user:${pw}@db.internal:5432/orders`; + const result = expectRedacted(redactor, text, pw, 'url-credentials'); + expect(result.text).toContain('app_user'); + expect(result.text).toContain('@db.internal:5432/orders'); + }); + + it('secret-assignment covers env, YAML, and JSON shapes (decrypted SOPS output)', () => { + const a = fake.password(); + const b = fake.password(); + const c = fake.password(); + const yaml = `database:\n host: db.internal\n password: ${a}\nazure:\n client_secret: "${b}"\n`; + const env = `AZURE_CLIENT_SECRET=${c}`; + for (const [text, secret] of [[yaml, a], [yaml, b], [env, c]] as const) { + const result = redactor.redact(text, ctx); + expect(result.text).not.toContain(secret); + } + expect(redactor.redact(yaml, ctx).text).toContain('host: db.internal'); + }); + + it('quoted-secret-assignment redacts a JSON "clientSecret" field', () => { + const secret = fake.password(); + expectRedacted(redactor, JSON.stringify({ clientId: fake.guid(), clientSecret: secret }), secret, 'quoted-secret-assignment'); + }); +}); + +describe('redaction: allowlist and false-positive resistance (negative)', () => { + let fake: FakeSecrets; + let redactor: Redactor; + + beforeEach(() => { + fake = new FakeSecrets(7); + redactor = defaults(); + }); + + function expectUntouched(text: string) { + const result = redactor.redact(text, ctx); + expect(result.text).toBe(text); + expect(result.findings).toEqual([]); + } + + it('git SHAs (full and short) survive', () => { + const sha = fake.gitSha(); + expectUntouched(`commit ${sha}\nAuthor: someone\n\n fix: thing (${sha.slice(0, 7)})`); + }); + + it('a git SHA assigned to a token-like key is still allowlisted', () => { + const sha = fake.gitSha(); + expectUntouched(`token: ${sha}`); + }); + + it('GUIDs (tenant, client, object IDs) survive', () => { + expectUntouched(JSON.stringify({ tenantId: fake.guid(), clientId: fake.guid(), objectId: fake.guid() })); + }); + + it('ARM resource IDs survive', () => { + expectUntouched( + `/subscriptions/${fake.guid()}/resourceGroups/rg-prod-eastus/providers/Microsoft.Storage/storageAccounts/contosodata` + ); + }); + + it('error codes and HTTP statuses survive', () => { + expectUntouched('AADSTS7000215: Invalid client secret provided. Status: 401 Unauthorized. AuthorizationFailed (403)'); + }); + + it('base64 image data survives', () => { + expectUntouched(JSON.stringify({ type: 'image', source: { type: 'base64', media_type: 'image/png', data: 'iVBORw0KGgo' + fake.base64Blob(20000) + '==' } })); + }); + + it('npm integrity hashes (sha512, 88 chars) survive', () => { + expectUntouched(`"integrity": "sha512-${fake.base64Blob(86)}=="`); + }); + + it('minified JS survives', () => { + expectUntouched( + 'function a(e,t){var n=e.password,r=t.token;return n&&r?{password:n,token:r,expires_in:3600}:null}' + + 'const s=new URLSearchParams({grant_type:"client_credentials"});' + ); + }); + + it('code that reads secrets (not literals) survives', () => { + expectUntouched('const password = getPassword();\nconst token = process.env.GITHUB_TOKEN;\npassword = os.environ["DB_PASSWORD"]'); + }); + + it('member-access references to secrets survive, even with digits (env.AUTH0_CLIENT_SECRET)', () => { + expectUntouched('client_id: env.AUTH0_CLIENT_ID,\n client_secret: env.AUTH0_CLIENT_SECRET,\n token: this.config.apiToken2;'); + }); + + it('placeholders and SOPS-encrypted values survive', () => { + expectUntouched('password: ${DB_PASSWORD}\nclient_secret: \napi_key: ENC[AES256_GCM,data:abc,iv:def,tag:ghi,type:str]'); + }); + + it('PWD and other env noise survive', () => { + expectUntouched('PWD=/home/user1/src/project2\nOLDPWD=/home/user1\nmax_tokens: 4096\ntokenizer: bert-base-uncased'); + }); + + it('Markdown about connection-string keywords survives', () => { + expectUntouched('Set `AccountKey=`, `SharedAccessKey=`, `Password=` or `Pwd=` in the connection string.'); + }); + + it('prose about tokens and passwords survives', () => { + expectUntouched('Rotate the bearer token every 90 days. Basic authentication is disabled. The password policy requires 14 characters.'); + }); +}); + +describe('redaction: idempotency', () => { + it('redacting already-redacted text is a no-op and tokens match no rule', () => { + const fake = new FakeSecrets(99); + const redactor = defaults(); + const text = [ + `AccountKey=${fake.azureStorageKey()}`, + `password: ${fake.password()}`, + `Authorization: Bearer ${fake.jwt()}`, + fake.privateKeyBlock(), + `client secret ${fake.azureClientSecret()}`, + `https://u:${fake.password().replace(/[#%^*!]/g, 'y')}@host/x`, + ].join('\n'); + const once = redactor.redact(text, ctx); + expect(once.findings.length).toBeGreaterThan(0); + const twice = redactor.redact(once.text, ctx); + expect(twice.text).toBe(once.text); + expect(twice.findings).toEqual([]); + }); + + it('every token for every default rule id is inert', () => { + const redactor = defaults(); + for (const id of redactor.ruleIds) { + for (const shape of [`[REDACTED:${id}]`, `password: [REDACTED:${id}]`, `Bearer [REDACTED:${id}]`, `AccountKey=[REDACTED:${id}];`]) { + const result = redactor.redact(shape, ctx); + expect(result.findings, `${id} in ${shape}`).toEqual([]); + } + } + }); + + it('a match that runs into an existing token redacts the part outside it, keeping the token', () => { + // Text already partly redacted, e.g. by an older rule set before `redact --rewrite`. + const redactor = defaults(); + const fake = new FakeSecrets(12); + const prefix = fake.chars('abcdefghijklmnopqrstuvwxyz0123456789', 10); + const once = redactor.redact(`Server=db;Password=${prefix}[REDACTED:jwt];`, ctx); + expect(once.text).toBe('Server=db;Password=[REDACTED:connection-string-secret][REDACTED:jwt];'); + const twice = redactor.redact(once.text, ctx); + expect(twice.text).toBe(once.text); + expect(twice.findings).toEqual([]); + }); +}); + +describe('redaction: entropy fallback', () => { + it('is off by default', () => { + const fake = new FakeSecrets(3); + const blob = fake.opaqueHighEntropy(); + const result = defaults().redact(`the signing value is ${blob}`, ctx); + expect(result.text).toContain(blob); + }); + + it('when enabled, fires near a keyword but not elsewhere or on allowlisted shapes', () => { + const fake = new FakeSecrets(4); + const redactor = createRedactor({ + ...DEFAULT_REDACTION_CONFIG, + entropy: { ...DEFAULT_REDACTION_CONFIG.entropy, enabled: true }, + }); + const blob = fake.opaqueHighEntropy(48); + const near = redactor.redact(`signing key is ${blob}`, ctx); + expect(near.text).not.toContain(blob); + expect(near.text).toContain('[REDACTED:high-entropy]'); + + const far = redactor.redact(`the build artifact digest ${fake.opaqueHighEntropy(48)}`, ctx); + expect(far.findings).toEqual([]); + + const sha = fake.gitSha(); + expect(redactor.redact(`key ${sha}`, ctx).text).toContain(sha); + }); +}); + +describe('redaction: JSONL line handling', () => { + const redactor = defaults(); + const fake = new FakeSecrets(11); + + it('redacts string values inside JSON and keeps the line valid JSON', () => { + const key = fake.azureStorageKey(); + const line = JSON.stringify({ + type: 'user', + message: { role: 'user', content: [{ type: 'tool_result', tool_use_id: 't1', content: `AccountKey=${key}` }] }, + uuid: 'u1', + }); + const out = redactJsonlLine(line, redactor, ctx); + expect(out).not.toContain(key); + const parsed = JSON.parse(out); + expect(parsed.uuid).toBe('u1'); + expect(parsed.message.content[0].content).toBe('AccountKey=[REDACTED:connection-string-secret]'); + }); + + it('redacts tool inputs (nested objects) too', () => { + const pw = fake.password(); + const line = JSON.stringify({ + type: 'assistant', + message: { role: 'assistant', content: [{ type: 'tool_use', id: 't1', name: 'Bash', input: { command: `mysql -u root --password=${pw}` } }] }, + }); + const out = redactJsonlLine(line, redactor, ctx); + expect(out).not.toContain(pw); + expect(() => JSON.parse(out)).not.toThrow(); + }); + + it('returns unchanged lines byte-for-byte (formatting preserved)', () => { + const line = '{"type":"user", "message":{"role":"user","content":"caf\\u00e9 hello"}}'; + expect(redactJsonlLine(line, redactor, ctx)).toBe(line); + }); + + it('redacts non-JSON lines (e.g. a torn last line) as raw text', () => { + const key = fake.azureStorageKey(); + const torn = `{"type":"user","message":{"content":"AccountKey=${key}`; + const out = redactJsonlLine(torn, redactor, ctx); + expect(out).not.toContain(key); + }); + + it('preserves CRLF line endings', () => { + const pw = fake.password(); + const line = JSON.stringify({ content: `password: ${pw}` }) + '\r'; + const out = redactJsonlLine(line, redactor, ctx); + expect(out.endsWith('\r')).toBe(true); + expect(out).not.toContain(pw); + }); + + it('tallies findings by rule id', () => { + const tally = new FindingsTally(); + redactJsonlLine(JSON.stringify({ a: `password: ${fake.password()}`, b: `password: ${fake.password()}` }), redactor, ctx, tally); + expect(tally.toArray()).toEqual([{ ruleId: 'secret-assignment', count: 2 }]); + expect(tally.total).toBe(2); + }); +}); + +describe('redaction: copyFileRedacted', () => { + let dir: string; + beforeEach(() => { dir = mkdtempSync(join(tmpdir(), 'episodic-memory-redact-copy-')); }); + afterEach(() => { rmSync(dir, { recursive: true, force: true }); }); + + it('streams a file, keeps one output line per input line, and keeps the trailing-newline shape', () => { + const fake = new FakeSecrets(5); + const redactor = defaults(); + const secrets: string[] = []; + const lines: string[] = []; + for (let i = 0; i < 3000; i++) { + if (i % 500 === 0) { + const s = fake.azureClientSecret(); + secrets.push(s); + lines.push(JSON.stringify({ type: 'user', i, message: { content: `secret is ${s}` } })); + } else { + lines.push(JSON.stringify({ type: 'assistant', i, message: { content: 'x'.repeat(i % 700) } })); + } + } + const src = join(dir, 'src.jsonl'); + const dest = join(dir, 'dest.jsonl'); + writeFileSync(src, lines.join('\n')); // no trailing newline + const tally = new FindingsTally(); + copyFileRedacted(src, dest, redactor, ctx, tally); + const out = readFileSync(dest, 'utf-8'); + for (const s of secrets) expect(out).not.toContain(s); + expect(out.split('\n').length).toBe(lines.length); + expect(out.endsWith('\n')).toBe(false); + expect(tally.total).toBe(secrets.length); + out.split('\n').forEach(l => expect(() => JSON.parse(l)).not.toThrow()); + }); + + it('handles multi-byte UTF-8 across chunk boundaries', () => { + const redactor = defaults(); + const src = join(dir, 'utf8.jsonl'); + const dest = join(dir, 'utf8-out.jsonl'); + const line = JSON.stringify({ content: '日本語テキスト🙂'.repeat(200_000) }); + writeFileSync(src, line + '\n' + line + '\n'); + copyFileRedacted(src, dest, redactor, ctx); + expect(readFileSync(dest, 'utf-8')).toBe(readFileSync(src, 'utf-8')); + }); + + // Windows ignores POSIX mode bits. + it.skipIf(process.platform === 'win32')('creates dest with the source mode, not the 0666 default', () => { + const src = join(dir, 'private.jsonl'); + const dest = join(dir, 'private-out.jsonl'); + writeFileSync(src, '{"type":"user"}\n'); + chmodSync(src, 0o600); + copyFileRedacted(src, dest, defaults(), ctx); + expect(statSync(dest).mode & 0o777).toBe(0o600); + }); +}); + +describe('redaction: settings and rules loading', () => { + let dir: string; + const saved = { ...process.env }; + + beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), 'episodic-memory-redact-cfg-')); + process.env.EPISODIC_MEMORY_CONFIG_DIR = join(dir, 'config'); + mkdirSync(process.env.EPISODIC_MEMORY_CONFIG_DIR, { recursive: true }); + delete process.env.EPISODIC_MEMORY_REDACTION; + delete process.env.EPISODIC_MEMORY_REDACTION_RULES; + delete process.env.EPISODIC_MEMORY_REDACTION_STRICT; + }); + + afterEach(() => { + process.env = { ...saved }; + rmSync(dir, { recursive: true, force: true }); + }); + + it('is on and strict by default', () => { + expect(getRedactionSettings(process.env)).toMatchObject({ enabled: true, strict: true }); + }); + + it('EPISODIC_MEMORY_REDACTION=off disables it', () => { + process.env.EPISODIC_MEMORY_REDACTION = 'off'; + expect(getRedactionSettings(process.env).enabled).toBe(false); + expect(loadRedactor(process.env)).toBeNull(); + }); + + it('loads the bundled defaults when no rules file exists', () => { + const redactor = loadRedactor(process.env)!; + expect(redactor.ruleIds).toEqual(DEFAULT_REDACTION_CONFIG.rules.map(r => r.id)); + }); + + it('picks up redaction-rules.json from the config dir and merges it with defaults', () => { + writeFileSync(join(process.env.EPISODIC_MEMORY_CONFIG_DIR!, 'redaction-rules.json'), JSON.stringify({ + rules: [{ id: 'internal-ticket-secret', pattern: 'TKT-SECRET-[0-9]{6}' }], + disableRules: ['basic-auth'], + })); + const redactor = loadRedactor(process.env)!; + expect(redactor.ruleIds).toContain('internal-ticket-secret'); + expect(redactor.ruleIds).toContain('azure-storage-key'); + expect(redactor.ruleIds).not.toContain('basic-auth'); + expect(redactor.redact('ref TKT-SECRET-123456', ctx).text).toBe('ref [REDACTED:internal-ticket-secret]'); + }); + + it('EPISODIC_MEMORY_REDACTION_RULES points at a custom file; includeDefaults:false replaces the defaults', () => { + const file = join(dir, 'custom.json'); + writeFileSync(file, JSON.stringify({ includeDefaults: false, rules: [{ id: 'only-this', pattern: 'zzz[0-9]+' }] })); + process.env.EPISODIC_MEMORY_REDACTION_RULES = file; + const redactor = loadRedactor(process.env)!; + expect(redactor.ruleIds).toEqual(['only-this']); + }); + + it('strict mode throws RedactionConfigError on a corrupt rules file', () => { + const file = join(dir, 'bad.json'); + writeFileSync(file, '{ "rules": [ this is not json'); + process.env.EPISODIC_MEMORY_REDACTION_RULES = file; + expect(() => loadRedactor(process.env)).toThrow(RedactionConfigError); + }); + + it('strict mode throws when the custom rules path does not exist', () => { + process.env.EPISODIC_MEMORY_REDACTION_RULES = join(dir, 'missing.json'); + expect(() => loadRedactor(process.env)).toThrow(RedactionConfigError); + }); + + it('non-strict mode warns and returns null (pass-through) on a corrupt rules file', () => { + const file = join(dir, 'bad.json'); + writeFileSync(file, 'nope'); + process.env.EPISODIC_MEMORY_REDACTION_RULES = file; + process.env.EPISODIC_MEMORY_REDACTION_STRICT = '0'; + const warn = vi.fn(); + expect(loadRedactor(process.env, warn)).toBeNull(); + expect(warn).toHaveBeenCalled(); + }); + + it('rejects invalid rules: bad regex, bad id, empty-matching pattern, bad secretGroup', () => { + const bad = [ + { id: 'bad-regex', pattern: '(unclosed' }, + { id: 'Bad Id!', pattern: 'abc' }, + { id: 'empty-match', pattern: 'a*' }, + { id: 'bad-group', pattern: 'abc', secretGroup: 2 }, + ]; + for (const rule of bad) { + expect(() => loadRedactionConfig({ includeDefaults: false, rules: [rule] } as any), rule.id).toThrow(RedactionConfigError); + } + }); + + it('rule-load errors never echo text being redacted (only config details)', () => { + const file = join(dir, 'bad.json'); + writeFileSync(file, JSON.stringify({ rules: [{ id: 'x', pattern: '(' }] })); + process.env.EPISODIC_MEMORY_REDACTION_RULES = file; + try { + loadRedactor(process.env); + expect.unreachable(); + } catch (e) { + expect((e as Error).message).toContain('x'); + } + }); +}); + +describe('redaction: key context (.NET / Azure shapes)', () => { + let fake: FakeSecrets; + let redactor: Redactor; + beforeEach(() => { + fake = new FakeSecrets(2112); + redactor = defaults(); + }); + + function expectGone(text: string, secret: string, ruleId?: string) { + const result = redactor.redact(text, ctx); + expect(result.text, text).not.toContain(secret); + if (ruleId) expect(result.text).toContain(`[REDACTED:${ruleId}]`); + return result; + } + + it('appsettings.json fields, no digit required when the key is quoted JSON', () => { + const pw = fake.passphrase(); + const text = JSON.stringify({ AzureAd: { TenantId: fake.guid(), ClientId: fake.guid(), ClientSecret: pw } }, null, 2); + const r = expectGone(text, pw, 'quoted-secret-assignment'); + expect(r.text).toContain('"TenantId"'); + }); + + it('C# object initializers and locals', () => { + const a = fake.passphrase(); + const b = fake.passphrase(); + expectGone(`var cred = new ClientSecretCredential(tenant, client) { ClientSecret = "${a}", };`, a, 'quoted-secret-assignment'); + expectGone(`string apiKey = "${b}";`, b, 'quoted-secret-assignment'); + }); + + it('`az webapp config appsettings list` name/value pairs (name first and value first)', () => { + const a = fake.passphrase(); + const b = fake.passphrase(); + const text = JSON.stringify([ + { name: 'WEBSITE_RUN_FROM_PACKAGE', slotSetting: false, value: '1' }, + { name: 'Stripe__ApiKey', slotSetting: false, value: a }, + { value: b, slotSetting: true, name: 'DB_PASSWORD' }, + ], null, 2); + const r = expectGone(text, a, 'name-value-secret'); + expect(r.text).not.toContain(b); + expect(r.text).toContain('WEBSITE_RUN_FROM_PACKAGE'); + expect(r.text).toContain('"value": "1"'); + }); + + it('Kubernetes env entries', () => { + const a = fake.passphrase(); + expectGone(`env:\n - {"name": "ConnectionStrings__Redis", "value": "redis:6380"}\n - {"name": "JWT_SIGNING_KEY", "value": "${a}"}`, a); + }); + + it('`az keyvault secret show` value (any secret name, nested attributes)', () => { + const a = fake.passphrase(); + const text = JSON.stringify({ + attributes: { created: '2026-01-01T00:00:00+00:00', enabled: true, recoveryLevel: 'Recoverable' }, + contentType: null, + id: 'https://contoso-kv.vault.azure.net/secrets/StorageThing/0123456789abcdef0123456789abcdef', + name: 'StorageThing', + tags: {}, + value: a, + }, null, 2); + const r = expectGone(text, a, 'azure-keyvault-secret'); + expect(r.text).toContain('contoso-kv.vault.azure.net/secrets/StorageThing'); + }); + + it('web.config appSettings in either attribute order', () => { + const a = fake.passphrase(); + const b = fake.passphrase(); + expectGone(`\n \n \n \n`, a, 'xml-appsettings-secret'); + const r = redactor.redact(``, ctx); + expect(r.text).not.toContain(b); + expect(redactor.redact('', ctx).findings).toEqual([]); + }); + + it('publish profiles and XML secret attributes/elements', () => { + const a = fake.passphrase(); + const b = fake.passphrase(); + const r = expectGone(``, a, 'xml-secret-attribute'); + expect(r.text).toContain('userName="$contoso-app"'); + expect(r.text).toContain('https://contoso-app.azurewebsites.net'); + expectGone(`${b}`, b, 'xml-secret-element'); + }); + + it('a GUID in a password slot is redacted (legacy create-for-rbac), but GUID IDs elsewhere are kept', () => { + const pw = fake.guid(); + const tenant = fake.guid(); + const r = expectGone(JSON.stringify({ appId: fake.guid(), password: pw, tenant }), pw); + expect(r.text).toContain(tenant); + }); + + it('leaves placeholders, labels and non-secret neighbours alone', () => { + const untouched = [ + '{"ClientSecret": "#{ClientSecret}#", "ApiKey": "__API_KEY__", "Password": "$(DbPassword)", "Token": "${TOKEN}", "Secret": ""}', + 'ErrorMessage = "Invalid password or username";', + '{"tokenType": "Bearer", "secretName": "db-password", "maxTokens": "4096", "passwordPolicy": "strict"}', + 'options.Password = configuration["Db:Password"];', + '', + '{"name": "TokenEndpoint", "value": "https://login.microsoftonline.com/common/oauth2/v2.0/token"}', + '' + 'a1b2c3d4-0000-1111-2222-333344445555' + '', + 'PWD="/home/user1/src"', + ]; + for (const text of untouched) { + const r = redactor.redact(text, ctx); + expect(r.findings, text).toEqual([]); + } + }); +}); + +describe('redaction: structured JSON (field names as context, keys redacted)', () => { + const redactor = createRedactor(DEFAULT_REDACTION_CONFIG); + const fake = new FakeSecrets(4242); + + it('redacts a value whose own field name is secret-looking, with no shape match', () => { + const pw = fake.passphrase(); + const line = JSON.stringify({ type: 'user', toolUseResult: { structuredContent: { clientId: 'app', clientSecret: pw } } }); + const out = redactJsonlLine(line, redactor, ctx); + expect(out).not.toContain(pw); + expect(JSON.parse(out).toolUseResult.structuredContent).toEqual({ clientId: 'app', clientSecret: '[REDACTED:secret-field]' }); + }); + + it('redacts the value of a {name, value} object whose name is secret-looking', () => { + const pw = fake.passphrase(); + const line = JSON.stringify({ result: [{ name: 'Db__Password', value: pw }, { name: 'Region', value: 'eastus' }] }); + const out = JSON.parse(redactJsonlLine(line, redactor, ctx)); + expect(out.result).toEqual([{ name: 'Db__Password', value: '[REDACTED:secret-field]' }, { name: 'Region', value: 'eastus' }]); + }); + + it('redacts a structured Key Vault secret bundle value', () => { + const pw = fake.passphrase(); + const line = JSON.stringify({ result: { id: 'https://kv.vault.azure.net/secrets/Anything/1', value: pw, attributes: { enabled: true } } }); + expect(redactJsonlLine(line, redactor, ctx)).not.toContain(pw); + }); + + it('redacts secrets used as object keys', () => { + const token = fake.githubToken(); + const line = JSON.stringify({ cache: { [token]: { user: 'octocat' } } }); + const out = redactJsonlLine(line, redactor, ctx); + expect(out).not.toContain(token); + expect(JSON.parse(out).cache['[REDACTED:github-token]']).toEqual({ user: 'octocat' }); + }); + + it('leaves harness structure alone (thinking signatures, usage, token_count, apiKeySource)', () => { + const lines = [ + JSON.stringify({ type: 'assistant', message: { content: [{ type: 'thinking', thinking: 'hmm', signature: fake.base64Blob(200) }], usage: { input_tokens: 12, output_tokens: 3 } } }), + JSON.stringify({ type: 'event_msg', payload: { type: 'token_count', info: { total_token_usage: { input_tokens: 1 } } } }), + JSON.stringify({ type: 'system', subtype: 'init', apiKeySource: 'none', tokenizer: 'cl100k' }), + JSON.stringify({ secretName: 'db-password', passwordPolicy: 'strict', password: '${DB_PASSWORD}', token: '' }), + ]; + for (const line of lines) expect(redactJsonlLine(line, redactor, ctx), line).toBe(line); + }); + + it('redacts a secret field whole when a text rule already redacted only part of it', () => { + const word = fake.chars('abcdefghijklmnopqrstuvwxyz', 12); + const line = JSON.stringify({ password: `${word} ${fake.jwt()}` }); + const out = redactJsonlLine(line, redactor, ctx); + expect(out).not.toContain(word); + expect(JSON.parse(out).password).toBe('[REDACTED:secret-field]'); + }); + + it('is idempotent on structured redactions', () => { + const line = JSON.stringify({ a: { clientSecret: fake.passphrase() }, b: [{ name: 'API_KEY', value: fake.passphrase() }] }); + const once = redactJsonlLine(line, redactor, ctx); + expect(redactJsonlLine(once, redactor, ctx)).toBe(once); + }); + + it('can be turned off via secretFields.enabled=false', () => { + const r = createRedactor({ ...DEFAULT_REDACTION_CONFIG, secretFields: { ...DEFAULT_REDACTION_CONFIG.secretFields, enabled: false } }); + const pw = fake.passphrase(); + expect(redactJsonlLine(JSON.stringify({ x: { clientSecret: pw } }), r, ctx)).toContain(pw); + }); +}); diff --git a/test/search-admission.test.ts b/test/search-admission.test.ts new file mode 100644 index 00000000..3f796870 --- /dev/null +++ b/test/search-admission.test.ts @@ -0,0 +1,216 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { initDatabase, insertExchange } from '../src/db.js'; +import { searchConversations } from '../src/search.js'; +import { readAdmittedConversation, shouldSkipConversation } from '../src/record-admission.js'; + +vi.mock('../src/embeddings.js', () => ({ + initEmbeddings: vi.fn(async () => {}), + generateQueryEmbedding: vi.fn(async () => new Array(384).fill(0.1)), +})); +const marker = 'DO NOT INDEX THIS CHAT'; +const session = '00000000-1111-4222-8333-444444444444'; +const message = (role: string, text: string) => ({ type: 'response_item', payload: { + type: 'message', role, content: [{ type: 'input_text', text }], +} }); +let root: string; +let previous: NodeJS.ProcessEnv; +beforeEach(() => { + previous = { ...process.env }; + root = fs.mkdtempSync(path.join(os.tmpdir(), 'em-search-admission-')); + process.env.EPISODIC_MEMORY_CONFIG_DIR = root; + process.env.TEST_DB_PATH = path.join(root, 'test.db'); + delete process.env.EPISODIC_MEMORY_DB_PATH; +}); +afterEach(() => { vi.restoreAllMocks(); process.env = previous; fs.rmSync(root, { recursive: true, force: true }); }); +function seed(id: string, timestamp = '2026-01-01T00:00:00Z') { + const file = path.join(root, `${id}-${session}.jsonl`); + fs.writeFileSync(file, [ + { type: 'session_meta', payload: { id: session } }, + message('user', `needle ${id}`), message('assistant', 'safe answer'), + ].map(x => JSON.stringify(x)).join('\n') + '\n'); + const db = initDatabase(); + insertExchange(db, { id, project: 'fixture', timestamp, archivePath: file, + lineStart: 2, lineEnd: 3, userMessage: `needle ${id}`, assistantMessage: 'safe answer', sessionId: session, + }, new Array(384).fill(0.1)); + db.close(); return file; +} +function exclude(file: string, start = 2, end = 3) { + fs.writeFileSync(path.join(root, 'record-exclusions.json'), JSON.stringify({ version: 1, + exchanges: [{ session_id: session, transcript: path.basename(file), line_start: start, line_end: end }], tool_calls: [], + })); +} + +describe('decoded user instruction markers', () => { + it('retains physical line coordinates while loading only a requested range', async () => { + const file = path.join(root, 'range.jsonl'); + fs.writeFileSync(file, [message('user', 'outside'), message('assistant', 'selected \u2028 text'), message('user', 'outside after')].map(x => JSON.stringify(x)).join('\r\n')); + const result = await readAdmittedConversation(file, 2, 2); + expect(result.split('\n')).toHaveLength(3); + expect(result.split('\n')[0]).toBe(''); + expect(result.split('\n')[1]).toContain('selected'); + expect(result).not.toContain('outside'); + fs.appendFileSync(file, '\n' + JSON.stringify(message('user', marker))); + expect(await readAdmittedConversation(file, 2, 2)).toBe(''); + }); + + it.each([ + ['assistant quote', message('assistant', marker)], + ['tool output', { type: 'response_item', payload: { type: 'function_call_output', output: marker } }], + ['Claude tool result', { type: 'user', message: { role: 'user', content: [{ type: 'tool_result', content: marker }] } }], + ])('does not opt out for %s', async (_, record) => { + const file = path.join(root, 'quoted.jsonl'); + const bytes = JSON.stringify(record) + '\n'; fs.writeFileSync(file, bytes); + expect(shouldSkipConversation(file)).toBe(false); + expect(await readAdmittedConversation(file)).toContain(marker); + expect(fs.readFileSync(file, 'utf8')).toBe(bytes); + }); + it.each([ + ['Codex response', message('user', marker)], + ['Codex event', { type: 'event_msg', payload: { type: 'user_message', message: marker } }], + ['Claude', { type: 'user', message: { role: 'user', content: marker } }], + ['Cursor', { role: 'user', message: { content: marker } }], + ['OMP', { type: 'message', message: { role: 'user', content: [{ type: 'text', text: marker }] } }], + ['opencode', { type: 'opencode_message', message: { role: 'user' }, parts: [{ type: 'text', text: marker }] }], + ])('honors JSON-escaped opt out in %s', async (_, record) => { + const file = path.join(root, 'escaped.jsonl'); + fs.writeFileSync(file, JSON.stringify(record).replaceAll('<', '\\u003c')); + expect(shouldSkipConversation(file)).toBe(true); + expect(await readAdmittedConversation(file)).toBe(''); + }); +}); + +describe('retained search admission', () => { + it('orders admitted vectors by computed distance before applying the result limit', async () => { + seed('far'); seed('near'); seed('middle'); + const db = initDatabase(); + const update = db.prepare('UPDATE vec_exchanges SET embedding = ? WHERE id = ?'); + for (const [id, coordinate] of [['far', 0.8], ['near', 0.1], ['middle', 0.3]] as const) { + update.run(Buffer.from(new Float32Array(new Array(384).fill(coordinate)).buffer), id); + } + db.close(); + + const results = await searchConversations('needle', { mode: 'vector', limit: 2 }); + expect(results.map(result => result.exchange.id)).toEqual(['near', 'middle']); + expect(results[0].similarity).toBeGreaterThan(results[1].similarity!); + }); + + it('applies the sidechain penalty to computed distance rather than storage order', async () => { + seed('sidechain'); seed('primary'); + const db = initDatabase(); + db.prepare('UPDATE exchanges SET is_sidechain = 1 WHERE id = ?').run('sidechain'); + db.close(); + + const results = await searchConversations('needle', { mode: 'vector', limit: 2 }); + expect(results.map(result => result.exchange.id)).toEqual(['primary', 'sidechain']); + expect(results[0].similarity).toBeCloseTo(results[1].similarity!); + }); + + it('finds an admitted vector beyond the native KNN candidate ceiling', async () => { + const file = seed('denied'); seed('safe'); + const db = initDatabase(); + db.transaction(() => { + for (let i = 0; i < 4096; i++) { + insertExchange(db, { id: `denied-${i}`, project: 'fixture', timestamp: '2026-01-01T00:00:00Z', + archivePath: file, lineStart: 2, lineEnd: 3, userMessage: 'needle denied', assistantMessage: 'safe', sessionId: session, + }, new Array(384).fill(0.1)); + } + db.prepare('UPDATE vec_exchanges SET embedding = ? WHERE id = ?') + .run(Buffer.from(new Float32Array(new Array(384).fill(0.2)).buffer), 'safe'); + })(); + db.close(); exclude(file); + expect((await searchConversations('needle', { mode: 'vector', limit: 1 })).map(x => x.exchange.id)).toEqual(['safe']); + }); + + it('never follows a substituted transcript symlink', async () => { + const file = seed('safe'); + fs.renameSync(file, file + '.original'); + fs.symlinkSync(file + '.original', file); + await expect(searchConversations('needle', { mode: 'text' })).rejects.toThrow(/changed during record admission/); + }); + it.each(['text', 'vector', 'both'] as const)('withholds a removed transcript without failing %s search', async mode => { + seed('safe'); const removed = seed('removed', '2026-02-01T00:00:00Z'); + fs.rmSync(removed); + const results = await searchConversations('needle', { mode, limit: 2 }); + expect(results.map(x => x.exchange.id)).toEqual(['safe']); + }); + it('keeps an indexed exchange searchable while its transcript grows during admission', async () => { + const file = seed('safe'); + const original = fs.readSync; + let appended = false; + vi.spyOn(fs, 'readSync').mockImplementation(((...args: Parameters) => { + const count = original(...args); + if (!appended && count > 0) { + appended = true; + fs.appendFileSync(file, JSON.stringify({ type: 'event_msg', payload: { type: 'agent_message', message: 'later' } }) + '\n'); + } + return count; + }) as typeof fs.readSync); + + const results = await searchConversations('needle', { mode: 'text', before: '2026-09-14' }); + expect(appended).toBe(true); + expect(results.map(result => result.exchange.id)).toEqual(['safe']); + }); + it('rejects an indexed-range rewrite even when the transcript also grows', async () => { + const file = seed('safe'); + const summary = file.replace('.jsonl', '-summary.txt'); + fs.writeFileSync(summary, 'summary'); + const offset = fs.readFileSync(file).indexOf('needle safe'); + expect(offset).toBeGreaterThanOrEqual(0); + const original = fs.readFileSync; + let changed = false; + vi.spyOn(fs, 'readFileSync').mockImplementation(((p: any, ...args: any[]) => { + if (p === summary && !changed) { + changed = true; + fs.appendFileSync(file, JSON.stringify({ type: 'event_msg', payload: { type: 'agent_message', message: 'later' } }) + '\n'); + const fd = fs.openSync(file, 'r+'); + try { fs.writeSync(fd, Buffer.from('secret safe'), 0, 11, offset); } + finally { fs.closeSync(fd); } + } + return (original as any)(p, ...args); + }) as typeof fs.readFileSync); + + await expect(searchConversations('needle', { mode: 'text' })).rejects.toThrow(/changed during record admission/); + expect(changed).toBe(true); + }); + it('rejects policy changes while building results', async () => { + const file = seed('safe'); + const summary = file.replace('.jsonl', '-summary.txt'); fs.writeFileSync(summary, 'summary'); + const original = fs.readFileSync; + vi.spyOn(fs, 'readFileSync').mockImplementation(((p: any, ...args: any[]) => { + if (p === summary) exclude(file); + return (original as any)(p, ...args); + }) as typeof fs.readFileSync); + await expect(searchConversations('needle', { mode: 'text' })).rejects.toThrow(/policy changed/); + }); + + it.each(['text', 'vector', 'both'] as const)('applies current exclusions before limit in %s mode', async mode => { + seed('safe'); const denied = seed('denied', '2026-02-01T00:00:00Z'); + exclude(denied); + const results = await searchConversations('needle', { mode, limit: 1 }); + expect(results.map(x => x.exchange.id)).toEqual(['safe']); + expect(await readAdmittedConversation(denied)).not.toContain('needle denied'); + }); + it('rechecks opt-out appended after indexing, but not quoted tool output', async () => { + const file = seed('later'); + fs.appendFileSync(file, JSON.stringify({ type: 'response_item', payload: { type: 'function_call_output', output: marker } }) + '\n'); + expect(await searchConversations('needle', { mode: 'text' })).toHaveLength(1); + fs.appendFileSync(file, JSON.stringify(message('user', marker)).replaceAll('<', '\\u003c') + '\n'); + expect(await searchConversations('needle', { mode: 'text' })).toEqual([]); + }); + it('omits a stale whole-conversation summary when any source record is excluded', async () => { + const file = seed('safe'); + fs.appendFileSync(file, JSON.stringify(message('user', 'excluded summary phrase')) + '\n'); + fs.writeFileSync(file.replace('.jsonl', '-summary.txt'), 'excluded summary phrase'); + exclude(file, 4, 4); + const results = await searchConversations('needle', { mode: 'text' }); + expect(results).toHaveLength(1); + expect(JSON.stringify(results)).not.toContain('excluded summary phrase'); + }); + it('fails closed on invalid exclusion policy', async () => { + seed('safe'); fs.writeFileSync(path.join(root, 'record-exclusions.json'), '{invalid'); + await expect(searchConversations('needle', { mode: 'text' })).rejects.toThrow(/exclusion policy/i); + }); +}); diff --git a/test/setup-isolated-config-cleanup.test.ts b/test/setup-isolated-config-cleanup.test.ts new file mode 100644 index 00000000..db0862c4 --- /dev/null +++ b/test/setup-isolated-config-cleanup.test.ts @@ -0,0 +1,61 @@ +import { afterEach, describe, expect, it } from 'vitest'; +import { spawnSync } from 'node:child_process'; +import { mkdtempSync, mkdirSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const repoRoot = join(dirname(fileURLToPath(import.meta.url)), '..'); +const fixtureRoots: string[] = []; + +afterEach(() => { + for (const root of fixtureRoots.splice(0)) { + rmSync(root, { recursive: true, force: true }); + } +}); + +describe('test configuration isolation cleanup', () => { + it.each([false, true])('removes its own state after a run (failure: %s)', (fail) => { + const root = mkdtempSync(join(tmpdir(), 'episodic-setup-cleanup-')); + fixtureRoots.push(root); + const temporary = join(root, 'temporary'); + const sibling = join(temporary, 'unrelated'); + mkdirSync(sibling, { recursive: true }); + writeFileSync(join(sibling, 'keep.txt'), 'preserved'); + + writeFileSync(join(root, 'vitest.config.mjs'), `export default ${JSON.stringify({ + test: { + include: ['fixture.test.mjs'], + setupFiles: [join(repoRoot, 'test/setup-isolated-config.ts')], + }, + })};`); + writeFileSync(join(root, 'fixture.test.mjs'), ` + import { it, expect } from ${JSON.stringify(join(repoRoot, 'node_modules/vitest/dist/index.js'))}; + import { mkdirSync, writeFileSync } from 'node:fs'; + import { join } from 'node:path'; + it('writes only isolated synthetic state', () => { + const config = process.env.EPISODIC_MEMORY_CONFIG_DIR; + expect(config.startsWith(${JSON.stringify(temporary)})).toBe(true); + mkdirSync(config, { recursive: true }); + writeFileSync(join(config, 'synthetic-state.txt'), 'fixture'); + expect(${fail}).toBe(false); + }); + `); + + const result = spawnSync(process.execPath, [ + join(repoRoot, 'node_modules/vitest/vitest.mjs'), 'run', + '--root', root, '--config', join(root, 'vitest.config.mjs'), + '--maxWorkers=1', '--minWorkers=1', + ], { + cwd: root, + env: { ...process.env, TMPDIR: temporary, TMP: temporary, TEMP: temporary }, + timeout: 15000, + encoding: 'utf-8', + }); + + expect(result.error).toBeUndefined(); + expect(result.status, result.stdout + result.stderr).toBe(fail ? 1 : 0); + expect(readdirSync(temporary).filter(name => name.startsWith('episodic-memory-vitest-'))).toEqual([]); + expect(readFileSync(join(sibling, 'keep.txt'), 'utf-8')).toBe('preserved'); + }); +}); diff --git a/test/setup-isolated-config.ts b/test/setup-isolated-config.ts index 71b3e795..8314b2ca 100644 --- a/test/setup-isolated-config.ts +++ b/test/setup-isolated-config.ts @@ -1,4 +1,5 @@ -import { mkdtempSync } from 'fs'; +import { afterAll } from 'vitest'; +import { mkdtempSync, rmSync } from 'fs'; import { tmpdir } from 'os'; import { join } from 'path'; @@ -10,9 +11,14 @@ import { join } from 'path'; */ const isolatedRoot = mkdtempSync(join(tmpdir(), 'episodic-memory-vitest-')); +afterAll(() => { + rmSync(isolatedRoot, { recursive: true, force: true }); +}); + process.env.EPISODIC_MEMORY_CONFIG_DIR = join(isolatedRoot, 'superpowers'); process.env.CLAUDE_CONFIG_DIR = join(isolatedRoot, 'claude'); process.env.CODEX_HOME = join(isolatedRoot, 'codex'); +process.env.OMP_HOME = join(isolatedRoot, 'omp'); // Clear any ambient summarizer-billing signals so tests get deterministic // cost-guard (#104) and timeout (#160) behavior regardless of the developer's diff --git a/test/summarizer-auth-isolation.test.ts b/test/summarizer-auth-isolation.test.ts new file mode 100644 index 00000000..b28a4d00 --- /dev/null +++ b/test/summarizer-auth-isolation.test.ts @@ -0,0 +1,70 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { buildSummarizerQueryOptions } from '../src/summarizer.js'; + +describe('isolated Claude summarizer authentication', () => { + let root: string; + + beforeEach(() => { + root = mkdtempSync(join(tmpdir(), 'episodic-auth-settings-')); + vi.stubEnv('CLAUDE_CONFIG_DIR', root); + vi.stubEnv('ANTHROPIC_BASE_URL', undefined); + vi.stubEnv('EPISODIC_MEMORY_API_BASE_URL', undefined); + }); + + afterEach(() => { + vi.unstubAllEnvs(); + rmSync(root, { recursive: true, force: true }); + }); + + it('keeps the user credential helper and endpoint without loading behavior settings', () => { + writeFileSync(join(root, 'settings.json'), JSON.stringify({ + apiKeyHelper: 'existing-credential-helper', + env: { ANTHROPIC_BASE_URL: 'https://fixture.invalid/api', UNRELATED: 'excluded' }, + hooks: { Stop: [{ hooks: [{ type: 'command', command: 'must-not-run' }] }] }, + enabledPlugins: { 'fixture@marketplace': true }, + permissions: { allow: ['Bash(*)'] }, + model: 'must-not-select', + })); + + const options = buildSummarizerQueryOptions({ model: 'haiku' }); + expect(options.settings).toEqual({ apiKeyHelper: 'existing-credential-helper' }); + expect(options.env).toMatchObject({ ANTHROPIC_BASE_URL: 'https://fixture.invalid/api' }); + expect((options.env as Record).UNRELATED).toBeUndefined(); + expect(options.settingSources).toEqual([]); + expect(options.tools).toEqual([]); + expect(options.strictMcpConfig).toBe(true); + expect(options.persistSession).toBe(false); + expect(options.model).toBe('haiku'); + }); + + it('preserves explicit inherited and episodic endpoint overrides', () => { + writeFileSync(join(root, 'settings.json'), JSON.stringify({ + env: { ANTHROPIC_BASE_URL: 'https://settings.invalid' }, + })); + vi.stubEnv('ANTHROPIC_BASE_URL', 'https://inherited.invalid'); + expect(buildSummarizerQueryOptions({ model: 'haiku' }).env).toMatchObject({ + ANTHROPIC_BASE_URL: 'https://inherited.invalid', + }); + vi.stubEnv('EPISODIC_MEMORY_API_BASE_URL', 'https://explicit.invalid'); + expect(buildSummarizerQueryOptions({ model: 'haiku' }).env).toMatchObject({ + ANTHROPIC_BASE_URL: 'https://explicit.invalid', + }); + }); + + it('leaves native environment and OAuth authentication available when settings are absent', () => { + vi.stubEnv('ANTHROPIC_API_KEY', 'synthetic-fixture-value'); + const options = buildSummarizerQueryOptions({ model: 'haiku' }); + expect(options.settings).toBeUndefined(); + expect(options.env).toMatchObject({ ANTHROPIC_API_KEY: 'synthetic-fixture-value' }); + expect(options.settingSources).toEqual([]); + }); + + it('refuses malformed authentication settings without exposing their contents', () => { + writeFileSync(join(root, 'settings.json'), '{"apiKeyHelper":"PRIVATE_FIXTURE_VALUE",'); + expect(() => buildSummarizerQueryOptions({ model: 'haiku' })) + .toThrow(/^Cannot read Claude summarizer authentication settings$/); + }); +}); diff --git a/test/summarizer-options.test.ts b/test/summarizer-options.test.ts index 1267b880..b7260a0e 100644 --- a/test/summarizer-options.test.ts +++ b/test/summarizer-options.test.ts @@ -110,6 +110,22 @@ describe('isResumeFailure', () => { }); describe('buildCodexSummarizerCommand', () => { + it('disables host capabilities before an isolated summary server can load them', () => { + const command = buildCodexSummarizerCommand({ prompt: 'Admitted text.', isolated: true }); + expect(command.isolated).toBe(true); + for (const option of ['features.shell_tool=false', 'features.plugins=false', 'features.hooks=false', + 'features.apps=false', 'features.code_mode_host=false', 'features.skip_host_skill_discovery=true', + 'skills.bundled.enabled=false', 'skills.include_instructions=false', 'project_doc_max_bytes=0', + 'tools.experimental_request_user_input.enabled=false', 'web_search="disabled"']) { + expect(command.args).toContain(option); + } + }); + + it('refuses to combine isolated admitted text with an original session identity', () => { + expect(() => buildCodexSummarizerCommand({ prompt: 'Admitted text.', isolated: true, sessionId: 'original' })) + .toThrow(/isolated.*session/i); + }); + it('starts the Codex app-server so the summarizer can fork ephemerally', () => { const command = buildCodexSummarizerCommand({ sessionId: '019e4c75-d5bf-7c71-9df7-77f5fb86b711', @@ -129,6 +145,55 @@ describe('buildCodexSummarizerCommand', () => { }); describe('runCodexCommand', () => { + it('disables inherited MCP and skill entries per subject and removes its temporary cwd', async () => { + const fake = ` + const fs = require('fs'); + const readline = require('readline'); + const emit = value => console.log(JSON.stringify(value)); + readline.createInterface({ input: process.stdin }).on('line', line => { + const m = JSON.parse(line); + if (m.method === 'initialize') emit({ id: m.id, result: {} }); + if (m.method === 'config/read') emit({ id: m.id, result: { config: { mcp_servers: { inherited: { command: 'false', enabled: true } } } } }); + if (m.method === 'skills/list') emit({ id: m.id, result: { data: [{ skills: [{ path: '/public-fixture/SKILL.md', enabled: true }], errors: [] }] } }); + if (m.method === 'thread/fork') process.exit(8); + if (m.method === 'thread/start') { + const p = m.params; + if (p.config?.mcp_servers?.inherited?.enabled !== false) process.exit(9); + if (p.config?.['skills.config']?.[0]?.enabled !== false) process.exit(10); + if ([p.dynamicTools, p.environments, p.selectedCapabilityRoots].some(x => !Array.isArray(x) || x.length)) process.exit(11); + if (p.cwd !== process.cwd() || !fs.existsSync(p.cwd)) process.exit(12); + if (!p.ephemeral || p.allowProviderModelFallback !== false) process.exit(13); + emit({ id: m.id, result: { thread: { id: 'isolated' } } }); + } + if (m.method === 'turn/start') { + emit({ id: m.id, result: { turn: { id: 'turn' } } }); + emit({ method: 'item/completed', params: { item: { type: 'agentMessage', text: process.cwd() } } }); + emit({ method: 'turn/completed', params: { turn: { id: 'turn', status: 'completed' } } }); + } + }); + `; + const result = await runCodexCommand({ command: process.execPath, args: ['-e', fake], + prompt: 'Admitted public fixture.', isolated: true, skipVersionCheck: true }); + expect(result).toMatch(/episodic-memory-summary-/); + const { existsSync } = await import('fs'); + expect(existsSync(result)).toBe(false); + }); + + it('does not echo a provider error payload from an isolated summary', async () => { + const fake = ` + const readline = require('readline'); + readline.createInterface({ input: process.stdin }).on('line', line => { + const m = JSON.parse(line); + console.log(JSON.stringify(m.method === 'initialize' + ? { id: m.id, result: {} } + : { id: m.id, error: { message: 'EXCLUDED_PUBLIC_FIXTURE' } })); + }); + `; + await expect(runCodexCommand({ command: process.execPath, args: ['-e', fake], + prompt: 'Admitted text.', isolated: true, skipVersionCheck: true })) + .rejects.toThrow(/^Isolated Codex summary failed; provider details suppressed$/); + }); + it('forks the session ephemerally and returns the completed agent message', async () => { const fakeAppServer = ` const readline = require('readline'); diff --git a/test/sync-cli-single-instance.test.ts b/test/sync-cli-single-instance.test.ts index b1505559..c43e8421 100644 --- a/test/sync-cli-single-instance.test.ts +++ b/test/sync-cli-single-instance.test.ts @@ -103,6 +103,24 @@ describe('sync-cli single-instance lock (#97)', () => { expect(result.stdout).toMatch(/Sync complete/); }); + it('exits nonzero without a success claim when record admission rejects a source', () => { + const session = '00000000-0000-0000-0000-000000000002'; + const transcript = `rollout-${session}.jsonl`; + writeFileSync(join(testDir, 'projects', 'project-a', transcript), [ + { type: 'session_meta', payload: { id: '00000000-aaaa-4bbb-8ccc-dddddddddddd' } }, + { type: 'response_item', payload: { type: 'message', role: 'user', content: [{ type: 'text', text: 'public fixture' }] } }, + ].map(row => JSON.stringify(row)).join('\n') + '\n'); + writeFileSync(join(testDir, 'config', 'record-exclusions.json'), JSON.stringify({ + version: 1, exchanges: [{ session_id: session, transcript, line_start: 2, line_end: 2 }], tool_calls: [], + })); + const result = runWith({ ...envOverrides, EPISODIC_MEMORY_SKIP_SUMMARIES: '1' }); + expect(result.stdout + result.stderr).toMatch(/session identity mismatch/); + expect(result.status).toBe(1); + expect(result.stdout).not.toMatch(/Sync complete/); + expect(existsSync(join(testDir, 'archive', 'project-a', transcript))).toBe(false); + expect(existsSync(join(testDir, 'config', 'logs', 'episodic-memory-sync.lock'))).toBe(false); + }); + it('releases the lock on normal exit — a subsequent run is not skipped', () => { const first = runWith(envOverrides); expect(first.status).toBe(0); diff --git a/test/sync-incremental-and-marker.test.ts b/test/sync-incremental-and-marker.test.ts index ecaa9aeb..d9b020a2 100644 --- a/test/sync-incremental-and-marker.test.ts +++ b/test/sync-incremental-and-marker.test.ts @@ -123,7 +123,7 @@ describe('sync: high-water mark (#152 Part A)', () => { const callsAfterFirstSync = embedSpy.mock.calls.length; - // Append a 3rd exchange and bump mtime forward so copyIfNewer copies it. + // Append a 3rd exchange and bump mtime forward so the archive writer copies it. appendFileSync(srcFile, exchangeLines(3, sessionId), 'utf-8'); const future = new Date(Date.now() + 5000); utimesSync(srcFile, future, future); diff --git a/test/sync.test.ts b/test/sync.test.ts index f35f9a00..551fd7b9 100644 --- a/test/sync.test.ts +++ b/test/sync.test.ts @@ -240,10 +240,12 @@ describe('sync command', () => { // Sync with indexing enabled const result = await syncConversations(sourceDir, destDir); - // Both files should be copied - expect(result.copied).toBe(2); + // Whole-conversation opt-out now rejects copying as well as indexing. + expect(result.copied).toBe(1); + expect(existsSync(join(destDir, 'project-a', 'marked.jsonl'))).toBe(false); + expect(existsSync(join(destDir, 'project-a', 'normal.jsonl'))).toBe(true); - // But only normal conversation should be indexed + // Only the normal conversation should be indexed expect(result.indexed).toBe(1); // Verify in database diff --git a/test/verify.test.ts b/test/verify.test.ts index ae70da81..5f2acfef 100644 --- a/test/verify.test.ts +++ b/test/verify.test.ts @@ -6,6 +6,7 @@ import path from 'path'; import os from 'os'; import { initDatabase, insertExchange } from '../src/db.js'; import { ConversationExchange } from '../src/types.js'; +import { summarizeConversation } from '../src/summarizer.js'; vi.mock('../src/embeddings.js', () => ({ initEmbeddings: vi.fn(async () => {}), @@ -45,6 +46,21 @@ describe('verifyIndex', () => { delete process.env.TEST_DB_PATH; }); + it('does not create an index database merely by verifying an archive', async () => { + const projectArchive = path.join(archiveDir, 'read-only-project'); + fs.mkdirSync(projectArchive, { recursive: true }); + const conversationPath = path.join(projectArchive, 'conversation.jsonl'); + fs.writeFileSync(conversationPath, [ + JSON.stringify({ type: 'user', message: { role: 'user', content: 'Hello' }, timestamp: '2024-01-01T00:00:00Z' }), + JSON.stringify({ type: 'assistant', message: { role: 'assistant', content: 'Hi' }, timestamp: '2024-01-01T00:00:01Z' }), + ].join('\n')); + + expect(fs.existsSync(dbPath)).toBe(false); + const result = await verifyIndex(); + expect(result.unindexed.map(item => item.path)).toContain(conversationPath); + expect(fs.existsSync(dbPath)).toBe(false); + }); + it('should skip excluded projects', async () => { // Create two projects in archive const projectA = path.join(archiveDir, 'project-a'); @@ -156,7 +172,7 @@ describe('verifyIndex', () => { expect(result.orphaned[0].path).toBe(exchange.archivePath); }); - it('detects outdated files (file modified after last_indexed)', async () => { + it('detects appended exchanges beyond the indexed high-water mark', async () => { // Create conversation file with summary const projectArchive = path.join(archiveDir, 'test-project'); fs.mkdirSync(projectArchive, { recursive: true }); @@ -193,15 +209,15 @@ describe('verifyIndex', () => { const lastIndexed = row.last_indexed; db.close(); - // Wait a bit, then modify the file - await new Promise(resolve => setTimeout(resolve, 10)); - - // Update the conversation file + // Append a complete exchange and make the timestamp comparison deterministic. const updatedMessages = [ ...messages, - JSON.stringify({ type: 'user', message: { role: 'user', content: 'New message' }, timestamp: '2024-01-01T00:00:02Z' }) + JSON.stringify({ type: 'user', message: { role: 'user', content: 'New message' }, timestamp: '2024-01-01T00:00:02Z' }), + JSON.stringify({ type: 'assistant', message: { role: 'assistant', content: 'New response' }, timestamp: '2024-01-01T00:00:03Z' }) ]; fs.writeFileSync(conversationPath, updatedMessages.join('\n')); + const future = new Date(lastIndexed + 10_000); + fs.utimesSync(conversationPath, future, future); // Verify detects outdated file const result = await verifyIndex(); @@ -212,6 +228,75 @@ describe('verifyIndex', () => { expect(result.outdated[0].fileTime).toBeGreaterThan(lastIndexed); }); + it('does not call an mtime-only archive refresh an index delta', async () => { + const projectArchive = path.join(archiveDir, 'test-project'); + fs.mkdirSync(projectArchive, { recursive: true }); + const conversationPath = path.join(projectArchive, 'refreshed.jsonl'); + fs.writeFileSync(conversationPath, [ + JSON.stringify({ type: 'user', message: { role: 'user', content: 'Hello' }, timestamp: '2024-01-01T00:00:00Z' }), + JSON.stringify({ type: 'assistant', message: { role: 'assistant', content: 'Hi' }, timestamp: '2024-01-01T00:00:01Z' }) + ].join('\n')); + fs.writeFileSync(conversationPath.replace('.jsonl', '-summary.txt'), 'Existing summary'); + const db = initDatabase(); + insertExchange(db, { id: 'refreshed', project: 'test-project', timestamp: '2024-01-01T00:00:00Z', + userMessage: 'Hello', assistantMessage: 'Hi', archivePath: conversationPath, lineStart: 1, lineEnd: 2 }, new Array(384).fill(0.1)); + db.close(); + const future = new Date(Date.now() + 10_000); + fs.utimesSync(conversationPath, future, future); + + const result = await verifyIndex(); + expect(result.outdated).toEqual([]); + expect(result.archiveRefreshes.map(item => item.path)).toEqual([conversationPath]); + }); + + it('reports a missing summary and an appended exchange independently', async () => { + const projectArchive = path.join(archiveDir, 'test-project'); + fs.mkdirSync(projectArchive, { recursive: true }); + const conversationPath = path.join(projectArchive, 'appended.jsonl'); + const messages = [ + JSON.stringify({ type: 'user', message: { role: 'user', content: 'Hello' }, timestamp: '2024-01-01T00:00:00Z' }), + JSON.stringify({ type: 'assistant', message: { role: 'assistant', content: 'Hi' }, timestamp: '2024-01-01T00:00:01Z' }) + ]; + fs.writeFileSync(conversationPath, messages.join('\n')); + const db = initDatabase(); + insertExchange(db, { id: 'original', project: 'test-project', timestamp: '2024-01-01T00:00:00Z', + userMessage: 'Hello', assistantMessage: 'Hi', archivePath: conversationPath, lineStart: 1, lineEnd: 2 }, new Array(384).fill(0.1)); + const lastIndexed = (db.prepare('SELECT last_indexed FROM exchanges WHERE id = ?').get('original') as { last_indexed: number }).last_indexed; + db.close(); + fs.appendFileSync(conversationPath, '\n' + [ + JSON.stringify({ type: 'user', message: { role: 'user', content: 'Later' }, timestamp: '2024-01-01T00:00:02Z' }), + JSON.stringify({ type: 'assistant', message: { role: 'assistant', content: 'Done' }, timestamp: '2024-01-01T00:00:03Z' }) + ].join('\n')); + const future = new Date(lastIndexed + 10_000); + fs.utimesSync(conversationPath, future, future); + + const result = await verifyIndex(); + expect(result.missing.map(item => item.path)).toEqual([conversationPath]); + expect(result.outdated.map(item => item.path)).toEqual([conversationPath]); + }); + + it('distinguishes an unindexed archive from an indexed external source', async () => { + const projectArchive = path.join(archiveDir, 'test-project'); + fs.mkdirSync(projectArchive, { recursive: true }); + const archivePath = path.join(projectArchive, 'unindexed.jsonl'); + const sourcePath = path.join(testDir, 'active.jsonl'); + const lines = [ + JSON.stringify({ type: 'user', message: { role: 'user', content: 'Hello' }, timestamp: '2024-01-01T00:00:00Z' }), + JSON.stringify({ type: 'assistant', message: { role: 'assistant', content: 'Hi' }, timestamp: '2024-01-01T00:00:01Z' }) + ].join('\n'); + fs.writeFileSync(archivePath, lines); + fs.writeFileSync(archivePath.replace('.jsonl', '-summary.txt'), 'Existing summary'); + fs.writeFileSync(sourcePath, lines); + const db = initDatabase(); + insertExchange(db, { id: 'external', project: 'test-project', timestamp: '2024-01-01T00:00:00Z', + userMessage: 'Hello', assistantMessage: 'Hi', archivePath: sourcePath, lineStart: 1, lineEnd: 2 }, new Array(384).fill(0.1)); + db.close(); + + const result = await verifyIndex(); + expect(result.unindexed.map(item => item.path)).toEqual([archivePath]); + expect(result.orphaned).toEqual([]); + }); + // Note: Parser is resilient to malformed JSON - it skips bad lines // Corruption detection would require file system errors or permission issues // which are harder to test. Skipping for now as missing summaries is the @@ -244,6 +329,26 @@ describe('repairIndex', () => { delete process.env.TEST_DB_PATH; }); + it('repairs a path reported as both missing and outdated only once', async () => { + const projectArchive = path.join(archiveDir, 'test-project'); + fs.mkdirSync(projectArchive, { recursive: true }); + const conversationPath = path.join(projectArchive, 'duplicate.jsonl'); + fs.writeFileSync(conversationPath, [ + JSON.stringify({ type: 'user', message: { role: 'user', content: 'Hello' }, timestamp: '2024-01-01T00:00:00Z' }), + JSON.stringify({ type: 'assistant', message: { role: 'assistant', content: 'Hi' }, timestamp: '2024-01-01T00:00:01Z' }) + ].join('\n')); + vi.mocked(summarizeConversation).mockClear(); + + await repairIndex({ + ...await verifyIndex(), + missing: [{ path: conversationPath, reason: 'No summary file' }], + outdated: [{ path: conversationPath, fileTime: 2, dbTime: 1 }], + orphaned: [], corrupted: [] + }); + + expect(summarizeConversation).toHaveBeenCalledTimes(1); + }); + it('deletes orphaned database entries during repair', async () => { // Initialize database with orphaned entry const db = initDatabase(); @@ -281,6 +386,44 @@ describe('repairIndex', () => { dbAfter.close(); }); + it('rejects a failed repair instead of reporting completion', async () => { + const input = path.join(archiveDir, 'not-a-transcript'); + fs.mkdirSync(input, { recursive: true }); + await expect(repairIndex({ missing: [{ path: input, reason: 'synthetic invalid source' }], + outdated: [], orphaned: [], screeningRejected: [], screeningUnavailable: [], corrupted: [] })).rejects.toThrow(/regular non-symlink file/); + }); + + it('fails closed on a corrupt rules file in strict mode, before changing the index', async () => { + const db = initDatabase(); + insertExchange(db, { + id: 'orphan-strict-1', + project: 'deleted-project', + timestamp: '2024-01-01T00:00:00Z', + userMessage: 'Deleted', + assistantMessage: 'Still indexed', + archivePath: path.join(archiveDir, 'deleted-project', 'deleted.jsonl'), + lineStart: 1, + lineEnd: 2 + }, new Array(384).fill(0.1)); + db.close(); + + const rulesPath = path.join(testDir, 'redaction-rules.json'); + fs.writeFileSync(rulesPath, '{ not json'); + process.env.EPISODIC_MEMORY_REDACTION_RULES = rulesPath; + try { + const issues = await verifyIndex(); + expect(issues.orphaned.length).toBe(1); + await expect(repairIndex(issues)).rejects.toThrow(/redaction/i); + } finally { + delete process.env.EPISODIC_MEMORY_REDACTION_RULES; + } + + const dbAfter = initDatabase(); + const row = dbAfter.prepare(`SELECT COUNT(*) as count FROM exchanges WHERE id = ?`).get('orphan-strict-1') as { count: number }; + expect(row.count).toBe(1); + dbAfter.close(); + }); + it('re-indexes outdated files during repair', { timeout: 30000 }, async () => { // Create conversation file with summary const projectArchive = path.join(archiveDir, 'test-project');