Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -372,9 +372,14 @@ This test uses your normal Claude Code auth and writes small test transcripts to
Manual indexing tools for bulk operations and maintenance. See `episodic-memory index --help` for full options.

Common operations:
- `--cleanup` - Index all unprocessed conversations
- `--verify` - Check index health
- `--repair` - Fix detected issues
- `--cleanup` - Index unprocessed exchanges; summaries may call an AI provider unless disabled
- `--verify` - Distinguish missing summaries, unindexed archives, appended exchanges, and archive mtime refreshes
- `--repair` - Re-embed and regenerate summaries for repair candidates; review the scope and provider cost first

An archive mtime refresh without new exchanges is reported separately and is
not an outdated-index repair target. An unindexed archive is reported, but
that status alone does not select it for `--repair`; a separate overlapping
repair category may still cause it to be processed.

### `episodic-memory search`

Expand Down
14 changes: 7 additions & 7 deletions cli/index-conversations.js
Original file line number Diff line number Diff line change
Expand Up @@ -50,16 +50,16 @@ USAGE:

COMMANDS:
(default) Index all conversations
--cleanup Process only unindexed conversations (fast, cheap)
--cleanup Process unindexed exchanges (may generate AI summaries)
--session ID Index specific session (used by hook)
--verify Check index health
--repair Fix detected issues
--verify Classify index and summary state
--repair Re-embed and summarize repair candidates (review first)
--rebuild Delete DB and re-index everything (requires confirmation)

OPTIONS:
--concurrency N Parallel summarization (1-16, default: 1)
-c N Short form of --concurrency
--no-summaries Skip AI summary generation (free, but no summaries in results)
--no-summaries Skip AI summary provider calls
--help, -h Show this help

EXAMPLES:
Expand All @@ -69,13 +69,13 @@ EXAMPLES:
# Index with 8 parallel summarizations (8x faster)
index-conversations --cleanup --concurrency 8

# Index without AI summaries (free, fast)
# Index without AI summary provider calls
index-conversations --cleanup --no-summaries

# Check index health
index-conversations --verify

# Fix any issues found
# Review candidates and provider cost before broad repair
index-conversations --repair

# Nuclear option (deletes everything, re-indexes)
Expand All @@ -85,7 +85,7 @@ WORKFLOW:
1. Initial setup: index-conversations --cleanup
2. Ongoing: Auto-indexed by sessionEnd hook
3. Health check: index-conversations --verify (weekly)
4. Recovery: index-conversations --repair (if issues found)
4. Recovery: review exact verifier categories before repair

SEE ALSO:
INDEXING.md - Setup and maintenance guide
Expand Down
28 changes: 21 additions & 7 deletions dist/index-cli.js
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,16 @@ import path from 'path';
import { getSyncLockPath } from './logging.js';
import { acquireFileLock, readLockHolder, releaseFileLock } from './file-lock.js';
const command = process.argv[2];
// Serialize the whole CLI with `sync-cli`: every command can initialize or
// migrate the database, and rebuild also deletes the database and summaries.
// Serialize the whole CLI with `sync-cli`: writers mutate the index/archive,
// and verification must not claim a consistent result while one is active.
const syncLockPath = getSyncLockPath();
const syncLock = acquireFileLock(syncLockPath);
if (!syncLock) {
const holder = readLockHolder(syncLockPath);
const holderLabel = holder !== null ? `pid ${holder}` : 'another process';
// stderr keeps this out of stdout consumers; status 0 so hooks don't fail.
// A skipped write may be retried by a hook; a skipped verifier is not a result.
console.error(`episodic-memory: sync already running (${holderLabel}); skipping`);
process.exit(0);
process.exit(command === 'verify' ? 2 : 0);
}
const releaseSyncLockOnce = () => {
if (releaseSyncLockOnce.done)
Expand Down Expand Up @@ -63,15 +63,25 @@ async function main() {
const issues = await verifyIndex();
console.log('\n=== Verification Results ===');
console.log(`Missing summaries: ${issues.missing.length}`);
console.log(`Unindexed archives: ${issues.unindexed.length}`);
console.log(`Orphaned entries: ${issues.orphaned.length}`);
console.log(`Outdated files: ${issues.outdated.length}`);
console.log(`Archive refreshes (no new exchanges): ${issues.archiveRefreshes.length}`);
console.log(`Corrupted files: ${issues.corrupted.length}`);
if (issues.missing.length > 0) {
console.log('\nMissing summaries:');
issues.missing.forEach(m => console.log(` ${m.path}`));
}
if (issues.missing.length + issues.orphaned.length + issues.outdated.length + issues.corrupted.length > 0) {
console.log('\nRun with --repair to fix these issues.');
if (issues.unindexed.length > 0) {
console.log('\nUnindexed archives:');
issues.unindexed.forEach(item => console.log(` ${item.path}`));
}
if (issues.outdated.length > 0) {
console.log('\nArchives with appended exchanges:');
issues.outdated.forEach(item => console.log(` ${item.path}`));
}
if (issues.missing.length + issues.unindexed.length + issues.orphaned.length + issues.outdated.length + issues.corrupted.length > 0) {
console.log('\nReview exact candidates before repair; --repair may regenerate summaries and embeddings.');
process.exit(1);
}
else {
Expand All @@ -81,10 +91,14 @@ async function main() {
case 'repair':
console.log('Verifying conversation index...');
const repairIssues = await verifyIndex();
if (repairIssues.unindexed.length > 0) {
console.log(`${repairIssues.unindexed.length} unindexed archive(s) require separate review; --repair does not select them by this status alone.`);
process.exitCode = 1;
}
if (repairIssues.missing.length + repairIssues.orphaned.length + repairIssues.outdated.length > 0) {
await repairIndex(repairIssues);
}
else {
else if (repairIssues.unindexed.length === 0) {
console.log('✅ No issues to repair!');
}
break;
Expand Down
10 changes: 9 additions & 1 deletion dist/verify.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@ export interface VerificationResult {
path: string;
reason: string;
}>;
unindexed: Array<{
path: string;
}>;
orphaned: Array<{
uuid: string;
path: string;
Expand All @@ -12,10 +15,15 @@ export interface VerificationResult {
fileTime: number;
dbTime: number;
}>;
archiveRefreshes: Array<{
path: string;
fileTime: number;
dbTime: number;
}>;
corrupted: Array<{
path: string;
error: string;
}>;
}
export declare function verifyIndex(): Promise<VerificationResult>;
export declare function repairIndex(issues: VerificationResult): Promise<void>;
export declare function repairIndex(issues: Pick<VerificationResult, 'missing' | 'orphaned' | 'outdated' | 'corrupted'>): Promise<void>;
65 changes: 37 additions & 28 deletions dist/verify.js
Original file line number Diff line number Diff line change
@@ -1,14 +1,17 @@
import fs from 'fs';
import path from 'path';
import Database from 'better-sqlite3';
import { parseConversation } from './parser.js';
import { initDatabase, getAllExchanges, getFileLastIndexed } from './db.js';
import { getArchiveDir, getExcludedProjects, findJsonlFiles, statIfExists } from './paths.js';
import { getAllExchanges } from './db.js';
import { getArchiveDir, getDbPath, getExcludedProjects, findJsonlFiles, statIfExists } from './paths.js';
import { isErroredSentinel } from './summary-sentinel.js';
export async function verifyIndex() {
const result = {
missing: [],
unindexed: [],
orphaned: [],
outdated: [],
archiveRefreshes: [],
corrupted: []
};
const archiveDir = getArchiveDir();
Expand All @@ -18,8 +21,13 @@ export async function verifyIndex() {
if (!fs.existsSync(archiveDir)) {
return result;
}
// Initialize database once for all checks
const db = initDatabase();
// Verification must not create or migrate the index it is measuring.
const dbPath = getDbPath();
const db = fs.existsSync(dbPath) ? new Database(dbPath, { readonly: true, fileMustExist: true }) : null;
const indexState = db?.prepare(`
SELECT MAX(line_end) AS maxLineEnd, MAX(last_indexed) AS lastIndexed
FROM exchanges WHERE archive_path = ?
`);
const projects = fs.readdirSync(archiveDir);
const excludedProjects = getExcludedProjects();
const excludedDirSet = new Set(excludedProjects);
Expand Down Expand Up @@ -47,27 +55,28 @@ export async function verifyIndex() {
// re-attempts it rather than reporting the conversation as healthy.
if (!fs.existsSync(summaryPath)) {
result.missing.push({ path: conversationPath, reason: 'No summary file' });
continue;
}
if (isErroredSentinel(fs.readFileSync(summaryPath, 'utf-8'))) {
else if (isErroredSentinel(fs.readFileSync(summaryPath, 'utf-8'))) {
result.missing.push({ path: conversationPath, reason: 'Previous summarization failed (error sentinel)' });
continue;
}
// Check if file is outdated (modified after last_indexed)
const lastIndexed = getFileLastIndexed(db, conversationPath);
if (lastIndexed !== null) {
const fileStat = fs.statSync(conversationPath);
if (fileStat.mtimeMs > lastIndexed) {
result.outdated.push({
path: conversationPath,
fileTime: fileStat.mtimeMs,
dbTime: lastIndexed
});
}
}
// Try parsing to detect corruption
const state = indexState?.get(conversationPath);
const maxLineEnd = state?.maxLineEnd ?? null;
if (maxLineEnd === null)
result.unindexed.push({ path: conversationPath });
// Parse independently of summary state and mtime. Only exchanges past
// the indexed high-water mark are a real index delta.
try {
await parseConversation(conversationPath, project, conversationPath);
const exchanges = await parseConversation(conversationPath, project, conversationPath);
if (maxLineEnd !== null) {
const fileTime = fs.statSync(conversationPath).mtimeMs;
const dbTime = state?.lastIndexed ?? 0;
if (exchanges.some(exchange => exchange.lineEnd > maxLineEnd)) {
result.outdated.push({ path: conversationPath, fileTime, dbTime });
}
else if (state?.lastIndexed != null && fileTime > state.lastIndexed) {
result.archiveRefreshes.push({ path: conversationPath, fileTime, dbTime });
}
}
}
catch (error) {
result.corrupted.push({
Expand All @@ -79,10 +88,10 @@ export async function verifyIndex() {
}
console.log(`Verified ${totalChecked} conversations.`);
// Check for orphaned database entries
const dbExchanges = getAllExchanges(db);
db.close();
const dbExchanges = db ? getAllExchanges(db) : [];
db?.close();
for (const exchange of dbExchanges) {
if (!foundFiles.has(exchange.archivePath)) {
if (!foundFiles.has(exchange.archivePath) && !statIfExists(exchange.archivePath)?.isFile()) {
result.orphaned.push({
uuid: exchange.id,
path: exchange.archivePath
Expand All @@ -106,10 +115,10 @@ export async function repairIndex(issues) {
deleteExchange(db, orphan.uuid);
}
// Re-index missing and outdated conversations
const toReindex = [
...issues.missing.map(m => m.path),
...issues.outdated.map(o => o.path)
];
const toReindex = [...new Set([
...issues.missing.map(m => m.path),
...issues.outdated.map(o => o.path)
])];
for (const conversationPath of toReindex) {
console.log(`Re-indexing: ${conversationPath}`);
try {
Expand Down
28 changes: 21 additions & 7 deletions src/index-cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,16 @@ import { acquireFileLock, readLockHolder, releaseFileLock } from './file-lock.js

const command = process.argv[2];

// Serialize the whole CLI with `sync-cli`: every command can initialize or
// migrate the database, and rebuild also deletes the database and summaries.
// Serialize the whole CLI with `sync-cli`: writers mutate the index/archive,
// and verification must not claim a consistent result while one is active.
const syncLockPath = getSyncLockPath();
const syncLock = acquireFileLock(syncLockPath);
if (!syncLock) {
const holder = readLockHolder(syncLockPath);
const holderLabel = holder !== null ? `pid ${holder}` : 'another process';
// stderr keeps this out of stdout consumers; status 0 so hooks don't fail.
// A skipped write may be retried by a hook; a skipped verifier is not a result.
console.error(`episodic-memory: sync already running (${holderLabel}); skipping`);
process.exit(0);
process.exit(command === 'verify' ? 2 : 0);
}
const releaseSyncLockOnce = () => {
if ((releaseSyncLockOnce as any).done) return;
Expand Down Expand Up @@ -71,17 +71,27 @@ async function main() {

console.log('\n=== Verification Results ===');
console.log(`Missing summaries: ${issues.missing.length}`);
console.log(`Unindexed archives: ${issues.unindexed.length}`);
console.log(`Orphaned entries: ${issues.orphaned.length}`);
console.log(`Outdated files: ${issues.outdated.length}`);
console.log(`Archive refreshes (no new exchanges): ${issues.archiveRefreshes.length}`);
console.log(`Corrupted files: ${issues.corrupted.length}`);

if (issues.missing.length > 0) {
console.log('\nMissing summaries:');
issues.missing.forEach(m => console.log(` ${m.path}`));
}
if (issues.unindexed.length > 0) {
console.log('\nUnindexed archives:');
issues.unindexed.forEach(item => console.log(` ${item.path}`));
}
if (issues.outdated.length > 0) {
console.log('\nArchives with appended exchanges:');
issues.outdated.forEach(item => console.log(` ${item.path}`));
}

if (issues.missing.length + issues.orphaned.length + issues.outdated.length + issues.corrupted.length > 0) {
console.log('\nRun with --repair to fix these issues.');
if (issues.missing.length + issues.unindexed.length + issues.orphaned.length + issues.outdated.length + issues.corrupted.length > 0) {
console.log('\nReview exact candidates before repair; --repair may regenerate summaries and embeddings.');
process.exit(1);
} else {
console.log('\n✅ Index is healthy!');
Expand All @@ -92,9 +102,13 @@ async function main() {
console.log('Verifying conversation index...');
const repairIssues = await verifyIndex();

if (repairIssues.unindexed.length > 0) {
console.log(`${repairIssues.unindexed.length} unindexed archive(s) require separate review; --repair does not select them by this status alone.`);
process.exitCode = 1;
}
if (repairIssues.missing.length + repairIssues.orphaned.length + repairIssues.outdated.length > 0) {
await repairIndex(repairIssues);
} else {
} else if (repairIssues.unindexed.length === 0) {
console.log('✅ No issues to repair!');
}
break;
Expand Down
Loading