Skip to content

feat(notes): local-first voice memo transcription into notes (ODIAA-2331) - #27

Merged
amirhmoradi merged 3 commits into
mainfrom
odiaa-2331-notes-transcribe
Sep 28, 2026
Merged

amirhmoradi merged 3 commits into
mainfrom
odiaa-2331-notes-transcribe

Conversation

@amirhmoradi

Copy link
Copy Markdown
Contributor

Summary

  • Adds tb notes transcribe <audioFile> and MCP tool note_transcribe: audio → text → saved note, with source (voice-memo:<filename>) and engine recorded in the note's front matter.
  • Local-first by default: at runtime, auto-detects an installed whisper.cpp (whisper-cli/whisper-cpp) or faster-whisper (whisper-ctranslate2) binary on PATH, or via TB_WHISPER_CPP_BIN/TB_FASTER_WHISPER_BIN/TB_WHISPER_CPP_MODEL env overrides. If neither is found, raises a clean NO_ENGINE error naming both engines and how to install/configure them.
  • No cloud STT path is implemented and no API keys are embedded — audio never leaves the machine. See "Cloud STT trade-off" below for the required escalation write-up.
  • Surface parity: cli/src/stt.js (+ self-contained mirror mcp/src/stt.js) → tb notes transcribe in cli/src/cli.js → note_transcribe in mcp/src/tools.js. No new extension route was needed (transcription runs entirely against a local file and a local subprocess, unlike note_to_draft which reuses the already-classified /compose extension route), so extension/src/access-control.js is unchanged.
  • Docs updated: SPEC.md (tool catalog + counts: CLI 43→44, MCP 21→22), docs/COMMANDS.md, CHANGELOG.md, skills/thunderbird-cli/SKILL.md (tool table + CLI-mapping table).
  • Tests: new cli/src/stt.js/mcp/src/stt.js coverage using stub engine binaries (test/fixtures/stub-whisper-cpp.mjs, test/fixtures/stub-faster-whisper.mjs) for the no-engine, engine-selection, model-required, unknown-engine, missing-audio, engine-failure, and end-to-end transcript→note round-trip paths, plus the new note_transcribe MCP tool tests. Also fixed a latent test/docs.test.mjs fragility where Commander's help-text line-wrapping (triggered by the new long transcribe command signature) was mis-parsed as bogus subcommands.

Cloud STT trade-off (for CTO → CEO routing, not shipped here)

Per ODIAA-2331/ODIAA-2306, any cloud speech-to-text provider is a separate vendor/stack decision requiring approval before it ships. This PR ships local-only engines; the options below are informational for that future decision:

  • Option A — Stay local-only (current, this PR). Uses user-installed whisper.cpp or faster-whisper. Pros: zero marginal cost, no data leaves the user's machine (strong privacy story, no BAA/DPA needed), works offline, no vendor lock-in or API key management. Cons: requires the user to install a third-party engine and (for whisper.cpp) a model file; transcription speed/quality depends on the user's hardware and chosen model size.
  • Option B — Add an optional cloud STT provider (e.g. a hosted Whisper API or similar managed transcription service). Pros: no local install/model download, consistent quality regardless of user hardware, potentially faster for large files. Cons: voice memos (often personal/sensitive) would leave the user's machine, requires API key provisioning and secret storage, recurring per-minute cost, new vendor security/privacy review, and a network dependency. This directly conflicts with the "local-first," "no plaintext secrets," and "no audio ever leaves this machine" constraints unless explicitly approved.

Recommendation: keep Option A as the only shipped path; revisit Option B only if user feedback shows the local-engine install friction is a real adoption blocker, and only after CTO/CEO sign-off on vendor choice, data handling, and cost.

Test plan

  • npm run test:notes — 100 passed, 0 failed (includes new stt.js + transcribe round-trip cases)
  • npm run test:mcp — 60 passed, 0 failed (includes new note_transcribe tool cases, tool count 21→22)
  • npm run test:docs — 84 passed, 0 failed (includes fix for Commander help-text line-wrap mis-parsing)
  • npm run test:all — all suites pass
  • npm run lint — clean (pre-existing manifest permission warnings unrelated to this change)
  • Manually verified the NO_ENGINE clean error path (no whisper.cpp/faster-whisper installed in this environment) names both engines and install hints

🤖 Generated with Claude Code

amirhmoradi and others added 3 commits September 27, 2026 23:16
…331)

Add `tb notes transcribe <audioFile>` and MCP tool `note_transcribe`,
transcribing audio to text with a locally installed speech-to-text
engine (whisper.cpp or faster-whisper, auto-detected on PATH or via
TB_WHISPER_CPP_BIN/TB_FASTER_WHISPER_BIN) and saving the result as a
note with source/engine metadata in its front matter.

No audio or text ever leaves the machine: there is no cloud STT path
and no API keys. When no local engine is found, the error names both
supported engines and how to install them. Cloud STT is a separate
vendor decision left for the PR to route through CTO/CEO approval.

Tests cover the stt.js engine-selection/error paths and the CLI/MCP
surfaces using stub engine binaries, with no real audio processing
required.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- SPEC.md: keep the transcribe write-up, take main's renamed
  "Calendars and calendar events" heading, and recompute the CLI/MCP
  command-count table from the actual merged source (71 CLI leaf
  commands, 38 MCP tools) rather than either side's stale numbers.
- test/mcp-test.mjs: update the two hardcoded tools/list-length
  assertions to 38 (main's 37 plus this branch's note_transcribe),
  confirmed by the actual tools/list response after merging.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
main advanced with PR #31 (deterministic skills: skill_today/week/clashes/from
and lib/skills.js), which touched the same files as the transcribe branch.

- cli/src/cli.js, mcp/src/tools.js: additive import conflicts — kept both the
  stt.js transcribe import and the lib/skills.js skill imports.
- SPEC.md: recomputed every CLI/MCP command-count mention from the merged
  source (75 CLI leaf commands via `.action(` count, 42 MCP tools via
  tool-definition count), including two counts that were already stale
  before this merge (architecture diagram, "curated tools" prose).
- test/mcp-test.mjs: updated the three hardcoded tools/list-length
  assertions to 42, confirmed by the live tools/list response
  (81/81 MCP tests passing).

Full test:all and lint pass clean (lint's one failure is an unrelated missing
built XPI artifact).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@amirhmoradi
amirhmoradi merged commit 7bda48c into main Sep 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant