Skip to content

feat(memories): titles, English glosses, and search translations - #161

Merged
spinsirr merged 5 commits into
mainfrom
feat/memory-title
Oct 21, 2026
Merged

spinsirr merged 5 commits into
mainfrom
feat/memory-title

Conversation

@corespeed-agent-ensemble

@corespeed-agent-ensemble corespeed-agent-ensemble Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR adds three things that help people and agents find Memories across languages:

  1. Memory title. An optional one-line name, at most 256 characters.
  2. English gloss. An optional line of English, at most 512 characters, for a Memory whose content is in another language. It gives the meaning and key terms.
  3. Search translations. A search can carry the same question in up to 3 other languages. Lore searches each one beside the query and returns one ranking.

Behavior changes

Title and gloss

  • validateMemoryTitle and validateMemoryGloss in lore-core hold both fields to one rule. The database CHECKs equal those validators. The fields flow through HTTP, the SDK, the CLI (--title, --no-title, --gloss, --no-gloss), MCP, Proposals, context packets, and the Workspace archive.
  • Search reads both fields through four new lexical "name" channels. A name match answers the Memory's first chunk. Titles and glosses are never chunked, embedded, or sent to the reranker.
  • Migrations:
    • 0015 adds the columns, rules, and Proposal fields.
    • 0016 copies each valid metadata.title into title, then validates the rules. If a request changes a Memory while the copy waits on that row, the request's change wins. bun run smoke:title-backfill checks this on PostgreSQL, and CI runs it.
    • 0017 builds the name indexes concurrently.
    • The schema is now revision 17. compatible_from stays 9, so an older application instance keeps working during a rolling deploy.
  • The MCP write tools ask the agent to send a gloss when the content is not in English.
  • MCP lore_list and lore_search rows cut a title past 300 or a gloss past 500 characters of JSON, and mark it with titleTruncated or glossTruncated. This keeps 25 maximal rows under the 128,000-character output limit. Detail and mutation responses return both whole.
  • In the web UI:
    • The editor has Title and English gloss fields.
    • Memory detail lists both under Properties.
    • Browse and search rows show the gloss as one muted line under the title.

Search translations

  • HTTP:
    • Memory search takes repeated translation parameters beside q. A translation without q is refused.
    • Episode evidence search takes translations.
    • A context packet takes memoryTranslations.
  • The SDK takes translations and memoryTranslations, and the CLI takes a repeatable --translation.
  • MCP lore_search, lore_search_evidence, and lore_retrieve_context tell the agent to send a faithful translation when the Workspace may hold another language.
  • Each translation runs as one more query, after the original query and before any planner rewrites, and goes through every lexical and dense channel. The same RRF fuses them all. The planner, feedback rounds, and reranker read only the original question.

Defaults and scope

  • Nothing changes unless a caller sends the new fields.
    • A search without translations runs exactly as before.
    • A Memory without a title or gloss behaves as before, and its first line is still its displayed title.
  • Lore never generates a gloss or a translation. The writing agent or the searching agent supplies it. Existing non-English Memories get no gloss: there is no backfill.

Decisions for the PM

  • Who writes glosses. The CoreSpeed HaaS fork generates glosses with Gemini Flash-Lite after each write. Lore OSS leaves glosses to the writer, and leans on search translations instead: an agent translates a short question more reliably than it summarizes content.
  • Wrong translations.
    • A wrong translation is in effect a second question, and its answers come back too.
    • The benchmark's worst case gives every case a translation that means a different question, one the Workspace can answer:
      • Answerable questions kept their Recall@5.
      • No-answer accuracy fell from 0.89–1.0 to 0.
    • An embedding check cannot tell right from wrong translations on qwen3-0.6b, because their similarity ranges overlap. So the guards are the MCP instruction to translate faithfully, and a reranker with LORE_RERANK_MIN_SCORE.
    • Without a reranker, the worst case can hand an agent an unrelated Memory on a question that has no answer.
  • The migration changes ETags. 0016 raises the version of each Memory that gets a copied title. A client holding that Memory's old ETag gets a version conflict on its next update. updated_at, list order, and cursors do not move.
  • New archive format. Export now writes lore-workspace-v3. A release before schema revision 15 cannot import it, so upgrade the target of a Workspace move first.
  • No translation echo. The context receipt does not repeat memoryTranslations. Three 10,000-character echoes would push Code evidence out of MCP's 128,000-character output limit.

Evidence

The retrieval-v1 suite is now version 3:

  • It adds 6 cross-lingual cases and 1 cross-lingual no-answer case, and gives translations to the 2 existing cross-lingual cases, for 8 cross-lingual cases in all.
  • It adds +translations variants, and +wrong-translations variants that measure the risk above.

Results, from Ollama qwen3-embedding:0.6b with no reranker or planner:

Setup Without translations With correct translations
Lexical only, cross-lingual Recall@5 0 1.0
Hybrid at threshold 0.5, cross-lingual Recall@1 0.75 0.75
Hybrid at threshold 0.5, cross-lingual Recall@5 1.0 1.0
Hybrid at thresholds 0.35–0.45, cross-lingual Recall@1 0.50–0.62 0.75–1.0

This suite holds only 34 Memories. On the HaaS production corpus, English questions about notes written mostly in CJK reached only Recall@5 63% with semantic search alone.

Local checks run on the merged head:

  • bun run test: 1,509 tests and 59 lore-core tests passed.
  • lint (no new warnings), typecheck, sdk:check, design:check, and architecture:check passed.
  • build, packages:smoke, and smoke:next passed.
  • The OpenNext Cloudflare build and wrangler deploy --dry-run passed.
  • A PGlite test upgrades a revision-14 database that holds data to revision 17.
  • On a real PostgreSQL 18, smoke:memory-core with a maintenance cycle passed, and db:migrate built a fresh database through 0017.

Dependency fix

bun audit --audit-level=high began failing on main too, from two new advisories:

  • markdown-it 15.0.2 renders one multi-line link reference definition in quadratic time (GHSA-gfr2-rv44-r23c).
  • vite 8.2.0, which vitest pulls in, has a Windows existence-check bypass (GHSA-587h-7wmj-h5pm).

A separate commit moves them to markdown-it 15.0.3 (and raises its floor) and vite 8.2.4, both within their ranges. All tests pass on the new versions.

Known follow-ups

  • Titles and glosses are not embedded or reranked. Changing that needs a new embedding-protocol revision and a versioned evaluation.
  • A CJK title matches a query only when they share two consecutive 3-grams, the same limit that applies to CJK content.
  • Workspace-owned Evaluation cases do not carry translations yet.

🤖 Generated with Claude Code

Agent and others added 5 commits October 21, 2026 21:28
A Memory gets an optional title (one line, at most 256 characters) and an
optional English gloss (one line, at most 512) for content in another
language. Both are held to lore-core validators and equal database CHECKs,
flow through HTTP, SDK, CLI, MCP, Proposals, the context packet, and the
Workspace archive (new format lore-workspace-v3), and are searched by four
lexical name channels that answer a Memory's first chunk. They are never
chunked or embedded.

Migrations 0015-0017 add the columns, rules, Proposal fields, and name
channels (0015), copy each valid metadata.title into title and validate the
rules (0016), and build the Workspace-leading name indexes concurrently
(0017). Schema revision 17, compatible_from 9.

The web UI edits both, shows them under Properties, and shows the gloss as
one muted line under the title in browse and search rows.

Search also takes translations: the same question in up to three other
languages, which the caller supplies. Memory search, Episode evidence
search, and context packets run each beside the query and fuse them with
the same RRF; the planner, feedback rounds, and reranker read only the
original. HTTP takes repeated `translation` parameters, `translations` in
an evidence search body, and `memoryTranslations` in a context packet. The
MCP search tools ask the agent to send faithful translations.

retrieval-v1 becomes suite version 3 with cross-lingual cases and
+translations / +wrong-translations variants; results are recorded in
docs/agent-conventions.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bun audit --audit-level=high now fails on main as well: markdown-it 15.0.2
renders one multi-line link reference definition in quadratic time
(GHSA-gfr2-rv44-r23c), and vite 8.2.0, which vitest pulls in, has a
Windows existence-check bypass (GHSA-587h-7wmj-h5pm). Move to markdown-it
15.0.3 (and raise its floor) and vite 8.2.4 within their ranges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lore_propose refused an update that changed or removed only the gloss:
its refinement listed content, scope, title, and metadata but not gloss.

lore_list and lore_search rows carried title and gloss without a budget,
so 25 rows of maximal names, metadata, and text passed the 128,000-
character output ceiling and the call returned only an error. Rows now
cut a title past 300 and a gloss past 500 characters of JSON and mark
titleTruncated or glossTruncated; detail and mutation responses return
both whole.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0016 picked its candidates and their trimmed titles in a CTE, then
joined them into the UPDATE. Under READ COMMITTED, an UPDATE that waits
on a row a request holds re-checks only its own quals against the row
the request commits; the CTE's values stay from the old snapshot. So a
request that titled a Memory, or rewrote or broke its metadata.title,
while the copy waited lost its change: the copy wrote the old
metadata.title and advanced the version again.

0016 now judges and reads each row in one UPDATE on memories, so the
re-check sees the committed row. 0016 is not applied anywhere yet.

bun run smoke:title-backfill proves it on PostgreSQL: a request holds
three rows uncommitted, the copy waits on them, the request commits, and
each request's change must win. CI runs it in the PostgreSQL job. The
smoke target checks and the lock-wait helper move to
scripts/checks/lib/smoke-database.ts so both smokes share them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@corespeed-hopper corespeed-hopper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at 9f20ff8 after four review rounds. No findings remain.

Reviewed the full change from merge base 99d0070, then each pushed delta. The migration now preserves concurrent title writes and reads updated metadata. I independently ran smoke:title-backfill on PostgreSQL 18. Both targeted MCP regressions passed: gloss-only proposals and bounded 25-item responses.

Immediately before approval, the current PR head matched the reviewed commit. All six required CI checks passed: static, application/Core tests, PostgreSQL, packages, web/Cloudflare build, and All checks. Evidence: https://github.com/corespeed-io/lore/actions/runs/39681118050

The PR description records the translation-quality trade-off, lexical-only names, changed migration ETags, and archive-v3 compatibility. Reviewer session: 16ffca81-94a7-409f-801f-608f937887b2.

@spinsirr
spinsirr merged commit f7abe1a into main Oct 21, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant