[schemas] Decision ledger with ranked recall - #488
Open
oSquashBlossomo wants to merge 2 commits into
Open
Conversation
Step-indexed decisions ledger over the agent-memory sidecar: auto-enrollment trigger for memory_type='decision' writes, importance and rationale columns, dependency edges, the match_decision_ledger() ranking RPC (relevance x importance x step-recency x dependency degree, with a full-text fallback so recall works with no embedding service), usage feedback bookkeeping, and a per-policy recall stats view. Sidecar-only: public.thoughts and every agent-memory table untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
match_decision_ledger() scored worse than its documentation claimed, in five ways found while running a 1024-dim port of this schema under a local agent brain. Each is fixed here and each is now documented in the RPC's own header and the README's ranking section. 1. The full-text query was AND-semantics. websearch_to_tsquery ANDs its terms, so a natural-language question matched a row only if that one row contained every content word in the question: 0 of 15 rows matched, leaving the FTS arm — 45% of the score — dead for exactly the no-embedding-service setup it exists to serve. The query's lexemes are ORed now: same configuration, same stemming and stopword list, one operator changed. An all-stopword query reduces to the empty tsquery and ranks every row 0, quietly. 2. Relevance clipped instead of saturating. LEAST(ts_rank_cd * 10, 1.0) pinned every non-trivial match to exactly 1.0, after which recency alone decided the order. x / (1 + x) is monotone over the whole range, bounded by 1, and keeps the same scale constant. 3. Cosine similarity was unclamped. It is -1 on opposed vectors, not 0, and fed straight into the blend. Every component is now clamped into [0,1] before it is weighted, importance included, and each row reports which arm produced its relevance in a new relevance_source column so a partly-embedded corpus is visible rather than silent. 4. The dependency normaliser is now a parameter frozen at 3 rather than a bare constant, with the degree clamped at 0 below. Normalising by the candidate set's own maximum degree — the obvious alternative — makes a row's score depend on what else happened to match, and scored the same decision 0.333 and 0.5 across two queries. 5. The recency constant was inert. exp(-d/120) spreads 0.0165 of the score across 15 steps, which does nothing at session scale; the half-life is 30 now. Step distance is measured from max(the caller's step, the stream's own newest step), which is defined for every row, so a stale or absent caller step no longer floors every row to 1.0 — and that made the wall-clock fallback, and p_half_life_days with it, unnecessary. The RPC is dropped by name before it is recreated, so re-running the file cannot leave the previous signature behind for PostgREST to choose between. Still additive only: no ALTER on public.thoughts or any agent-memory table, RLS and service_role grants unchanged, and the migration still ends with NOTIFY pgrst. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Hey @oSquashBlossomo — welcome to Open Brain Source! 👋 Thanks for submitting your first PR. The automated review will run shortly and check things like metadata, folder structure, and README completeness. If anything needs fixing, the review comment will tell you exactly what. Once the automated checks pass, a human admin will review for quality and clarity. Expect a response within a few days. If you have questions, check out CONTRIBUTING.md or open an issue. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contribution Type
What does this do?
Adds a step-indexed decisions ledger on top of the agent-memory schema: every
memory_type = 'decision'write is auto-enrolled (via trigger — the write path doesn't change) with a step index, adjustable importance, an optional rationale, and dependency edges to other decisions. A ranked-recall RPC,match_decision_ledger(), scores entries by relevance × importance × step-recency × dependency degree — with a full-text fallback so recall works with no embedding service at all — and a per-policy stats view makes recall policies comparable side by side.Why this is more than
memory_type = 'decision'The agent-memory schema already stores decision rows, but recall reaches them only through vector similarity on the linked thought. In Stefania Druga's memory-harness experiments (Sakana.ai, AI Engineer 2026), a ranked decisions ledger beat plain vector RAG on long-horizon recall on both accuracy and token cost. This sidecar adds what a ledger needs and vector search lacks: where in the task a decision happened (
step_index), an explicit priority (importance), the why kept apart from the what (rationale), which decisions rest on which (agent_decision_edges), and a ranking RPC that blends those signals. Lifecycle state stays inagent_memories— the ledger reads it, never duplicates it.Requirements
upsert_thought→public.thoughts) — ledger rows hang off memories whose thoughts arrive through it.agent_memoriesand its sidecar tables).public.thoughtsand every agent-memory table are untouched (noALTERon any existing table). New tables carry RLS,service_rolepolicies, and explicit grants per the agent-memory pattern, and the migration ends withNOTIFY pgrst, 'reload schema';. Safe to run more than once.An optional companion patch (separate, not in this PR) wires the ledger into
integrations/agent-memory-apias arecall_policy: "ranked_ledger"option on/recall, with the policy recorded on every recall trace — README Step 3 documents the exact change.The ranking is measured, not guessed
A 1024-dim port of this schema has been running under a local agent brain, and running it turned up five things the first cut of
match_decision_ledger()got wrong. All five are fixed here, and the RPC's own header says why:websearch_to_tsqueryandplainto_tsqueryAND their terms, so a natural-language question matches a row only if that one row contains every content word in the question — 0 of 15 rows matched that way, which left the FTS arm, and with it 45% of the score, dead for exactly the no-embedding-service setup it exists to serve. The query's lexemes are ORed instead: same configuration, same stemming and stopword list, one operator changed.LEAST(ts_rank_cd * 10, 1.0)pins every non-trivial match to exactly1.0, after which recency alone decides the order;x / (1 + x)is monotone over the whole range and bounded by 1.[0,1]before it is weighted. Cosine similarity is-1on opposed vectors, not 0. Each row also reportsrelevance_source(cosine/fts/none), so a partly-embedded corpus is visible in the trace rather than silently ranking on structure alone.p_dependency_saturationdefaults to 3.exp(-d/120)spreads 0.0165 of the score across 15 steps, which is inert at session scale. Step distance is measured frommax(p_current_step, the stream's own newest step)— defined for every row, so a caller that passes a stale step or none at all no longer floors every row to a recency of1.0.All weights, the half-life and the saturation point remain function parameters; these are the defaults, not a hardcoding.
Testing
Tested on my own Open Brain instance in two forms:
service_rolegrants) installs green on a scratch Postgres with the Supabase environment present, re-runs idempotently, and is checked by a dual-install parity suite that diffs its installed schema against my production port line by line — every remaining difference is a reviewed allowlist entry. The five fixes above are no longer among them: they are shared by both installs now, which is the point of contributing them rather than keeping them local. What still differs is my deployment's own policy (a pin sidecar, a recall gate, a role model instead of RLS), not the ranking.Checklist
metadata.jsonis complete and validrequires.open_brain: true)