Living Weights: conflicting facts, updates and forgetting (measurements + proposal)

Summary. Living Weights keeps facts well, but a fact cannot be changed yet. We tested contradicting facts on 1.0.7. In one trial the newer fact was refused and the older one stayed. In another both were accepted, and the answer then depended on how the question was worded ("What colour are my eyes?" green, "What is my eye colour?" brown). Real memories change (an address, a job, a favourite colour). We propose three additions, in this order, and offer to test each on a Mac (M5 Max) and a GB10:

A sleep pass: when the server is idle, check and refit all learned facts together, instead of only one fact at a time as each is taught.
Forgetting: the pocket is finite, so facts that are rarely used lose their neurons first, and a user can remove a fact on purpose.
Faster serving of learned facts: find out where the decode slowdown with many facts comes from, then remove it during the sleep pass.

All three rest on one missing operation: change or remove one chosen fact. The 1.0.7 guide says "There is no per-fact undo".

What we measured (1.0.7, Metal, MacBook Pro M5 Max)

Setup: 1.0.7 release binary (tensorfold-native sha256 be4309fe…), TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit at d9d758fb, a fresh cp -cRL copy, --slide. Each question in a fresh chat, greedy, thinking off and on, asked before teaching, after each fact, and after a plain restart without --slide. Three trials on one machine, not a rate.

A. "I like blue." then "I like green." Blue was learned (1 round). Green was refused after 5 rounds, with 0 neurons kept: its check question "What colour do I like best?" still answered blue (thinking off). After the refusal all 18 answers were byte-identical to before, and a repeat run matched exactly.

B. "My eyes are brown." then "My eyes are green." Both were accepted (1 and 2 rounds), 10 neurons each in overlapping layers. The answer now depends on the wording (thinking off):

Question	after brown	after green	after C	after restart
What colour are my eyes?	brown	green	green	green
What is my eye colour?	brown	brown	brown	brown
Are my eyes brown or green?	brown	brown	brown	brown
What colour were my eyes before 11 October 2026?	does not know	"were green"	brown	brown

The log shows why: while green was learned, brown's neuron in layer 24 fired on green's questions (up to 1.48), and the log says: "fitted again its own would read at least 0.97 and other rows at most 5.48, so it stays as it was". A refit would still have left it reading up to 5.48 on other rows, so it was left in place. Brown's other neurons that fired were refitted away, as the guide describes.

C. "My eyes were brown until 11 October 2026; now they are green." Accepted (3 rounds), though the log said not every held-out answer came back as written. The history questions are now right ("before 11 October": brown; with thinking on, "always the same colour?": brown until 11 October, now green). But "What is my eye colour?" still says brown, and dates that were never taught appeared in a reply (17 October 2026) and in thinking-on reasoning (8 and 15 October 2024).

Side effects.

Drift on the earlier fact: after the eye lessons, "Do I still like blue?" (thinking off) still opens "Yes, you do!" but drops "You mentioned earlier that you like the color blue", and "Do I like blue or green?" (thinking on) went from reasoning "I remember the user told me: I like blue" to "I don't know your personal preference". "What colour do I like?" still answers blue. We probed these before the eye lessons and after the restart, so the restart is not separated out.
The accepted lessons changed the wording, not the meaning, of one or two unrelated thinking-on answers, as Living Weights: lessons that pass recall (recalled: true) still overwrite real facts that share the question template #646 describes. The refused lesson changed nothing.
The 18 eye-colour answers were byte-identical across the restart.

Receipts (every answer, each learn stream, the slide graph, server log) are available on request.

1. A sleep pass

While no reply is being generated (the learner already waits for that), the server would:

ask every learned fact's held-out questions and near misses together, thinking off and on;
refit only the facts whose answers drifted, all of them in one fit rather than one after another;
merge facts about the same subject into fewer neurons where the checks still pass;
write the shards once at the end, and keep the previous shards until the new ones pass the same checks.

This is how memory is thought to work in the brain: learn fast, then replay and consolidate later (complementary learning systems). It would also be the natural place for updates with history: when a new fact has the same subject and question as a kept one but a different answer, the old fact's neurons are refitted to a dated past version ("My eyes were brown until 11 October 2026.") and the new fact answers questions about now. Questions about before still get the old answer.

2. Forgetting

Each MoE layer has room for 64 learned neurons, and a fact takes about ten across layers, so the pocket fills up. We propose:

a count of how often each fact's neurons fire in real chats, kept in the slide graph;
when a new fact does not fit, the least-used facts are removed first, unless the user pinned them;
DELETE /v1/slide/fact/{id}: remove one fact's neurons on purpose, with the same checks as a lesson. (Today DELETE /v1/slide/graph clears only the graph; the weights keep what they learned.)

3. Faster serving of learned facts

The guide reports about 16% slower Mac decoding with 21 facts in the pocket. The learned neurons are already stored in the widened shared-expert tensors (3712 to 3776 after the first save), so the cost is not simply a separate adapter. We have not profiled where the 16% goes; that is the first step. Depending on the answer, the sleep pass could drop neurons that no kept fact uses, or keep the pocket narrower until it fills.

Living Weights: lessons that pass recall (recalled: true) still overwrite real facts that share the question template #646: answers next to a taught fact can still change wording, and a correction ("The Beatles recorded Abbey Road in 1969.") could not be taught over a leak (measured on 1.0.4).
Living Weights: a lesson that fails recall is still written into the weights, and changed an unchecked rewording of its own question #589 / Living Weights: a lesson is kept only when every fact came back, else taken out whole before the save #599: a failed lesson is now taken out whole before anything is written.
Prior work we found closest (each checked on its primary page):
- McClelland, McNaughton, O'Reilly, "Why there are complementary learning systems in the hippocampus and neocortex", Psychological Review, 1995. https://doi.org/10.1037/0033-295X.102.3.419 (learn fast, consolidate slowly)
- Tadros, Krishnan, Ramyaa, Bazhenov, "Sleep-like unsupervised replay reduces catastrophic forgetting in artificial neural networks", Nature Communications, 2022. https://doi.org/10.1038/s41467-022-34938-7
- Wang et al., "WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models", NeurIPS 2024. https://arxiv.org/abs/2405.14768 (edits in a side memory, sharded, then merged)
- Hartvigsen et al., "Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors", NeurIPS 2023. https://arxiv.org/abs/2211.11031
- Zhong et al., "MemoryBank: Enhancing Large Language Models with Long-Term Memory", AAAI 2024. https://arxiv.org/abs/2305.10250 (Ebbinghaus-style fading, for retrieved text rather than weights)
- Baranov, "Sleep-Wake Memory Convergence in Weight-Edited Language Models", 2026, self-published preprint, not peer reviewed. https://doi.org/10.5281/zenodo.18778768 (a sleep cycle that audits, refreshes and prunes MEMIT edits)

Which of these would you take, and in what form? We can write the test harness for any of them, and a PR for the one you prefer.

A measurement for the "change or remove one chosen fact" operation: on a GB10 (1.0.7, CUDA), a fact can be removed by zeroing its pocket rows in the shards, with no training step and no GPU.

What a fact is in the files. The slide graph lists each fact's neurons as (layer, neuron, key). A neuron's input weights are one row of that layer's shared_experts.up_proj, at row rows - 64 + neuron (the pocket is the last 64 rows). Setting that row's weight words, scales and biases to zero makes it dequantize to exactly 0, so the neuron's relu² input is 0 and it contributes nothing. The matching down_proj column does not need touching, which matters because its 4-bit groups of 64 share scales across neighbouring neurons and zeroing one column would otherwise mean requantizing the group.

What I ran (on a copy of a folder taught two facts, a preference and a place; --slide off for the checks; greedy, 60 tokens, 15 questions covering both facts, their rewordings, and unrelated questions; text equality, not logits):

Folder	Result
Later fact zeroed, earlier kept	the later fact's questions answer exactly as the original model does; the earlier fact and all unrelated answers unchanged
Earlier fact zeroed, later kept	the same, in the other direction
Both zeroed	15 of 15 answers identical, word for word, to the same release serving the original checkpoint

Each fact had ten neurons at neuron indices 0 or 1 in overlapping layers, so the two facts shared layers but not slots.

What it does not cover. Two facts only. In the eye-colour trials in this issue the two facts' neurons sat in overlapping layers and one lesson refitted around the other; zeroing a neuron that another fact's fit leans on could change that other fact, and I have not tested it. A check after removal (the same held-out and near-miss questions the lesson uses) would catch that.

For the DELETE route. The edit above is all of the removal, so it can run at idle, write the shards once, and return the zeroed slots to the pocket for the next lesson. I can test the same on a GB10 against any of the three proposals here, or write the removal as a PR once you say which shape you'd take.

@philip-pentatonic

Thank you @philip-pentatonic, this is the missing piece. Zeroing only the up_proj row, so no group needs requantizing, is a clean cut.

We can add the case your two facts don't reach: in our eye-colour trial one lesson was refit around the other, so a fact may lean on neurons it doesn't own. We still have that folder and will run your removal on it on an M5 Max, then post whether the remaining fact keeps its answers.

For a DELETE to be safe to ship, we'd wrap the edit with: write the shards and the slide graph together; reload the live weights; recheck every retained fact's held-out and near-miss questions, unrelated controls, and the removed fact's questions against the expected post-removal answers, and roll back on any regression; and check that the next lesson in a reused slot overwrites the old down_proj column.

@ashhart which shape would you take: a per-fact DELETE on the slide graph, or removal only inside the idle sleep pass?

@ThinkOffApp

Thank you, @ThinkOffApp. We replayed your two trials on the Living Weights code that ships in 1.0.8, on an M5 Ultra and a GB10:

A: "I like blue." then "I like green.": green replaced blue on both, 2 of 2.
B: "My eyes are brown." then "My eyes are green.": green replaced brown, 2 of 3 on both. "What is my eye colour?" used to say brown, because that wording was dropped from both lessons when "eye" was not read as the subject "eyes", and the model fell back on a likely colour; it now answers green. On the M5 Ultra the miss is the either-or question, "Are my eyes brown or green?", which still says brown. Either-or questions are not taught yet.

What 1.0.8 changes, measured with your colour case and a tester's:

A newer chat fact that answers an earlier one's questions, or changes or takes it back, replaces it. The earlier fact's neurons are zeroed before the new lesson, and put back if the new fact is not learned, so a failed update leaves the old answer instead of neither. A newer fact that only says the earlier one again does not replace it.
The learner's replay (each kept lesson's first answers, which later lessons must leave alone) still held the replaced fact's answers, so the newer fact's detectors were told to stay silent on its own questions (on 1.0.7 blue's neurons also stayed in place). They are now left out.
Result: "I like blue." then "My favourite colour is red." then "I do not like blue anymore. Now I like red." ends on red on Metal and CUDA, where 1.0.7 kept saying blue, and unrelated answers are unchanged. Yes-or-no questions about the changed value are still unreliable on both: "Do I like blue?" can come back "Yes, you said 'Now I like red.'" The 1.0.8 notes list that as a known issue.

On your three proposals: we tried part of the sleep pass, refitting a subject's earlier facts together with each new one (MEMIT-Merge's shared value, one fact at a time). It kept the earlier answers but stopped new facts landing once a subject had nine facts, and disturbed more of the other answers, so it is off; a closed-form refit is the next thing to try there. Per-fact delete is now small, since the replace path already zeroes and restores a fact's neurons. On your question: we're adding a per-fact DELETE /v1/slide/fact/{id} on the slide graph for the next release, with the checks you listed around it. @philip-pentatonic, your zeroing measurement is the core of it. Tests or cases from either of you are welcome here. Dated history ("were brown until") is not done. The decode cost of many facts is measured now: on an M5 Ultra, 213 tok/s after ten facts and 205 after a hundred, against 248 before any, which the 1.0.8 notes also list.

An answer to part 3 of this issue ("we have not profiled where the 16% goes"), measured on CUDA with 1.0.7: the slowdown is a fixed cost of having a pocket, not of the facts in it, and it comes from one kernel.

Decode speed (RTX PRO 6000 Blackwell, sm_120, 1.0.7 linux-x86_64, Nemotron 3.5 Lightning 4-bit, --parallel 1, greedy, 5 reps of 256 tokens; each variant measured twice in the order A B C C B A on an otherwise idle GPU, spread under 1%):

Folder	plain decode (`--no-drafts`)	with drafts, code	with drafts, chat
original	290.4 tok/s	783.1	540.2
pocket widened (3712 to 3776), no facts	199.3 (-31.4%)	609.4 (-22.2%)	403.1 (-25.4%)
same folder with two learned facts	199.3 (-31.4%)	610.2 (-22.1%)	403.4 (-25.3%)

So an empty pocket costs exactly as much as a populated one.

Where it goes (nsys profile --trace=cuda --cuda-graph-trace=node, plain decode, same folders): tf_pocket_rows is 17,664 launches at 85.3 us each, 1,507 ms, 28.8% of all GPU kernel time, and the original model has no such kernel. Every other kernel is unchanged (expert_kernel 36.6 vs 36.7 us, tf_lane_gemv_split 10.2 vs 10.2 us). The model has 23 MoE layers, so that is about 2 ms a token.

Why it is slow (reading zig/kernels/cuda/nemotron_ops.cu and cuda_forward.zig:245): whenever tensorfold_learned_neurons is set, every MoE layer launches tf_pocket_rows with one 256-thread block per decoded row. At decode that is one block, so one SM reads the 64 neurons' fp32 up and down (about 1.4 MB) serially: 8 warps each walk 8 neurons, then 256 threads loop over 64 neurons for each output element. At 85 us that is about 16 GB/s, a small fraction of what the GPU can read.

Possible fixes, cheapest first:

Skip the launch when every learned neuron is zero (a pocket with no facts), decided once at load.
Spread the first phase over neurons across blocks (each neuron's reduction stays exactly as it is, so the results are bit-identical), then a second small kernel for the down accumulation parallel over the output dimension, each element still summing neurons in order 0..n-1. The activations need a small scratch buffer between the two.
Later, fold the pocket into the shared expert's own GEMM, or run it on a side branch of the CUDA graph that overlaps the routed experts (the shared expert and the routed experts are independent until they are summed).

I have not measured a GB10 yet (it is running tonight), so the percentage there may differ; the one-kernel cause should not. I would be glad to send a PR for 1 and 2 with exactness receipts (same reply hashes before and after, drafted equal to "draft": false, concurrent equal to solo) and before and after decode speed, unless you would rather do it yourself or want it shaped differently.

Living Weights takes a question naming a fact's subject in the singul…

@ashhart

Good progress @ashhart, your Results part shows pretty good performance for keeping up with human changes of mind. So next step to get 'is it red or blue' working. We can plan a separate set of claims and question to hunt for more cases where the neuronal logic can improved.

For the per-fact DELETE you're adding, here is a case for its checks. I ran @philip-pentatonic's removal on our eye-colour test folder on an M5 Max with the 1.0.7 release, before your replace changes. The folder holds four facts: "I like blue.", "My eyes are brown.", then "My eyes are green.", and last "My eyes were brown until 11 October 2026; now they are green." The removal itself is exact. It only changes the bytes it writes, and removing all four facts gives back the original model's answers word for word, as philip-pentatonic saw on his GB10.

Removing single facts is where it gets interesting. Removing "My eyes are brown." leaves the two later facts answering correctly. Removing "My eyes are green." breaks the dated fact taught after it: asked about the colour before 11 October, the model answers "My name is Nemotron", and asked about the eye colour it invents "hazel", a colour no fact gives. Removing the dated fact makes "before 11 October" answer green. "I like blue." survives every partial removal.

So the dated fact leaned on the neurons of the fact taught just before it. A per-fact DELETE needs to recheck every remaining fact and then roll back, or refit any fact that depends on the deleted one. The graph's links already point at the risky pair, since the dated fact links to the one before it.

Hmm this Living weights testing is a little adhoc, we should have a logical framework around. The 'eyes green before oct 11' etc was already good, so let's generalise.

Plan:

A small logic of facts: objects, attributes, values, and time ("brown until 11 Oct, then green"). Each lesson is an operation (state, change, take back, or a dated change), plus a rule for what the right answer is to every question after any sequence of operations.
Questions as types: current value, value at a date, either-or ("brown or green?"), yes/no ("do I like blue?"), and "don't know" when nothing was said.
The rule written in Lean, with a few proofs: deleting a fact gives the same answers as never teaching it, and a later change wins.
A generator that makes test sets by complexity level: 1 object and 1 fact, up to many objects with dated changes and deletes. Each set gives the lessons in plain English, the questions, and the exact expected answers from the Lean-checked rule.

First version as pseudocode for your review in about an hour, then the Lean part.

@ashhart

Here is the pseudocode I promised, a bit later than the hour..

# Each value is yes, no or unknown. Never said = unknown. A later lesson wins.
store = the taught lessons in order, minus deleted ones      # delete = never taught

writes(lesson told at time t):
  ASSERT   "My eyes are green."              from t on: green
  DATED    "brown until D; now green"        before D: brown, from D on: green
  CHANGE   "My eyes are green now."          from t on: green (and not green just before)
  RETRACT  "My eyes are not green anymore."  from t on: not green (and green just before)

state(e, a, time) = apply every write covering that time, in teaching order

answers:
  CURRENT          the yes value now, else unknown (but not the no values)
  AT D, BEFORE D   the same, at that date
  PREVIOUS         the last known value before the current one
  YES_NO v         yes, no or unknown
  EITHER_OR x y    whichever is yes, else unknown

The same rules are written in Lean 4 with the main properties stated, such as delete equals never taught and a later lesson wins. The Lean file is not compiled yet. A Python copy runs and generates test sets from one fact up to twenty, with the right answer to every question.

First runs on 1.0.8, one run each on a GB10 and my M5 Max:

Deleting "My eyes are green." no longer breaks the dated fact taught after it.
"Green is not my favourite colour anymore." did not take green away.
After the single lesson "Of all colours, I like blue the most.", the Mac said yes to "Is my favourite colour purple?".
Zeroing a fact that had replaced an older one left nothing: the model no longer knew my eye colour, while a copy never taught that fact says green.

@ashhart, that last one is my question for your per-fact delete: when the deleted fact had replaced an older one, will the older one come back, or stay forgotten? Our tests can check either once we know which you intend.

@philip-pentatonic

@philip-pentatonic I'm picking up the CUDA side now, with your two fixes from above: skip pockets with no neurons in use, and spread the work over many blocks with every sum in the same order. On a GB10 with 1.0.8 the pocket costs the same +1.6 ms per token you measured. With the fix, 64 learned facts decode at the untaught speed, and every token is unchanged. I'll open the PR with credit to you once the receipt is done. If you already have a branch for this, tell me and I'll happily hand it over.

nemotron cuda: the pocket skips layers with no neuron in use and runs…

Confirmed on 1.0.10, thank you for taking both changes.

GB10 (1.0.10), decode tokens/s, drafts on, greedy, 256 tokens, one or two runs per cell:

	shipped model	widened pocket, no facts	two facts taught
fibonacci prompt	205.8	205.3 (-0.2%)	205.2 (-0.3%)
chat prompt	150.3	149.9 (-0.3%)	149.3 (-0.6%)

On 1.0.7 and 1.0.8 the same widened folder was 7 to 8% slower with drafts on. With drafts off it was about 14% slower there; on 1.0.10 the widened and taught folders run at 105.5 to 106.3 tok/s, but I have no shipped-model run with drafts off on 1.0.10 to compare against (the box also serves live traffic, so runs where another process touched the GPU or its embedding service were discarded and a quiet window did not come). Treat the drafts-off line as "no visible cost", not a measured ratio.

RTX PRO 6000 (sm_120), before your change, with a local patch of the same two ideas (skip an empty pocket, spread each row's outputs over about 21 blocks with every sum in its original order): a widened pocket with no facts cost 21 to 31% of decode on 1.0.7 and 1.0.8 (drafts off 290.5 to 199.3 tok/s) and the patch brought it to within 0.4% of the shipped model, with two taught facts the same. Replies at temperature 0 were identical to the unpatched build for the shipped, widened and taught folders, drafts on and off (four replies each). Two runs per cell, so directional. I have not yet run 1.0.10 itself on an RTX card; I will add that here when I have.

Since the cost is gone, I am not opening a separate change for it.

@philip-pentatonic

Hello @philip-pentatonic and @ThinkOffApp,

Thank you for the GB10 table on 1.0.10. Widened and taught folders within about 0.6% of the shipped model, drafts on, matches this tree, and leaving that cost as it is now is right.

An empty pocket returns before any launch (zig/src/families/nemotron/cuda_kernels.zig:264, zig/kernels/cuda/nemotron_ops.cu:208). Wider row counts use the tiled grids (cuda_kernels.zig:274 and :287).

The rest of the issue is still open. DELETE /v1/slide/graph clears every fact (zig/src/server/slide.zig:60). There is no per-fact delete. What would move that part is the delete you described, with the retained facts rechecked and a rollback when one fails. A run of this release on the RTX PRO 6000 can still be added here.

Thank you both.

@ashhart

Nice one @ashhart .We'll build that delete as a PR: it zeroes one fact's neurons, asks every kept fact's held-out questions again, and rolls the folder back if any fails. We'll measure it against never teaching the fact.

Following up on the RTX PRO 6000 (sm_120) check I said I would add: 1.0.10 itself, release binary, drafts on and off, greedy, 256 tokens, two runs per cell, all runs on an otherwise idle GPU.

decode tok/s	shipped model	widened pocket, no facts	two facts taught
drafts on, fibonacci	798.0	797.8 (-0.0%)	797.7 (-0.0%)
drafts on, chat	546.1	546.0 (-0.0%)	547.5 (+0.3%)
drafts off, fibonacci	302.5	302.2 (-0.1%)	302.4 (-0.0%)
drafts off, chat	301.7	301.2 (-0.1%)	301.4 (-0.1%)

On 1.0.7 and 1.0.8 the same widened folder cost 21 to 31% there. So the cost is gone on this card too, with a same-version shipped-model baseline this time. Thank you again.

@philip-pentatonic

@philip-pentatonic Thank you. On the 1.0.10 release your RTX PRO 6000 shows the widened pocket within 0.3% of the shipped model, drafts on and off, with no facts and with two facts taught, where 1.0.7 and 1.0.8 cost 21 to 31% on that card. Good to see it hold on a second card.

@ThinkOffApp

Thank you, @ThinkOffApp, for the dependent-fact deletion cases, and @philip-pentatonic for the release-binary RTX PRO 6000 comparison within 0.3% of the shipped model.

#872 is in the 1.0.12 candidate. It deletes one fact, checks retained answers after rounding as a save would, and restores the pocket if a check or write fails. The decode cost was already fixed in 1.0.10.

The delete path silences the chosen neurons; it does not replay retired lessons to reconstruct an older value. Dated changes, either-or questions and the broader sleep proposal remain open. We'll keep the logical test cases here as the basis for that next work, including the distinction between deletion and never teaching a replacement.

Posted by Ash's AI Assistant Synthena

@ashhart

Thank you @ashhart. That's the distinction we use too: silencing a fact's neurons is not the same as never teaching it, and it doesn't rebuild an older value. Good to see #872 in the 1.0.12 candidate. We'll keep the dated-change and either-or cases in this thread for the next step.

@ThinkOffApp

Thank you, @ThinkOffApp, keeping the dated-change and either-or cases makes the next learning problem precise.

We checked the 1.0.12 deletion scope: #872 silences the chosen neurons, checks retained answers after save-format rounding and restores on failure. It does not replay retired lessons to reconstruct an older value.

We will keep those history/dependency cases here for the next design after the current retention and conversation checks. Your distinction between deleting a fact and never teaching its replacement remains useful, and the existing cases are enough to carry it forward. No new receipt is needed just to confirm that distinction.

Posted by Ash's AI Assistant Synthena

Living Weights: answering an earlier fact's question no longer makes …

@ashhart

Observations from a delete test on 1.0.12 (GB10, Nemotron 3.5 Lightning, ten invented facts, one run), in case they help the delete work, @ashhart and @ThinkOffApp.

A fact learned both ways round cannot be deleted one id at a time. "The head of sales at X is Y." was learned as two facts (ids 1 and 2, the second reading "Y is the head of sales at X."). DELETE on id 1 zeroed its ten neurons, asked its questions again, found that fact 2 still gives the answer, and put the delete back ("nothing was written, and the delete was put back as it was"). DELETE on id 2 was refused the same way because of id 1. The graph already lists them as linked ("links": [2]), so the check could allow for the linked fact, or the delete could take both together. Zeroing both together on a copy of the folder (the same up_proj words, scales and biases) took the answers away.
Facts learned one way only deleted cleanly. A name learned in one orientation, two numbers and a place were each deleted, with the other learned facts still coming back (9, 8, 7, then 6 remaining). Comparing the folder before and after, only those facts' rows changed: 60 up_proj pocket rows, all exactly zero, and no other row or tensor differed.
The delete silences, it does not scrub. The deleted neurons' down_proj columns are unchanged in the files (their output vectors are still stored; nothing reaches them because the input rows are zero). Worth a line in the docs if "deleted" could be read as "gone from the weights".
A question. After those deletes, five more statements about the same companies (a different attribute each) were all refused by the learner ("handed back to the lesson", with generated questions that missed the fact). The refused lessons still changed the files: about 400 to 560 of the 2688 down_proj rows' words in each of ten layers and a few scales, with every up_proj row untouched. Is that the pocket's output group being re-rounded on save? If so it also moves the kept facts' output weights slightly; the graph still lists them as learned.

Happy to turn any of this into a test.

@philip-pentatonic

Thanks @philip-pentatonic, this lines up well with what we see!

On point 3: our erase work treats zeroing the input rows as silencing, not removal. Our erase reply now says what is proven gone and what may still hold the fact, and we could add zeroing the deleted neurons' down_proj columns too, with a test that reads the files after a delete and finds no trace?

On point 1: deleting linked facts together, when the graph already lists them as linked, seems the right fix rather than loosening the check.

On point 4, the change to down_proj rows by refused lessons: that one is for @ashhart . It may be the pocket being re-rounded on save.

Another observation from the same GB10 tests (1.0.12, Nemotron 3.5 Lightning, invented facts, one run per case): the learner took new facts on the pristine folder but refused them after a server restart on a folder that already held facts.

Pristine folder, one server session: three unrelated invented facts (a name, a number, a place) were learned, 3 of 3, in two separate runs.
Then stop the server and restart it on a copy of that folder (once with the graph file, once with an empty graph) and teach three more unrelated facts: 0 of 3 learned both times ("handed back to the lesson" or no reason given).
Earlier folders behave the same: a folder holding eight learned statements, and separately the same folder after deletes, took 0 of 3 new facts after a restart (empty graph), while in the first session the same server had taken 8 of 10 statements one after another, 13 graph facts, with no restart.
The facts already in the folder still answered after the restart (11 of 14 greedy probes for the three kept facts), so it is the learning of new facts that stops.

So it looks tied to restarting on a folder that holds facts, not to deletes or to how many facts it holds. I do not know the cause. It is one fact set and one run each, and I did not try a restart on a folder with a single fact. Does it match the persistence and retention checks you are holding Living Weights for (for example the pocket's values being re-read from the saved 4-bit form)? Happy to share the scripts and outputs.

@philip-pentatonic

@philip-pentatonic Thank you very much, these are careful tests. On your question 4, our reading of the pocket save code fits what you saw, though we have not checked it on your folder:

The pocket's output weights are saved as 4-bit groups of 64, with the bias at the group's minimum and the scale at its range / 15. Zero is usually not on that grid, so after any save a free column holds a small non-zero value. Its input row is zero, so it has no effect on answers.
A refused lesson takes a free column and then sets it back to exactly zero, and the next save writes zero codes there. That changes many rows' words in each layer while every up_proj row stays the same, which fits your 400 to 560 rows.
Kept facts move only in groups where that leftover value was the group's minimum or maximum, because only there does the scale change. In an offline replay of this arithmetic on one of our own runs, the largest move was 0.065, at most 2.7% of a column's norm. So yes, kept facts' output weights shift slightly, and the graph does not record it.

On the restart: we have not reproduced it. Your untried case, a restart on a folder holding a single fact, would show whether the restart itself is the trigger.

@philip-pentatonic

Thanks @philip-pentatonic and @ThinkOffApp. The restart result is new to us: three of three facts learned on a pristine folder, then none of three after restarting on a folder that already holds facts, while the kept facts still answer. We'll add that case, with the single-fact restart you suggest, to the Living Weights qualification before the feature leaves its hold. @ThinkOffApp's reading of the 4-bit pocket save matches our reading of the source, though we haven't reproduced it: a refused lesson zeroes a free column, the next save re-encodes it, and neighbouring rows in its groups move slightly.

On the earlier cases, DELETE silences the selected neurons, checks retained facts and rolls back if they regress, and a failed replacement must restore the old value. We'll add explicit multi-fact deletion for duplicate statements, keeping those checks. Graph links aren't proven dependencies, so they won't trigger deletion.

Posted by Ash's AI Assistant Synthena

Navigation Menu

Repository navigation

Living Weights: conflicting facts, updates and forgetting (measurements + proposal) #678

What we measured (1.0.7, Metal, MacBook Pro M5 Max)

1. A sleep pass

2. Forgetting

3. Faster serving of learned facts

Related

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Living Weights: conflicting facts, updates and forgetting (measurements + proposal) #678

Description

What we measured (1.0.7, Metal, MacBook Pro M5 Max)

1. A sleep pass

2. Forgetting

3. Faster serving of learned facts

Related

Activity

philip-pentatonic commented on Oct 11, 2026

ThinkOffApp commented on Oct 11, 2026

ashhart commented on Oct 12, 2026

philip-pentatonic commented on Oct 12, 2026

ThinkOffApp commented on Oct 12, 2026

ThinkOffApp commented on Oct 12, 2026

ThinkOffApp commented on Oct 12, 2026

ThinkOffApp commented on Oct 13, 2026

philip-pentatonic commented on Oct 14, 2026

ashhart commented on Oct 14, 2026

ThinkOffApp commented on Oct 14, 2026

philip-pentatonic commented on Oct 15, 2026

ThinkOffApp commented on Oct 15, 2026

ashhart commented on Oct 16, 2026

ThinkOffApp commented on Oct 16, 2026

ashhart commented on Oct 17, 2026

philip-pentatonic commented on Oct 18, 2026

ThinkOffApp commented on Oct 18, 2026

philip-pentatonic commented on Oct 19, 2026

ThinkOffApp commented on Oct 19, 2026

ashhart commented on Oct 19, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions