Skip to content

Teach the classifier from the user's own corrections #70

Description

@SociableSteve

Problem

The classifier's GTD rules are a fixed system prompt. It can be corrected the same way a hundred
times and will make the same call the hundred and first, because nothing it is shown at call time
says anything about the corrections.

The corrections are real, specific and repeated: a sender whose mail always ends up in reference
rather than someday, a class of pull request that is always review rather than next_action, a
kind of item the user consistently leaves in the inbox on purpose. Spec 04 anticipated exactly this
and named the way out while listing it as a v1 non-goal: "Learning or fine-tuning from corrections
in v1. The classifications table exists so that few-shot examples from real corrections can be
added later."

Expected outcome

The classifier is shown a small set of the user's own past corrections, and stops repeating the
ones it has been corrected on.

  • A correction the user made is capable of changing a later classification of a similar item. This
    is the criterion the issue exists for, and it is demonstrable: the same input classified before
    and after a correction on a similar item can differ.
  • A human decision still beats a machine one absolutely (spec 00). Nothing here lets an example
    override a status the user set, or re-open a task that has left the inbox.
  • What was shown to the model is recoverable for any given classification, alongside the model and
    prompt version already recorded. A behaviour change traceable to the prompt today stays traceable
    when the examples are what changed.
  • The set is bounded and the bound is stated, so a growing history cannot grow the payload without
    limit or crowd out the rules themselves.
  • Selection is deterministic. The same history and the same task select the same examples, so a
    classification is reproducible and a wrong lesson is findable rather than intermittent.
  • Nothing is sent that spec 09's content policy would not already allow to be sent for the item it
    came from. A correction on an item is not a licence to send more of that item than the policy
    permits.
  • A correction the user reverses stops being taught.
  • An install with no corrections yet classifies exactly as it does today.

Notes

Depends on the decision log, which is what makes a correction a durable fact rather than an
inference from current task state. Spec 04's non-goal and acceptance criteria move with this.

Depends on the classification measurement too, and for a reason worth stating: the characteristic
failure of few-shot examples is a confident regression on the cases the examples do not cover.
Without a baseline agreement rate there is no way to tell afterwards whether this helped, hurt, or
did nothing.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions