Judge each post with a brief of the product, and let a fitted model decide who is a lead

[kev1n]

Why

After #94, production buyer leads were 38% good and 18% bad on the 442 leads labelled 2026-09-22. Experiments on 2,362 labelled posts traced this to how Jev was used, not to Jev itself. Ten posts per request distracted the model. Similarity was asked as broad yes/no questions. The market knowledge Jev can't infer, such as price tier and adjacent products, was never in its state. Answers were combined with hand-set 0.5 cutoffs.

What

• Brief. The onboarding reading (one Muse call, still on low effort) also writes a brief:

  • the product's precise kind;
  • 6–10 neighbouring kinds it is not, each with why not;
  • who buys and who doesn't;
  • price, free plan and limits;
  • 6 example asks that are leads and 6 near misses.

  It is stored in projects.brief and never shown as the profile, because it may use what the site doesn't say.

  • Projects made before briefs get one from a new brief job. It's seeded at boot, starting 3 minutes in and spaced 5 seconds apart, so the outgoing revision can't claim it.
  • A profile edit queues a brief job too.
  • The job queues the rescore, so verdicts are judged once, with the brief.

• Judge. One post per request (SCORE_BATCH_SIZE 1; the shared reading stays at 10). It asks five narrow signal questions, plus three built from the brief:

  • wanted_kind: a choice between the product's kind and its neighbours;
  • author_group: which buyer or non-buyer group the author belongs to;
  • is_lead_like: a yes/no whose criteria carry the brief's good asks and near misses as examples.

• Decision. Seller, helper, resolved and no-need are still settled by the shared reading. After that, a logistic model over every answer decides whether someone is a lead. Its weights are in leadModelWeights.ts: one set with a brief, one without. The feed score is built from the same model.

• SCORER_VERSION is now 2026-09-22.2.

Measured

Out of fold, grouped by product, on 2,362 posts labelled against 172 products' own sites (2,257 of them with a brief), at the old gates' own recall:

	shown	good	bad	good kept
gates, one post per request, with a brief	590	62%	8%	363 / 650
this PR, with a low-effort brief	540	67%	7%	363 / 650
this PR, without a brief	587	64%	8%	374 / 672

End to end with scorer:eval on the 442 leads labelled 2026-09-22 (in-sample for the weights):

	shown	good	bad	good kept
#94	207	38%	18%	78 / 106
this PR	146	49%	13%	71 / 106

Effort. Timed and scored through this PR's pipeline on 167 sites and 2,256 posts:

reading effort	wait, median / p90	good	bad
no brief	29s / 36s	65%	9%
low (shipped)	32s / 42s	68%	7%
medium	53s / 77s	69%	6%
high	75s / 110s	70%	6%

Low adds about 3 seconds to onboarding. The brief job for older projects runs on low too, and the weights are fitted on low-effort briefs.

Cost.

• The brief is about $0.002 per product.
• The judge makes 10× the Jev requests at the same token price, which is under a cent per thousand posts.
• The rescore re-judges every stored verdict, about 205k posts.

Tests

vitest passes all 547 tests. jobs.test.ts "seeds a scan" now has a 60s timeout, because it walks every project in the test database.

🤖 Generated with Claude Code