Add numpy-only simulation/tuning harness for comparing aggregators

[wxiao0421]

Summary

Picking an aggregator (and its hyperparameters) is data-dependent, but there was no lightweight way to measure that choice against labeled examples. This PR adds sentinel.simulation, a numpy-only harness for tuning — no Ray, S3, or experiment trackers required.

• New module src/sentinel/simulation.py exposing LabeledGroup, score_groups, evaluate_groups, compare_aggregators, and run_grid_search, all re-exported from the public API in src/sentinel/__init__.py.
• Score once, compare cheaply. score_groups runs the expensive model step a single time; compare_aggregators and run_grid_search then sweep all six aggregators and hyperparameter combinations over those cached scores. Sweeping min_score_to_consider is free because _apply_threshold re-applies the exact rule calculate_rare_class_affinity uses, on already-computed scores.
• Three families of evaluation metrics so tuning isn't limited to ranking quality:
  • Ranking: roc_auc, recall_at_n, precision_at_n, rank_ratio
  • Threshold/classification: precision, recall, f1, false_positive_rate at a chosen or automatically best-F1 cutoff
  • Separation/distribution: mean_separation, cohens_d, ks_statistic (threshold-free)

Results on real data

The notebook section runs end-to-end on 30 controversial vs 30 Lex Fridman episodes and its outputs are committed, so reviewers see the actual conclusion rather than just the code.

With the library defaults (top_k=5, min_score_to_consider=0.1) the aggregators are bunched together and the choice looks unimportant:

aggregator	roc_auc
skewness	0.894
max_score	0.860
percentile_score	0.852
top_k_mean	0.848
softmax_weighted_mean	0.797
mean_of_positives	0.796

The grid search is where it pays off:

aggregator	top_k	min_score_to_consider	roc_auc	f1
skewness	5	0.0	0.977	0.935
skewness	3	0.0	0.970	0.933
skewness	10	0.0	0.953	0.915
top_k_mean	3	0.0	0.904	0.885

Two conclusions. Sentinel's default aggregator is the right one — the top three configurations are all skewness. But the default noise floor of min_score_to_consider=0.1 is costing accuracy on this dataset; dropping it to 0.0 takes skewness from 0.894 to 0.977. That is exactly the kind of finding the harness exists to surface, and it is not visible without sweeping.

Also included

• README.md: new "Simulation-based tuning" section with a worked snippet.
• examples/sentinel_against_hate.ipynb: the new tuning section (comparison table, per-family bar charts, grid search, ROC-AUC heatmap) plus a "What we found" write-up.
• Reproducibility fixes to the notebook. Two sampling steps were unseeded and the transcript list came back in filesystem order, so every run evaluated a different set of episodes. Both samples are now seeded and the glob is sorted. The 100 evaluation episodes are drawn with a seeded random.sample rather than an alphabetical slice — the corpus is grouped by show, so [:100] would have covered only the shows sorting first and skipped special-reports entirely.
• Fixed the neutral-data download cell, which failed with poetry: command not found: a %%bash cell starts a non-login shell that never picks up ~/.local/bin. It now reuses sys.executable, which is already the Poetry venv.
• Docs regenerated via docs/generate_docs.py (modules.rst, sentinel.rst).
• Drive-by cleanups: fixed examples/Example_Threshold_Script.py (cache_model kwarg and index path), and renamed tests/test_sriracha_local_index.py to tests/test_sentinel_local_index.py to match the project name.

Notes for reviewers

• The notebook diff is large because it was re-executed; no pre-existing cell's code changed apart from the sampling and download fixes noted above. The rest is regenerated output from one consistent run.
• The "What we found" cell is deliberately qualitative. With 30 episodes per class, differences of a few thousandths of ROC-AUC are inside the margin of error, and separate runs did reorder the leading aggregators, so hard-coded figures in prose would go stale on the next run.
• Sanity check: compare_aggregators and run_grid_search agree exactly where they overlap (skewness at top_k=5, min_score=0.1 is 0.894444 in both), confirming the cheap post-hoc thresholding matches the full re-scoring path.
• tqdm progress-bar widgets and absolute local paths were stripped from the saved outputs.
• Remaining caveat, pre-existing: the evaluation data is fetched at runtime from an unpinned third-party repo and Hugging Face, so the seeds make runs reproducible on one machine but not across arbitrary checkouts.

Test plan

• pytest tests/test_simulation.py — 14 new tests covering all three metric families and score_groups plumbing
• pytest tests/ — full suite passes (54 tests)
• python docs/generate_docs.py leaves docs/source/ clean, so the CI docs-sync check passes
• Notebook re-executed end-to-end from a clean kernel; all 21 code cells ran in order, nbformat.validate passes, no cell contains an error output

[wxiao0421]

Stack map

This PR is the base of a four-PR stack implementing the improvement plan. Each PR targets the one below it, so each diff shows only its own changes.

main
 └── #31  Simulation/tuning harness  (this PR)
      └── #32  Persist the corpus + deterministic loading      [bug fixes]
           ├── #33  index.subsample()                          [new method]
           │    └── #34  Grid-search index size / ratio axes   [new args]
           └── #35  SentinelLocalIndex.from_texts()            [new classmethod]

Suggested review and merge order: #31 → #32 → #33 → #34, with #35 mergeable any time after #32.

#32 goes first among the fixes because there is no point tuning parameters while loading is non-deterministic — you cannot tell a real improvement from random variation in which negatives happened to survive. It is also the easiest to approve, since all three of its parts are defensibly bug fixes rather than features.

#35 deliberately branches off #32 rather than sitting on top of the stack: its only real dependency is the _take_rows helper, so it does not need to wait on #33 and #34.

PR	What it does	Risk
#32	corpus.json, corpus/embedding alignment, seeded loading	Low — additive and backward compatible
#33	Resize an index without re-encoding	Low — new method, nothing existing changes
#34	Sweep index size and ratio in run_grid_search	Low — new args default to None
#35	Build an index in one line	Low — new classmethod

Verified the stack combines cleanly: merging #35 into the tip of #34 produces no conflicts, and the combined tree passes all 99 tests with a clean flake8.

CI note: .github/workflows/test.yml only triggers on pull requests targeting main, so #32–#35 show no checks until they are retargeted. Each was run locally (pytest, flake8, docs/check_docs_sync.py) and each PR description records the results.

[vcai4071]

Not a new change from this PR, but an observation: If there are repeated/duplicate messages, it seems calculate_rare_class_affinity dedupes when it builds a dict of text --> score. We could do something like

scores_by_text = result.observation_scores
observation_scores = np.asarray(
    [scores_by_text[text] for text in observations], dtype=float
)

to restore the full-length array.

[wxiao0421]

This correction makes sense. Will fix with a follow up PR.

[vcai4071]

Edge case: with _average_ranks, ties are averaged, so it may be possible for precision_at_n > 1. For example, five positives tied at the top, the rank for all 5 will be 3. With top_n_eff=3, tp_in_top would be 5?

[wxiao0421]

Will address in follow up PR as well.

[vcai4071]

LGTM! A couple of minor comments.