Measure which parts of a prompt your model actually needs, by deleting them and scoring what breaks.
In 2020 I compiled one of the most-copied "complete English stopword lists" on GitHub: 11,017
characters of words to throw away, posted as
a comment on a stopwords
gist that has drawn 64 forks and 78 comments since 2010, and cited in published research since.
Lists like that delete not, no, without, unless and except first. Plausibly the
highest-leverage tokens in the whole prompt. Stopless is me asking whether that list was ever a
good idea for an LLM, and measuring the answer.
Stopless calls a word that a classical list would delete but that your model measurably needs an anti-stopword. Finding them is an intersection you can compute: elements whose removal demonstrably hurts, and which a named stopword list would have thrown away on sight.
No API key and no network. The demo replays recorded responses, and runs from a fresh clone:
git clone https://github.com/jimmyjames177414/stopless && cd stopless
uv run stopless analyze examples/negation_qa/prompts.jsonl \
--cassette examples/negation_qa/cassette.json \
--words "not,without,no,never,only,please,actually,generally" \
--classes article --repeats 5
The corpus is 120 invented yes/no questions about invented nouns (
examples/negation_qa/build_corpus.pygenerates it). The responses were recorded from Stopless's own bundledMockProvider: a deterministic toy model, not a language model. The effect sizes are large because that toy model is negation-sensitive by construction. This output demonstrates that the measurement pipeline works end to end. It says nothing whatsoever about how any real model behaves.
element effect 95% CI tok Δ n
----------------------------------------------------------------
ANTI-STOPWORDS (on the reference stopword list, but removal hurts)
no -95.0pp [-100.0, -90.0] ~-1 12
not -94.5pp [-98.0, -90.5] ~-1 40
MUST KEEP (removal hurts; not on the reference stopword list)
never -96.7pp [-100.0, -91.7] ~-2 12
without -94.2pp [-98.3, -89.2] ~-2 24
UNDERPOWERED / TASK-DEPENDENT (widen the run before believing anything)
only -5.0pp [-15.0, +0.0] ~-1 4
actually +0.0pp [-3.3, +3.3] ~-3 30
please +0.7pp [-2.7, +4.7] ~-3 30
generally +0.7pp [-2.0, +3.3] ~-4 30
LOW INFORMATION (small effect, bounded)
the +0.3pp [-2.0, +3.0] ~-1 60
a +0.5pp [-1.2, +2.2] ~-4 120
not present in this corpus: an
model mock-negation-qa:v1 · 120 items · 5 repeats · 2,410 model calls · median observed MDE 5.7pp
control accuracy 97.5% · scorer exact_match:v1 · ablation delete · scope affected · stochastic
95% percentile bootstrap over items (10,000 resamples) · p-values sign-flip permutation, Holm across 10 element(s)
anti-stopword band = must-keep ∩ en_nltk (179 words) · prompt tokens 83,645 (estimated)
replayed from cassette.json (recorded by examples/negation_qa/build_corpus.py (stopless 0.1.0); responses came from a synthetic/mock provider)
The call count is a cold cache. Run it a second time and the last block reads 0 model calls, because responses are cached under ~/.cache/stopless/; the bands, effects and
intervals are unchanged.
Four things in that table are worth pointing at.
not and no are anti-stopwords; without and never are not. The measurement is
identical: all four are catastrophic to remove. The difference is that not and no are on
NLTK's English stopword list and without and never are not, so only the first pair can
reach the anti-stopword band at all. The band is an intersection with a named list, and
the report names it. A larger list (scikit-learn's 318-word set, stopwords-iso) would move
without across. See data/reference_stopwords/README.md.
only is not reported as safe, even though its true effect here is zero. It appears in
four items. Four items cannot distinguish "no effect" from "we could not see one", so it
lands in task-dependent and the run tells you to widen it.
Nothing is reported as safe-to-remove. safe-to-remove means the whole confidence
interval sits inside ±1pp, and 120 items at this noise level cannot certify that. The
filler words land one band down in low-information. That is the tool working: it will not
tell you something is free until it can show you it is free.
Every number carries an interval, an N, and a token delta. A deletion effect without those is not a finding, it is a vibe.
The same machinery, pointed at a CLAUDE.md / AGENTS.md / system prompt. Each line,
bullet, heading and section is deleted in turn and scored against a real task suite:
uv run stopless instructions examples/instruction_ablation/AGENTS.example.md \
--suite examples/instruction_ablation/tasks.jsonl \
--cassette examples/instruction_ablation/cassette.json \
--repeats 5Synthetic, again: an invented instruction file, a 40-item invented suite, and the same deterministic toy model. It reads three of its own instructions and ignores the rest, which is exactly what makes the ground truth checkable.
MUST KEEP (removal hurts; not on the reference stopword list)
line[3] '- Reply with a single word and nothing…' -97.0pp [-99.0, -94.5] ~-13 40
line[4] '- Write the answer in lowercase.' -97.0pp [-99.0, -94.5] ~-11 40
section[0] '# Agent instructions: closed-question a…' -97.0pp [-99.0, -94.5] ~-177 40
section[1] '## Output format - Reply with a single…' -97.0pp [-99.0, -94.5] ~-47 40
line[8] '- If the question contains a negation,…' -47.5pp [-63.0, -31.5] ~-20 40
section[2] '## Reasoning - Read the context before…' -47.0pp [-62.5, -31.5] ~-55 40
UNDERPOWERED / TASK-DEPENDENT (widen the run before believing anything)
line[0] '# Agent instructions: closed-question a…' -0.5pp [-4.5, +3.5] ~-15 40
line[6] '## Reasoning' -0.5pp [-4.0, +3.0] ~-5 40
line[12] '- Never apologise for giving a short an…' -0.5pp [-4.0, +3.0] ~-15 40
line[5] '- Do not add punctuation, and do not re…' +1.0pp [-2.0, +4.0] ~-17 40
line[9] '- Prefer the supplied context over your…' +1.5pp [-1.5, +4.5] ~-17 40
line[11] '- Be concise and professional.' +1.5pp [-1.0, +4.0] ~-9 40
line[1] 'You are answering closed questions abou…' +2.0pp [-0.5, +4.5] ~-19 40
line[2] '## Output format' +2.0pp [-0.5, +5.0] ~-6 40
line[10] '## Tone' +2.0pp [+0.0, +4.5] ~-3 40
line[13] '- Thank the user when they supply extra…' +2.0pp [-0.5, +5.0] ~-14 40
line[7] '- Read the context before reading the q…' +2.5pp [+0.0, +5.0] ~-13 40
section[3] '## Tone - Be concise and professional.…' +2.5pp [+0.5, +4.5] ~-41 40
model mock-negation-qa:v1 · 40 items · 5 repeats · 3,800 model calls · median observed MDE 3.9pp
control accuracy 97.0% · scorer exact_match:v1 · ablation delete · scope affected · stochastic
section[0] is the H1, so its section is the whole file: deleting the entire instruction
file costs 97 points, which bounds what the file is worth in total. Two formatting bullets
account for all of it. The ## Tone section costs nothing.
uv run stopless power --items 120 --repeats 5 --baseline 0.975 POWER: what a run this size could detect
items 120
repeats per arm 5
assumed accuracy 97.5%
assumed pairing rho 0.5
alpha / power 0.05 / 0.8
minimum detectable effect 1.79pp
An effect smaller than that will not be distinguishable from zero at this N.
Scaling: the MDE falls with the square root of items, so detecting half the
effect costs four times the run.
items needed for a given effect (repeats=5, accuracy=97.5%):
detect 1.0pp → 383 items
detect 2.0pp → 96 items
detect 5.0pp → 16 items
detect 10.0pp → 4 items
Telling you up front that 20 items cannot detect a 2% effect is more useful than any single measurement.
Nothing here is published to a package index. Run it straight from the repository:
uvx --from git+https://github.com/jimmyjames177414/stopless stopless --helpPython 3.10+, and one runtime dependency (PyYAML) so that a cold start stays fast.
Everywhere below, stopless … means either of two things that work: the
uvx --from git+https://github.com/jimmyjames177414/stopless stopless … form above, or
uv run stopless … inside a clone. The demo commands are written the second way because they
read files from examples/, which ships in the source tree rather than the wheel.
Any OpenAI-compatible /v1/chat/completions endpoint works: Ollama, vLLM, LM Studio,
llama.cpp's server, OpenRouter, Together, Groq, DeepSeek, OpenAI. Provider selection is a URL,
never a code change. A contributor with Ollama and no credit card can run everything in this
repo.
ollama pull qwen3:8b
stopless analyze your_prompts.jsonl \
--model qwen3:8b --base-url http://localhost:11434/v1 \
--classes negation,hedging,filler,article \
--scorer exact_match --repeats 5 --out results/Your corpus is JSONL, one object per line:
{"id": "qa.0001", "prompt": "…", "expected": "no"}Always dry-run first. It prints the exact number of model calls and an estimated token count before anything is sent:
stopless analyze your_prompts.jsonl --model qwen3:8b --classes negation --dry-run--max-calls N is a hard stop. Responses are cached under ~/.cache/stopless/, so a
re-run with a different scorer costs nothing.
A CXS v0.1 results directory (see schemas/):
results/<experiment_id>/
manifest.json # what was run, on what, with which seed and determinism claim
interventions.json # every arm, including the mandatory noop control
trials.jsonl # one raw execution per line, raw, never aggregated
outcomes.jsonl # scores, separate from trials so a run can be re-scored for free
report.json # aggregates and statistics
report.md # the human-readable version
signal-map.json # ← the durable artifact
A results directory is written once, when a run completes. There is no resume: a run stopped
by --max-calls or a crash produces no results directory, though its responses are in the
cache so re-running does not pay for them twice.
signal-map.json is the point. It is a committable, diffable statement of for this model,
on this task, these elements are safe to delete and these will bite you. It carries its own
caveats, so it cannot be quoted without them. Diff it across model releases.
stopless report results/<id>/ re-renders any finished run without calling anything.
- Select. An element is a word (
lexical:not), a lexical class expanded into its members (--classes negation), or a structural unit (structural:line[3]). - Ablate. Delete it from every prompt, then repair the damage: whitespace, orphaned
punctuation, sentence-initial capitals. Unrepaired text measures "can the model cope
with malformed input?", which is a different question.
--ablation-mode maskswaps in a neutral placeholder instead, to partly control for length. - Run the grid. items × arms × repeats, always including a
noopcontrol, cached and budgeted. - Score.
exact_match,contains,numeric_tolerance,json_field,regex, or your ownscript. - Measure. Paired difference per item, 95% percentile bootstrap resampling items, sign-flip permutation p-values, Holm correction, and an MDE so a null result is interpretable.
- Classify. From the interval, never the point estimate.
| Band | Rule |
|---|---|
safe-to-remove |
the whole interval sits inside ±1pp |
low-information |
the whole interval sits inside ±3pp, but is wider than ±1pp |
task-dependent |
the interval spans both "matters" and "does not matter", so the run cannot say |
must-keep |
the whole interval sits below −3pp: removal measurably hurts |
anti-stopword |
must-keep and the element is on the reference stopword list |
removal-helps |
the whole interval sits above +3pp: the model did better without it |
A band is never assigned from a point estimate. The type system enforces it: the
classifier accepts an object that cannot be constructed without both interval bounds. So does a
test that hands it a −18pp midpoint inside a [-40, +4] interval and asserts the answer is
task-dependent, not must-keep.
A zero-width bootstrap interval gets special treatment. Four items that all scored
identically produce [0.0, 0.0], which read literally would certify safe-to-remove on four
observations. No observed variation is not the same as no uncertainty, so the band is decided
from the rule-of-three bound (±300/n pp) instead.
Stopless is not the first tool to ablate parts of a prompt, and it does not claim to be.
| Prior work | How it scores importance | Its goal |
|---|---|---|
| ContextCite (MIT, NeurIPS 2024) | real ablation plus a linear surrogate over random ablations | attribute one generated statement to context sources, per query |
| LLMLingua / LLMLingua-2 (Microsoft) | proxy: small-LM perplexity, information entropy, a GPT-4-distilled token classifier | produce a cheaper working prompt |
| Selective Context | proxy: self-information | compress |
| TokenSHAP, AttriBoT | token-level attribution against model output or generation probability | explain a generation |
| LLM-Microscope | removes punctuation, stopwords and articles; reports the effect on MMLU and BABILong | a research finding |
| NLTK / spaCy / scikit-learn / stopwords-iso | none: fixed, hand-made, model-agnostic | IR term filtering |
| Stopless | real ablation, scored on task outcome, aggregated over a corpus | a reusable per-model, per-task element vocabulary, and anti-stopword discovery |
ContextCite is the closest prior art. The honest distinction is one of question, not technique: ContextCite answers "which source produced this sentence, for this one query?" Stopless answers "across 200 prompts, which recurring elements can I delete from prompts of this kind, for this model, and which will bite me?" Per-query attribution versus a corpus-level reusable vocabulary.
The suspicion that stopword removal is not free predates LLMs entirely: classical NLP was already comparing it against downstream task accuracy, e.g. Comparative analysis of effect of stopwords removal on sentiment classification (IEEE). Stopless is that question asked of a model rather than a classifier, and answered per model and per task rather than once.
Negation fragility in LLMs is a long-established finding and is not ours. Removing stopwords and measuring the effect has been published. What we could not find published is the combination: word- and phrase-level ablation scored against a real task metric, aggregated into a durable artifact, and applied to agent instruction files. Full sourcing, including the claims we are forbidden from making, is in NOVELTY.md.
Statistical methodology follows Miller, Adding Error Bars to Evals (arXiv:2411.00640). Cited, not claimed.
- Deletion changes length, and length itself changes behaviour.
--ablation-mode maskholds word count roughly constant, but not token count and certainly not attention mass. The report prints the token delta beside every effect so the confound is visible. It is reported, not solved. - Elements interact and v0.1 does not measure that. Removing
notandwithouttogether is not the sum of their separate effects. Every number here is a single-element effect. - Results do not transfer. They are specific to one model, one task, one corpus, and one
scorer. The tool refuses to imply otherwise, and
signal-map.jsonrecords all four. - The demo corpus is synthetic and the demo model is a toy. Findings about real models require real runs, which cost real money.
- Scoring is exact-match style. Open-ended generation needs an LLM judge, which v0.1 deliberately does not ship. A judge is itself sensitive to the words being ablated, so a judged ablation measures the judge and the subject at the same time.
anti-stopworddepends on which reference list you compare against, and different published lists disagree aboutwithout,exceptandunless.- The bundled NLTK stopword list was transcribed by hand, not downloaded.
data/reference_stopwords/README.mdgives you a one-line command to diff it against your own copy, and asks you to open an issue if it differs. - Structural selectors are regexes, not a parser. Sentence splitting knows about a dozen abbreviations and nothing more.
stopless power's a priori MDE rests on an assumed within-item correlation (rho, default 0.5). Raising it shrinks the MDE, so it is the assumption to be sceptical of. The observed MDE reported after a run does not depend on it.
Wanted, and tracked as help-wanted issues:
- an LLM-judge scorer, with the confound controls it needs to mean anything
- automatic element discovery (n-gram mining): v0.1 only ablates elements you declare
- interaction effects between elements
- cross-model consistency reports: is
notsafe on model A but not on model B? - local HuggingFace model support without an HTTP server
Per SHARED-FOUNDATION D10, none of these ships as a stub. A stub that looks implemented is worse than an admitted gap.
The easiest useful contribution is a data file, not code:
data/lexical_classes/*.yaml are plain YAML word lists, deliberately
outside src/, so adding a class or a language never means reading Python. See
CONTRIBUTING.md.
git clone https://github.com/jimmyjames177414/stopless && cd stopless
uv venv && uv pip install -e ".[dev]"
uv run pytest -m "not live" # 348 tests, no API key, no network
uv run ruff check . && uv run ruff format --check .
uv run mypy --strict src/Stopless writes CXS v0.1, a file format, not a library. It imports no
sibling project and no sibling project imports it. tests/test_cxs_conformance.py validates
real output from a real run against the vendored schemas, because a schema nobody validates
against is a wish.
- NOVELTY.md: what is and is not new here, sourced, including the claims this project may not make.
docs/ablation.md: how deletion and masking repair the text afterwards.docs/selectors.md: lexical, class and structural selectors.docs/classification.md: the band rules in full.docs/statistics.md: bootstrap, permutation, Holm, MDE.docs/instruction-files.md: ablating aCLAUDE.md/AGENTS.md.
@jimmyjames177414 · Apache-2.0
One of nine open-source tools for measuring what context and tools
actually do to AI systems:
stopless · stopbench · mincontext · validwhile · errorbars
assumptionledger · toolsweep · knowwhen · inconclusive