Repository navigation
Research backlog round 1: schema 0.3.0, --strip-comments, yarn.lock, --split-large-files, coa/rc third condition - #13
Merged
Conversation
… control Three report-self-containment items from dataset/findings/README.md, all offline-verifiable, none changing any default score: 1. Pipeline timings. The findings' "Latency" section could only cite hand-timed observations because reports recorded no duration, so the chunk-parallelisation change was never measurable from the corpus. run_diff_pipeline now records per-stage wall-clock seconds (fetch, diff, llm, feeds, the concurrent analysis stage, total) in a new RiskReport.timings field, rendered in the provenance footer. 2. Evidence caveats. Reports carried a single diff_truncated boolean, while the ua-parser-js and requests write-ups both note that minified bundles are cut head-first — exactly where a payload tends to sit. DiffSummary now names truncated_files (chunker) and skipped_files (engine, over CHAINWATCH_MAX_DIFF_FILE_BYTES), the truncation notice tells the LLM how many chars it is missing, and the aggregator turns these plus multi-chunk max-aggregation into RiskReport.caveats — built from diff_summary only, never from LLM text — with a CAVEATS section in the Rich output. 3. --strip-comments (diff and scan). Recommendation #8 measured 20–28.5 points of a score coming from prose inside a diff, via a hand-built placeholder-vs-empty-stub comparison. The new chainwatch.diff.preprocess module makes the "same code, no prose" condition a one-flag rerun: whole-line comments and block comments that close within the same hunk are removed for JS/TS, Python (including docstrings at line start), shell, batch, and PowerShell; trailing inline comments, unclosed blocks, and shebangs are kept — never strip code is the governing rule. The report records comments_stripped / comment_lines_stripped and a caveat; the chunker preamble tells the LLM comments were removed. SCHEMA_VERSION 0.2.0 -> 0.3.0. Every addition has a default; all 28 committed dataset reports still validate (checked). dataset/README.md's schema table documents the new fields. Tests: 287 passed (37 new), coverage 90.6%, ruff and mypy clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SrGsKknvZvNgFwvTVnHdUA
Two offline engineering items from the review backlog.
yarn.lock (scan). The parser handles both the classic v1 format
("# yarn lockfile v1", `version "1.2.3"`) and Yarn Berry's YAML-flavoured
format (`__metadata:` block, `version: 1.2.3`). Only registry packages are
returned: workspace, file/link/portal, patch, exec, git and URL specifiers
have no registry predecessor to diff against and are skipped. Berry's
alias form (`alias@npm:real@range`) resolves to the real package name. A
dependency that is itself named `version` cannot be mistaken for an
entry's version field: the field must sit at two-space indent and its
value must start with a digit. The previous "not yet supported" rejection
and its test are gone; README, scan help and the smoke tests are updated,
and an unrecognised Gemfile.lock now stands in as the exit-2 case.
--split-large-files (diff and scan). dataset/findings/README.md notes that
a minified bundle is cut head-first at the chunk budget — exactly where a
payload tends to sit. With this flag an oversized file diff is split into
consecutive labelled parts sent as separate chunks instead; a single giant
line (the normal shape of a minified file) is hard-split so nothing is
dropped. Each part is a paid LLM call, so parts per file are capped by a
new CHAINWATCH_MAX_SPLIT_PARTS_PER_FILE setting (default 16, roughly
500 KB of diff at the default budget); past the cap the remainder is
truncated with the same bookkeeping as before. The report records
large_files_split and split_files (schema 0.3.0, still unreleased on this
branch), the truncation caveat now says which rule made the cut, and a new
caveat notes that no single call saw a split file whole. Default behaviour
is unchanged so existing corpus numbers keep their meaning. On the real
lodash 4.17.20 -> 4.17.21 pair, lodash.min.js goes from head-truncated to
6 chunks with nothing omitted.
Tests: 307 passed (20 new), coverage 91.0%, ruff and mypy clean. All 28
committed dataset reports still validate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SrGsKknvZvNgFwvTVnHdUA
Eight LLM calls (claude-sonnet-4-6, live feeds), each one chunk of at most 8k tokens, recorded under the corpus's excluded-from-counts naming conventions so no precision/recall number changes. coa/rc (dataset/malicious/coa-rc/): the narrated placeholder trees exactly as committed, run with --strip-comments. Both pairs drop from HIGH to MEDIUM (coa 56.0 -> 43.5, rc 60.0 -> 46.5), which attributes 12.5-13.5 of the 20-28.5 point narrated-vs-silent gap measured on 2026-08-11 to the prose itself. The stripped scores also sit 7.5-15.0 points above the silent-stub scores, and the model's reasoning says why: "+14 lines added, content not shown" is read as suspicious opacity and scored on network_calls/obfuscation, where genuinely empty files were not. That is a second leakage path, from the tool's own notice text rather than an attacker's comments, and is now named in FINDINGS.md. No dimension reaches the floor rule's 9.0/0.9 trigger; recommendation #7 is unchanged. lodash 4.17.20 -> 4.17.21 (dataset/benign/lodash/): a baseline rerun reproduces the 2026-08-07 base score exactly (4.0, same non-zero dimensions), and --strip-comments moves it to 3.0 with 47 comment lines removed: a one-point benign-side delta. The rerun carries the corpus's first timings block (18.8 s total, 17.8 s LLM for three chunks). Docs: third-condition section in coa-rc FINDINGS.md, sourcing table rows, recommendation #8 update and latency instrumentation note in findings/README.md, lodash rerun note in benign/FINDINGS.md, pending paragraph in dataset/README.md. All 32 dataset JSON files validate; the report-*.json glob still counts 18. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SrGsKknvZvNgFwvTVnHdUA
…alls Two follow-ups to the previous commit, both filed under the corpus's excluded-from-counts naming so no precision/recall number changes. coa 2.0.2 -> 2.0.3, stripped condition repeated once (inconclusive-report-coa-2.0.2-to-2.0.3-STRIPPED-run2.json): 45.5 / base 40.5 against the first run's 43.5 / 38.5. Two runs 2.0 points apart, with four of six dimensions identical, is the only run-to-run variance point so far; it places both the 12.5-13.5 point prose effect and the 7.5-15.0 point opacity effect well outside that band. The repeat also caught the model citing the tool's own "23 comment lines were removed" preamble note as evidence of substantive hidden content: a third leakage path, from tooling notices rather than attacker prose, now recorded as an open question in coa-rc/FINDINGS.md. lodash 4.17.20 -> 4.17.21 with --split-large-files (experiment-split-large-files-4.17.20-to-4.17.21.json): lodash.min.js seen whole across six chunks instead of head-first. Base 3.5 against the baseline's 4.0, still 0.0 / LOW. The truncated tail hid nothing on this benign pair, at 42.1 s of LLM time against 17.8 s for three chunks. Docs updated in coa-rc FINDINGS.md and SOURCING.md, benign/FINDINGS.md, and findings/README.md (recommendation #8 and the latency section). All 34 dataset JSON files validate; report-*.json still counts 18. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SrGsKknvZvNgFwvTVnHdUA
erxxc
added a commit
that referenced
this pull request
Sep 9, 2026
Re-land research backlog round 1 on main (#13 merged into its stacked base)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First round on the research backlog in
dataset/findings/README.md. Stacked on #12 (theuv.lockchore) so the tracked lockfile stays put; once that merges this PR shows only its own four commits.Schema 0.3.0 — report self-containment
timings: per-stage wall-clock seconds (fetch, diff, llm, feeds, analysis, total). The latency section could only cite hand-timed numbers before.caveats[]built fromdiff_summaryonly, never from LLM text: which files were cut head-first at the token budget, which were never diffed at all (overCHAINWATCH_MAX_DIFF_FILE_BYTES), multi-chunk max-aggregation, stripped comments, split files. Rendered as a CAVEATS section.--strip-comments(diff and scan): removes whole-line and block comments before the LLM sees the diff — the narrative-leakage control from recommendation Performance and detection improvements: bounded concurrency, PyPI Rekor, new-dependency provenance #8 as a one-flag rerun. Never strips code it cannot verify is a comment.All additions have defaults; every committed report still validates.
Offline engineering
scan: classic v1 and Berry; registry packages only; Berry aliases resolve to the real name.--split-large-files(diff and scan): sends an oversized file diff in consecutive parts instead of cutting it head-first, with giant single lines hard-split. Parts per file are capped byCHAINWATCH_MAX_SPLIT_PARTS_PER_FILE(default 16) to bound spend. Off by default so corpus numbers keep their meaning.Live experiments (15 LLM calls,
claude-sonnet-4-6, all one-chunk except lodash)--strip-commentsdrop from HIGH to MEDIUM (coa 56.0 → 43.5, rc 60.0 → 46.5), attributing 12.5–13.5 of the 20–28.5 point narrated-vs-silent gap to prose. The remaining 7.5–15 points come from the model treating hidden content as suspicious in itself — a second leakage path, from the tool's notice text. A repeat of the coa run landed 2.0 points away and caught a third path: the model citing the "23 comment lines were removed" preamble note as evidence. Recommendation Ground-truth corpus: real-attack reconstructions, DoS-dimension fix, two independent stress-test incidents #7 is unchanged (no dimension reaches the floor trigger).--strip-commentsmoves it to 3.0;--split-large-files(minified bundle seen whole, 6 chunks) moves it to 3.5. All still 0.0 / LOW.All new reports use the excluded-from-counts naming conventions (
inconclusive-*,experiment-*); the canonicalreport-*.jsonglob still counts 18. Write-ups indataset/malicious/coa-rc/FINDINGS.md,dataset/benign/FINDINGS.md,dataset/findings/README.md(#8 and Latency),dataset/README.md.Test plan
pytest: 307 passed, coverage 91.0% (gate 85%)ruff check src/ tests/andmypy src/: cleanReview notes
ruff-formatpre-commit hook skipped: 25 upstream files are not format-clean and CI does not check formatting, so letting it run would have reformatted whole files.dataset/README.mdsays report timestamps are normalised to midnight, but the committed reports carry real times; the new files follow the files, not the sentence. Left for a decision.coa-rc/FINDINGS.md.🤖 Generated with Claude Code
https://claude.ai/code/session_01SrGsKknvZvNgFwvTVnHdUA