feat: audit what title_filter.positive is actually doing - #2731
Closed
WJiangH wants to merge 1 commit into
Closed
Conversation
The keyword layer decides the fate of most of what gets scanned — measured on one run, 8,289 found and 6,499 rejected on title alone — and reports nothing about how. There is no way to tell which keywords earn their place, which are carried by others, or which are quietly filling the pipeline with roles that score 1.0. Tuning it has been guesswork. scan.mjs already knows: matchedTitleKeywords() computes exactly which keywords a title hit, then discards the answer roughly eight thousand times per scan because only content_filter scoping needed it. This replays that function over the offers in scan-history.tsv and joins their evaluated scores. The metric is `unique`, not `hits`: how many offers a keyword is the only one to catch. A keyword with 200 hits and 0 unique can be deleted without losing a posting; one with 3 hits and 3 unique is the sole reason those three were ever seen. Raw hit counts hide both cases. On a real 49-keyword filter over 539 offers it found: - 8 keywords that have never matched anything — including RLHF and Model Serving, lifted straight from the CV. They describe work, not job titles. Copying CV vocabulary into a title filter does not work, and this is the first evidence of why. - 3 redundant, every match also caught by a neighbour. - 8 noisy: EDA admits 12 offers averaging 1.0/5, Agentic 22 averaging 1.4 — while also being one of the larger unique contributors, so the trade is real rather than a free deletion. Read-only, zero-LLM, and it never proposes an edit — it reports, the user decides. 10 self-test cases including the >=5-hits / <2.5-average boundary. Does not yet answer "what am I missing": that needs the rejected titles, and the scanner records only a count of them. --help says so explicitly rather than letting the omission look like an empty result. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
📝 WalkthroughWalkthroughAdds ChangesTitle Filter Audit
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related issues
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment Warning |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@filter-audit.mjs`:
- Around line 71-89: Deduplicate `titleFilter.positive` before building `stats`
and replaying `matchedTitleKeywords()` so each configured keyword is represented
by one matcher and one statistics record. Preserve the existing hit and
unique-count logic for distinct keywords, and add a self-test covering duplicate
positive keywords to verify a single title counts as one unique match.
- Around line 60-66: Update readScoresByUrl to handle duplicate normalized URLs
deterministically instead of relying on Map.set() overwriting based on
reportTexts order. Retain sufficient report identity to select a documented
canonical report, or explicitly reject duplicates, and ensure avgScore and noisy
consume only that deterministic selection.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: e98650dc-1572-4c43-98f0-3cdfd885e441
📒 Files selected for processing (3)
AGENTS.mdfilter-audit.mjsupdate-system.mjs
Comment on lines
+60
to
+66
| export function readScoresByUrl(reportTexts) { | ||
| const map = new Map(); | ||
| for (const text of reportTexts) { | ||
| const u = text.match(/\*\*URL:\*\*\s*<?([^\s>\n)]+)/); | ||
| const s = text.match(/\*\*Score:\*\*\s*([0-9](?:\.[0-9])?)/); | ||
| if (u && s) map.set(u[1].replace(/[)\s]+$/, ''), Number(s[1])); | ||
| } |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# List report URLs that occur in more than one report file.
rg -n --glob '*.md' '^\*\*URL:\*\*' reports \
| sed -E 's#^([^:]+):[0-9]+:\*\*URL:\*\*[[:space:]]*<?([^ >)]+).*#\2\t\1#' \
| sort \
| awk -F '\t' '
{
files[$1] = files[$1] ? files[$1] ", " $2 : $2
count[$1]++
}
END {
for (url in count) {
if (count[url] > 1) print url "\t" files[url]
}
}
' \
| sortRepository: santifer/career-ops
Length of output: 157
🏁 Script executed:
set -euo pipefail
printf '%s\n' '--- candidate files ---'
git ls-files | rg '(^|/)(filter-audit\.mjs|scan\.mjs|.*report.*|.*tracker.*)$' | head -200
printf '%s\n' '--- filter-audit.mjs outline/content ---'
if [ -f filter-audit.mjs ]; then
wc -l filter-audit.mjs
ast-grep outline filter-audit.mjs || true
cat -n filter-audit.mjs | sed -n '1,260p'
else
found="$(fd -t f -i '^filter-audit\.mjs$' . | head -1)"
printf 'found=%s\n' "$found"
if [ -n "$found" ]; then
wc -l "$found"
ast-grep outline "$found" || true
cat -n "$found" | sed -n '1,260p'
fi
fi
printf '%s\n' '--- readScoresByUrl references ---'
rg -n -C 4 'readScoresByUrl|readdir|reports|reportTexts' --glob '*.mjs' --glob '*.md' . | head -300
printf '%s\n' '--- report directory status ---'
if [ -d reports ]; then
find reports -maxdepth 2 -type f -print | sort | head -200
printf '%s\n' '--- URL records ---'
rg -n --glob '*.md' '^\*\*URL:\*\*' reports || true
else
printf '%s\n' 'reports directory is absent'
fiRepository: santifer/career-ops
Length of output: 34948
🏁 Script executed:
set -euo pipefail
printf '%s\n' '--- report-producing code ---'
rg -n -C 8 'URL:\*\*|URL:|report.*(write|path|file)|reports/|reserveReportNumbers|readdirSync\(.*REPORT' \
--glob '*.mjs' --glob '*.md' --glob '*.ts' --glob '*.tsx' . | head -500
printf '%s\n' '--- report fixtures and format ---'
for f in examples/sample-report.md \
web/src/lib/report/FORMAT.md \
test-fixtures/upgrade/state-v1.16/reports/001-acme-2026-06-20.md \
test-fixtures/upgrade/state-v1.16/reports/002-globex-2026-06-25.md \
test-fixtures/upgrade/state-v1.18/reports/001-acme-2026-06-20.md \
test-fixtures/upgrade/state-v1.18/reports/002-globex-2026-06-25.md; do
if [ -f "$f" ]; then
printf '\n--- %s ---\n' "$f"
rg -n -C 3 'URL|Score|date|report|re-?evaluat|duplicate|unique|number|filename' "$f" | head -160 || true
fi
done
printf '%s\n' '--- documentation references to repeated evaluation/report identity ---'
rg -n -i -C 3 're-?evaluat|same url|duplicate url|duplicate report|report number|report identity|canonical report|latest report|rerun' \
--glob '*.md' --glob '*.mjs' --glob '*.yml' . | head -400 || trueRepository: santifer/career-ops
Length of output: 50377
🏁 Script executed:
node - <<'JS'
'use strict';
function readScoresByUrl(reportTexts) {
const map = new Map();
for (const text of reportTexts) {
const u = text.match(/\*\*URL:\*\*\s*<?([^\s>\n)]+)/);
const s = text.match(/\*\*Score:\*\*\s*([0-9](?:\.[0-9])?)/);
if (u && s) map.set(u[1].replace(/[)\s]+$/, ''), Number(s[1]));
}
return map;
}
function audit(history, scoresByUrl) {
let scored = 0;
let scoreSum = 0;
for (const row of history) {
const score = scoresByUrl.get(row.url);
if (typeof score === 'number') {
scored += 1;
scoreSum += score;
}
}
return {
scored,
avgScore: scored ? Number((scoreSum / scored).toFixed(2)) : null,
noisy: history.length >= 5 && scored > 0 &&
Number((scoreSum / scored).toFixed(2)) < 2.5,
};
}
const oldReport = '**URL:** <https://example.test/job>\n**Score:** 5.0';
const newReport = '**URL:** <https://example.test/job>\n**Score:** 1.0';
const history = Array.from({ length: 5 }, (_, i) => ({
url: 'https://example.test/job',
title: `Role ${i}`,
}));
for (const reports of [[oldReport, newReport], [newReport, oldReport]]) {
const scores = readScoresByUrl(reports);
console.log(JSON.stringify({
inputOrder: reports[0] === oldReport ? 'old,new' : 'new,old',
selectedScore: scores.get(history[0].url),
audit: audit(history, scores),
}));
}
JSRepository: santifer/career-ops
Length of output: 338
Define a deterministic policy for duplicate report URLs.
If duplicate URLs are valid inputs, Map.set() keeps the last score, so avgScore and noisy depend on directory order. Retain report identity and select a canonical report, or reject duplicates.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@filter-audit.mjs` around lines 60 - 66, Update readScoresByUrl to handle
duplicate normalized URLs deterministically instead of relying on Map.set()
overwriting based on reportTexts order. Retain sufficient report identity to
select a documented canonical report, or explicitly reject duplicates, and
ensure avgScore and noisy consume only that deterministic selection.
Comment on lines
+71
to
+89
| const positives = Array.isArray(titleFilter?.positive) ? titleFilter.positive : []; | ||
| const stats = new Map(positives.map((k) => [k, { keyword: k, hits: 0, unique: 0, scored: 0, scoreSum: 0, examples: [] }])); | ||
| let unmatched = 0; | ||
|
|
||
| for (const row of history) { | ||
| const hit = matchedTitleKeywords(row.title, titleFilter); | ||
| if (hit.length === 0) { | ||
| // Present in history but matched nothing: the filter changed since this | ||
| // row was recorded. Worth surfacing rather than silently dropping. | ||
| unmatched++; | ||
| continue; | ||
| } | ||
| const score = scoresByUrl.get(row.url); | ||
| for (const k of hit) { | ||
| const s = stats.get(k); | ||
| if (!s) continue; | ||
| s.hits++; | ||
| if (hit.length === 1) s.unique++; | ||
| if (typeof score === 'number') { s.scored++; s.scoreSum += score; } |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Deduplicate configured keywords before calculating unique.
stats uses the keyword string as its Map key, but matchedTitleKeywords() returns one hit per compiled matcher. If title_filter.positive contains the same keyword twice, one title produces two hits for one statistics record. The audit then sets unique to zero and can classify the keyword as redundant incorrectly.
Normalize duplicate positive values before replaying the matcher. Add a self-test for duplicate keywords.
Proposed fix
export function audit(titleFilter, history, scoresByUrl) {
- const positives = Array.isArray(titleFilter?.positive) ? titleFilter.positive : [];
+ const positives = [...new Set(
+ Array.isArray(titleFilter?.positive) ? titleFilter.positive : []
+ )];
+ const normalizedTitleFilter = { ...titleFilter, positive: positives };
const stats = new Map(positives.map((k) => [k, { keyword: k, hits: 0, unique: 0, scored: 0, scoreSum: 0, examples: [] }]));
@@
- const hit = matchedTitleKeywords(row.title, titleFilter);
+ const hit = matchedTitleKeywords(row.title, normalizedTitleFilter);📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| const positives = Array.isArray(titleFilter?.positive) ? titleFilter.positive : []; | |
| const stats = new Map(positives.map((k) => [k, { keyword: k, hits: 0, unique: 0, scored: 0, scoreSum: 0, examples: [] }])); | |
| let unmatched = 0; | |
| for (const row of history) { | |
| const hit = matchedTitleKeywords(row.title, titleFilter); | |
| if (hit.length === 0) { | |
| // Present in history but matched nothing: the filter changed since this | |
| // row was recorded. Worth surfacing rather than silently dropping. | |
| unmatched++; | |
| continue; | |
| } | |
| const score = scoresByUrl.get(row.url); | |
| for (const k of hit) { | |
| const s = stats.get(k); | |
| if (!s) continue; | |
| s.hits++; | |
| if (hit.length === 1) s.unique++; | |
| if (typeof score === 'number') { s.scored++; s.scoreSum += score; } | |
| const positives = [...new Set( | |
| Array.isArray(titleFilter?.positive) ? titleFilter.positive : [] | |
| )]; | |
| const normalizedTitleFilter = { ...titleFilter, positive: positives }; | |
| const stats = new Map(positives.map((k) => [k, { keyword: k, hits: 0, unique: 0, scored: 0, scoreSum: 0, examples: [] }])); | |
| let unmatched = 0; | |
| for (const row of history) { | |
| const hit = matchedTitleKeywords(row.title, normalizedTitleFilter); | |
| if (hit.length === 0) { | |
| // Present in history but matched nothing: the filter changed since this | |
| // row was recorded. Worth surfacing rather than silently dropping. | |
| unmatched++; | |
| continue; | |
| } | |
| const score = scoresByUrl.get(row.url); | |
| for (const k of hit) { | |
| const s = stats.get(k); | |
| if (!s) continue; | |
| s.hits++; | |
| if (hit.length === 1) s.unique++; | |
| if (typeof score === 'number') { s.scored++; s.scoreSum += score; } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@filter-audit.mjs` around lines 71 - 89, Deduplicate `titleFilter.positive`
before building `stats` and replaying `matchedTitleKeywords()` so each
configured keyword is represented by one matcher and one statistics record.
Preserve the existing hit and unique-count logic for distinct keywords, and add
a self-test covering duplicate positive keywords to verify a single title counts
as one unique match.
This was referenced Aug 13, 2026
Contributor
Author
|
Closing this one myself, per the queue conversation on #2692 — not withdrawing it. This is one of four (#2730, #2731, #2732, #2733) I'm closing together: they're a single group — scan-filter observability — none of them blocks anything, and holding four spots in your queue for a related group is exactly the cost you described. That takes your side from seven to three. Nothing is lost by it:
I'll reopen when the rest of the queue is down, or sooner if you want this one specifically — just say so here. There's also a decent chance this group comes back as one PR rather than four, since the four pieces are closer to each other than they are to anything else. |
This was referenced Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2694.
The gap
The keyword layer decides the fate of most of what gets scanned — measured on
one run, 8,289 found and 6,499 rejected on title alone — and reports nothing
about how. No way to tell which keywords earn their place, which are carried
entirely by neighbours, or which quietly fill the pipeline with 1.0/5 roles.
Every tuning guess costs real evaluation tokens to disprove.
scan.mjsalready computes the answer and throws it away:matchedTitleKeywords()knows exactly which keywords a title hit, then discards it ~8,000 times per scan
because only
content_filterscoping needed it.What it does
Replays that same function over
data/scan-history.tsvand joins the scoresfrom
reports/. Read-only, zero-LLM, never proposes an edit — it reports,the user decides.
The metric is
unique, nothits: how many offers a keyword is the onlyone to catch. A keyword with 200 hits and 0 unique can be deleted without losing
a posting; one with 3 hits and 3 unique is the sole reason those three were seen.
Raw hit counts hide both cases.
What it found on a real 49-keyword filter
RLHFandModel Serving, lifted straight from the CV. They describe work, not jobtitles. Copying CV vocabulary into a title filter does not work, and this is
the first evidence of why.
EDAadmits 12 offers averaging 1.0/5;Agentic22 averaging1.4 while also being one of the larger unique contributors, so the trade is
real rather than a free deletion. A count alone cannot tell you that.
Historical rows matching no current keyword are counted and surfaced rather than
silently dropped — that number means the filter changed since they were recorded.
Testing
Rebased onto today's
main;node test-all.mjs→ 3432 passed, 0 failed.--self-test→ 10 passed. Live--summaryrun included above.One fixture note: an earlier cut of the self-test had my expected values wrong
(1.5 vs the actual 1.625 average, and a hit count under the ≥5 threshold), not
the code. The fixtures now include a row landing exactly on the noisy-rule
boundary so the threshold is pinned rather than approximated.
Summary by CodeRabbit
New Features
Chores