fix(evals-core): redact credentials in published results - #246
Open
tanya732 wants to merge 1 commit into
Open
Conversation
|
Warning Review limit reachedNext included review available in 55 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (6)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
`serializers.ts` is the single choke point for everything a run publishes —
the JSON results file, the HTML report, the trace rendered in a dashboard —
and none of it was redacted. `redactSecrets` already existed but was wired
only into `formatCommandTrace` for LLM-bound traces.
Wire the publish path through the same scrubber at six sites, and close
three matcher gaps found by measuring against shipped reports.
Redaction runs at serialisation, after graders and the scorer, so it cannot
change a verdict or a score; graders still read the real workspace. It also
keeps the report template free of redaction logic.
Client ids, user ids, `AUTH0_DOMAIN`, audiences and `session_id` are
deliberately left readable: they are identifiers and public configuration,
not credentials, and they are what makes a trace reviewable.
Matcher fixes:
- Numeric guard on `VALUE`. `No token = 401. Valid token but wrong scope =
403.` published as `No token = [REDACTED SECRET] Valid…`, losing the status
code and the sentence break while telling a reader a secret was exposed
where none was. A credential is never a bare number.
- `redactArgs` consults the key. In `{ Authorization: 'Bearer …' }` the
credential's name is the object key, invisible to the text rules, and the
value is often too short for the opaque-token floor.
- A rule for a credential word embedded inside a token with no key beside
it, as in a grader detail that echoes the needle it searched for. Gated on
a separator-flanked word, a 6+ character suffix and a digit, so
`reset_password_2fa` and `change_password_v2` keep their values.
New patterns for value-shaped secrets carrying no name: PEM private key
blocks (ordered before the shape rules so the base64 body is not chewed
into, leaving BEGIN/END behind) and vendor-prefixed keys, which sit below
the opaque-token floor and carry no credential word.
`traces.test.ts` used `'x'.repeat(500)` to exercise truncation. That is
indistinguishable from an opaque token and now redacts to the marker,
leaving nothing to truncate, so the fixture is word-broken and the
redaction behaviour is asserted in two separate tests instead.
Verified: 573 tests pass in evals-core. Two shipped reports replayed through
the new code leave no credential behind, with client id, domain and audience
preserved. 28 PROMPT.md files give 5 correct redactions and no false
positives; 123 shipped text fields give none. Every new pattern has a paired
negative test — a false positive here costs a run its security score, since
the judge reads the marker as evidence a secret was exposed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tanya732
force-pushed
the
feat/add-redaction-in-serializers
branch
from
August 26, 2026 12:48
9f2030c to
baec173
Compare
tanya732
marked this pull request as ready for review
August 26, 2026 12:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
serializers.tsis the single choke point for everything a run publishes — JSON results, HTML reports, and dashboard traces.redactSecretsalready existed (#238), but was only wired intoformatCommandTracefor LLM-bound traces.This PR wires the publish path through the same scrubber across 6 sites and fixes three matcher gaps found by measuring against shipped reports.
Redaction happens during serialization, after graders and scoring, so it cannot affect verdicts or scores. It also keeps the report templates free of redaction logic.
Approach
Extends the existing
utils/redact.tsand wiresredactSecrets/redactArgsinto:formatStepserialiseTracemapGradersserialiseBaselineserialiseAgentserialiseErrorformatStepredacts before truncation so a partial credential cannot be leaked.Deliberately left readable: client IDs, user IDs,
AUTH0_DOMAIN, audiences, andsession_id. These are identifiers/configuration rather than credentials and are useful for verifying SDK wiring and correlating traces.Matcher fixes
VALUEguard: prevents status codes such as401and403from being incorrectly redacted.redactArgsconsults object keys: catches credentials such as{ Authorization: 'Bearer …' }while keeping end-anchored matching to avoid false positives liketoken_endpoint_auth_method.reset_password_2fa.Additional coverage
sk-,ghp_,xox...redactArgssupport for nested objects and arrays.envcontent passed as a string valueTesting Results
evals-core: 573/573 passing (+21 tests)PROMPT.mdfiles: 5 correct redactions, 0 false positivesTesting
Checklist