feat(evals): add express-openid-connect MFA step-up eval - #244
feat(evals): add express-openid-connect MFA step-up eval#244sanchitmehtagit wants to merge 2 commits into
Conversation
Adds eval and scaffold for MFA step-up using express-openid-connect.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
subhankarmaiti
left a comment
subhankarmaiti
left a comment
There was a problem hiding this comment.
Nice eval, the step-up flow is the right thing to be measuring here and the scaffold runs cleanly (npm start listens, node --check passes, express-openid-connect resolves to 2.20.2). A few things on the graders before merge, mainly around signal strength.
| --- | ||
| id: express_oidc_mfa | ||
| name: Express OpenID Connect MFA Step-Up | ||
| scaffold: src/evals/scaffolds/express-oidc/auth0 |
There was a problem hiding this comment.
Tiny nit, non-blocking. All 13 other scaffold-using evals name the eval dir the same as its scaffold dir, so express-oidc for both would keep that consistent.
There was a problem hiding this comment.
Keeping the current approach. The eval id is already express_oidc_mfa which matches the scaffold name, and the dir name doesn't affect eval discovery.
Addressing PR Feedback decision
Approach
- ✅
Keep current(recommended)
Cosmetic-only; eval ID is already express_oidc_mfa - ⬜ Rename to express-oidc
For consistency
Reply
- ✅ Post (auto)
- ⬜ Skip (reply manually)
- ⬜ Other
|
The "Available evals" table in |
- Drop `authorizationParams` L1 grader that passes on unmodified scaffold
- Add `wroteFile('.env', ...)` L4 grader to verify .env creation
- Add `amr`/`acr` to `identityClaimFilter` in scaffold so L5 judge premise holds
- Trim scaffold comment to remove task hint
- Add `express_oidc_mfa` row to README Available evals table
2 participants
By submitting a PR to this repository, you agree to the terms within the Auth0 Code of Conduct. Please see the contributing guidelines for how to create and submit a high-quality PR for this repo.
Description
Adds an eval that measures whether an LLM agent can correctly implement MFA step-up in an Express app using
express-openid-connect.The task prompts the agent to protect a
/transferroute: check theamrclaim fromreq.oidc.idTokenClaims, redirect to Auth0 viares.oidc.login()withacr_valueswhen MFA is absent, and verify MFA completion server-side after the step-up redirect. The eval ships with a matching scaffold - a minimal Express app with Auth0 login already wired - so agents start from a realistic base rather than a blank file.Graders cover L1-L5:
acr_values,amr,idTokenClaims,oidc.login,authorizationParams)speakeasy,otplib,loginWithRedirect, etc.)amr/acrread from server-side verified claims onlyreq.oidc.idTokenClaimsrather thanreq.oidc.user(which is filtered and may omitamr/acr)References
Testing
npm run build && npm testpasses.Eval can be run manually with
npm run evals -- --eval express_oidc_mfa --mode agent.This change adds test coverage for new/changed/fixed functionality
Checklist