feat(evals): add MFA API evals for auth0-auth-js and auth0-server-js

[subhankarmaiti]

Two evals covering the MFA API path, where the app exchanges credentials itself and has to drive enrollment, challenge and verify rather than letting Universal Login prompt. Both mirror the existing step-up siblings under mfa/.

auth0_auth_js_mfa builds HTTP routes for a mobile client: authenticator-app setup with a QR code, challenge for an already-enrolled user, recovery-code sign-in, and listing and removing factors. auth0_server_js_mfa does the same for a session-backed web app, plus the SMS path, and requires the session to end up signed in so /profile and the transfers call keep working.

Scaffolds are minimal Express + TypeScript, one per package, with no MFA wiring, so the L1 needles can only come from the model's own output.

@auth0/auth0-api-js is not covered: it validates tokens and has no MFA surface.

Replaces #211, which had three defects this derivation caught:

• the storeOptions judge asserted that all the MFA methods take store options as a second argument; only verify does (server-mfa-client.ts:30,41,52,68)
• a notContainsInSource check on a session-secret value that appears in neither the prompt nor the scaffold, so it always passed
• challenge_type was missing from the server-js L2 set while binding_code was present

Both now pass every grader on agent+mcp+skills with claude-opus-5 :

Eval	Graders	Overall	Correctness	Hallucination	Security
auth0_auth_js_mfa	34/34	98.4 A	100	100	100
auth0_server_js_mfa	40/40	93.7 A	100	100	100

Getting there took two rounds of grader fixes. The last was the holistic judge in both files, which opened with "judge against the installed package types, not recall". That is unsatisfiable, because node_modules is excluded from the grading corpus, so the types were never in the judge's input. The judge fell back on recall anyway, cited the instruction as grounds to reject, and failed code it had just agreed was structurally complete, flagging real fields (recoveryCodes, oobCode, bindingCode) as invented. Both holistic judges now use the three-clause shape the react, vue, angular, swift and android siblings use, naming symbols the judge can find in the corpus.

Still unmeasured: that job runs one configuration and one model, with no baseline, so nothing here shows the graders discriminate. Worth a baseline and a second model before treating them as calibrated.

The server-js Docs Quality 66 is an allowlist gap, not an agent problem. The agent fetched raw.githubusercontent.com/auth0/auth0-auth-js/main/packages/auth0-server-js/MFA.md successfully, and that host is not in scoring.docUrlSources in eval.config.js, so the lookup scored 33 + 33 + 0. Adding it is a separate change. The server-js Setup Friction 90 and Error Recovery 80 are one provider error.

One deliberate deviation from docs/ADDING_EVALS.md: no committed .env.example, and so no wroteFile('.env') grader. compile_command is tsc and needs no environment.

The speakeasy and otplib L2 needles have no field evidence. They are carried over because mfa/react/graders.ts:16-17 already asserts them and the siblings are the specification.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: d234108a-9352-462a-8023-fd52dcd7f874

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}