feat(evals): add SPA JS MFA step-up eval

[subhankarmaiti]

Adds an MFA step-up eval for @auth0/auth0-spa-js, covering the vanilla-JS browser SPA case alongside the existing react, angular, vue and nextjs step-up evals.

Two things differ from those siblings, both driven by the SDK. The step-up assertion accepts either mechanism — the acr_values authorization parameter or interactiveErrorHandler: 'popup' — because acr_values is typed on AuthorizationParams but the popup handler is what EXAMPLES.md and static/step-up.html actually document, so requiring acr_values alone would fail a solution that follows the SDK's own guidance. The policy-URI check moves to a conditional L5 judge for the same reason.

There is also a new L4 check for cacheMode: 'off'. The token cache keys on [prefix, clientId, audience, scope] and does not include acr_values or max_age, so a silent step-up request without it returns the cached pre-MFA token and the step-up silently does nothing.

Grader mix is 13 deterministic to 7 judge across L1-L5 plus the holistic judge, in line with the other step-up evals.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 9e3fd6e7-0952-418e-9e2d-7c9208fce4c2

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sanchitmehtagit]

L1/L2 graders are well-structured and the L3 judge is nicely nuanced. Two blocking issues below. Happy to re-review once those land.

[sanchitmehtagit]

This regex hard-fails two solution shapes that every other grader in this file declares valid. A redirect-based step-up or an interactiveErrorHandler: 'popup' solution (the SDK's own documented approach) never calls getTokenSilently, so cacheMode is meaningless on those paths. Yet both fail L4 while passing L5 judge #3, which explicitly accepts max_age: 0 or a redirect login. The two levels end up contradicting each other.

I'd replace this with a judge that matches the L5 accept set:

judge(
  'When requesting the stepped-up token, does the solution avoid returning the stale pre-MFA ' +
    'cached token -- via cacheMode: \'off\', a distinct step-up scope/audience, or a redirect/popup ' +
    'login? (The SDK cache key is clientId+audience+scope only; acr_values/max_age alone do not bust the cache.)',
  GraderLevel.L4,
)

[sanchitmehtagit]

Both of these L1 graders (@auth0/auth0-spa-js and createAuth0Client on the next line) are already satisfied by the unmodified scaffold -- package.json declares the dependency and src/app.js imports createAuth0Client. An agent that writes nothing scores 2/5 on L1, which inflates baseline scores.

Would it make sense to swap these for MFA-specific symbols the scaffold cannot satisfy -- getTokenSilently, the multi-factor policy URI, or interactiveErrorHandler?

[sanchitmehtagit]

contains('amr') is a 3-character substring scan over all workspace files. It passes on any comment, README line, or identifier embedding those letters -- it does not verify the claim is actually read from getIdTokenClaims().

A tighter regex would require an actual claim access:

matches(
  String.raw`(claims|idTokenClaims)\s*(\?\.|\.)\s*\[?['"']?amr`,
  'AMR claim read from ID token claims',
  GraderLevel.L1,
)

[sanchitmehtagit]

notContains scans every workspace file including .md files agents frequently write. A correct solution whose design note says "we deliberately skip @auth0/auth0-react (vanilla JS only)" fails L2 hallucination despite being correct.

notContainsInSource would fix this for the package-name needles (@auth0/auth0-react, @auth0/guardian, speakeasy, otplib) -- it skips non-source files so prose mentions do not count as hallucinations.

[sanchitmehtagit]

This judge only checks that amr is tested before the transfer. A solution that triggers step-up and then unconditionally proceeds (never re-reading claims after the popup) satisfies this grader. If the user cancels the popup (PopupCancelledError), the transfer runs unguarded.

One option would be to add a second L4 judge:

judge(
  'After triggering step-up, does the code re-read the ID token claims and confirm "mfa" is ' +
    'present in amr before running the transfer, and abort if the popup is cancelled or times out?',
  GraderLevel.L4,
)