fix: reject small-order and off-curve public keys in EdDSA verification

[ivokub]

Description

Fixes a signature-forgery vulnerability in std/signature/eddsa (Verify/IsValid).

Bug. The verifier never validated the public key. For a small-order public
key A (any torsion-subgroup point, including the identity (0,1)), the term
[cofactor]·[H(R,A,M)]·A vanishes, so the verification equation
[cofactor]·([S]G − [H]A − R) = O no longer depends on the secret key. Anyone
could prove a "valid" signature over an arbitrary message without the secret
key, e.g. with A = (0,1), S = 1, R = G.

Fix.

1. Reject small-order public keys: IsValid returns 0 whenever
   [cofactor]A = O. This is exactly the torsion subgroup — precisely the
   class for which the equation degenerates. For [cofactor]A ≠ O the
   equation is a Schnorr relation in the prime-order subgroup, so forgery
   remains equivalent to discrete log. Cost: 2–3 point doublings.
2. Assert in-circuit that A and R are on the curve. The twisted Edwards
   group formulas use unchecked divisions which are undefined for off-curve
   inputs (a vanishing denominator makes intermediate values
   prover-controlled), and off-curve points would bypass the small-order
   check.

Scope notes:

• Mixed-order keys (A = A_r + torsion) remain accepted, intentionally: the
  cofactored equation annihilates the torsion component, so such a key is
  functionally its prime-order projection. Enforcing full subgroup membership
  ([order]A = O) would cost a full scalar multiplication for no soundness
  gain.
• Honest keys from gnark-crypto keygen lie in the prime-order subgroup and are
  unaffected — no breaking change for valid use.

Type of change

• Bug fix (non-breaking change which fixes an issue)

How has this been tested?

• New regression test TestEddsaSmallOrderPublicKey: all 5 supported
  curve configurations (BN254, BLS12-381, Bandersnatch, BLS12-377, BW6-761 —
  covering both cofactor-4 and cofactor-8 branches) × Groth16/PLONK. Honest
  witness accepted; identity A, order-2 A, off-curve A, and off-curve
  R rejected.
• Regression test confirmed to fail against the unfixed implementation.
• The audit-finding reproducer (identity key, S=1, R=G, full Groth16
  prove/verify) no longer yields a satisfying witness.
• go test ./std/signature/eddsa/ ./std/algebra/native/twistededwards/ ./examples/rollup/ pass (rollup is the only in-repo caller).

How has this been benchmarked?

• Constraint count, BN254 R1CS eddsa circuit: 7769 → 7799 (+30, +0.4%).
  No performance-critical paths changed.

Checklist:

• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation (godoc on Verify/IsValid)
• I have added tests that prove my fix is effective or that my feature works
• I did not modify files generated from templates
• golangci-lint does not output errors locally
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules

---

Note

High Risk
Fixes a signature-forgery vulnerability in cryptographic verification logic; behavior change for adversarial public keys but honest gnark-crypto keys remain valid.

Overview
Hardens in-circuit EdDSA verification (std/signature/eddsa) against a forgery class where small-order public keys (e.g. identity) let anyone satisfy the cofactored equation without the secret key.

IsValid now asserts A and R are on-curve (bad inputs make the circuit unsatisfiable), factors cofactor clearing into clearCofactor, and returns invalid when [cofactor]A is the identity while still requiring the usual signature equation. Mixed-order keys ([sk]G plus torsion) remain accepted; godoc calls out the divergence from gnark-crypto native verification and implications for hashing public-key coordinates.

Adds circuit tests across supported curves for small-order/off-curve rejection and for mixed-order acceptance, with native bigPoint helpers to craft witnesses.

^{Reviewed by Cursor Bugbot for commit fa81278. Bugbot is set up for automated code reviews on this repo. Configure here.}

[yelhousni]

LGTM real vulnerability, sound fix. Verified: test fails on unfixed code (all 5 curves), PR head green, +30 constraints exact (7,769 → 7,799).
A few asks though if it makes sense:

1. Document the divergence from gnark-crypto. Native validatePublicKeyPoint requires full subgroup membership; the circuit now requires only [cofactor]A ≠ O, so mixed-order keys are accepted in-circuit and rejected natively. Godoc should say so.
2. Warn identity-using callers. Mixed-order acceptance means one secret key → 8 valid public keys (cofactor 8). examples/rollup/circuit.go:169 hashes A.X, A.Y as the account identity, so this is live in-tree.
3. Fix IsValid's contract wording. It still says "returns 1 ... and 0 otherwise," but off-curve inputs now make the circuit unsatisfiable, not 0. One clause.
4. Add a positive test for a mixed-order key to pin the deliberate design decision.

[ivokub]

LGTM real vulnerability, sound fix. Verified: test fails on unfixed code (all 5 curves), PR head green, +30 constraints exact (7,769 → 7,799). A few asks though if it makes sense:

1. Document the divergence from gnark-crypto. Native validatePublicKeyPoint requires full subgroup membership; the circuit now requires only [cofactor]A ≠ O, so mixed-order keys are accepted in-circuit and rejected natively. Godoc should say so.
2. Warn identity-using callers. Mixed-order acceptance means one secret key → 8 valid public keys (cofactor 8). examples/rollup/circuit.go:169 hashes A.X, A.Y as the account identity, so this is live in-tree.
3. Fix IsValid's contract wording. It still says "returns 1 ... and 0 otherwise," but off-curve inputs now make the circuit unsatisfiable, not 0. One clause.
4. Add a positive test for a mixed-order key to pin the deliberate design decision.

Thanks for the comments! I implemented.

I kept the rollup example as is -- it is an old example and actually it doesn't create TX malleability -- the cofactor public keys map to different identities. Which ... may even be good in the rollup context, adds more obfuscation :)