Skip to content

fix(keystore): fixed-width ECDSA r/s and X/Y encoding (leading-zero bug) + CI that actually runs - #17

Closed
amirradjou wants to merge 204 commits into
orbitdb:mainfrom
amirradjou:hq/fix-leading-zero-encoding
Closed

amirradjou wants to merge 204 commits into
orbitdb:mainfrom
amirradjou:hq/fix-leading-zero-encoding

Conversation

@amirradjou

@amirradjou amirradjou commented Sep 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Fixes the ~1.5% random signature / public-key verification failures that made the test suite flaky and silently dropped ~1-2% of log entries, then makes CI actually run and gate.

big.Int.Bytes() strips leading zero bytes, so about 1 in 128 P-256 signatures (r || s) and 1 in 128 public keys (X || Y) were encoded 63 bytes long. VerifyMessage split those at len/2 and ReconstructPublicKeyFromHex rejected them (invalid public key length: 63), so freshly created identities could not verify their own signatures and every Log read path (Get, Values, Traverse, JoinEntry) dropped the affected entries with Skipping entry with invalid signature.

What changed

  • keystore (keystore/key-store.go): encode r/s and X/Y with big.Int.FillBytes into fixed 32-byte halves (CoordinateSize, SignatureSize, PublicKeySize); new EncodePublicKey is the single public-key encoder; VerifyMessage and ReconstructPublicKeyFromHex now require the exact length instead of "at least 64, split in the middle".
  • identities / providers: CreateIdentity uses keystore.EncodePublicKey; VerifyIdentity and Identities.Verify use keystore.ReconstructPublicKeyFromHex instead of their own midpoint-split parsers.
  • Regression tests (all fail against the old encoding, verified by temporarily re-introducing it):
    • keystore: 3000 fresh keys sign -> verify with fixed-width assertions; deterministic vectors with a known leading-zero X, Y, r and s (fixed-width verifies, the legacy 63-byte form is rejected); zero-padding unit test; malformed-length rejection.
    • identities: 3000 identities Create -> VerifyIdentity -> Sign -> Verify; malformed public-key rejection.
    • identities/providers: known short-X key through CreateIdentity/VerifyIdentity; 3000-identity loop.
    • oplog: 1000 fresh identities NewEntry -> VerifyEntrySignature (+ CBOR round trip); 300 Log.Append -> Get/Values read-backs.
  • chore(lint): clear the four pre-existing staticcheck findings (two unused mu fields, an unused test helper, a no-op type assertion) so staticcheck can gate.
  • ci: runs-on: ${{ matrix.os }} (ubuntu/macos/windows at the go.mod floor 1.23, plus ubuntu on stable), go mod download + build + go test -race -count=1 -coverprofile + a -count=20 loop over the crypto packages, upload-artifact@v4; a separate lint job with go mod verify, go mod tidy -diff, gofmt, go vet, staticcheck and govulncheck via go run ...@latest; permissions: contents: read; concurrency cancel; .github/dependabot.yml (gomod grouped + github-actions).

Verification (local, Go 1.26.8, linux/amd64)

Before (upstream/main @ 1064195):

$ go test -count=50 ./keystore/ ./identities/... ./oplog/
ok    orbitdb/go-orbitdb/keystore
FAIL  orbitdb/go-orbitdb/identities             (TestVerifyIdentity x1, TestSignAndVerify x1)
ok    orbitdb/go-orbitdb/identities/identitytypes
FAIL  orbitdb/go-orbitdb/identities/providers   (TestVerifyIdentity x1, TestCreateIdentity x2)
FAIL  orbitdb/go-orbitdb/oplog                  (TestLog_Values x3, TestLog_AppendAndGet x3, TestLog_Join, TestLog_Traverse, TestVerifyEntrySignature)
=> 3 of 5 packages FAIL, 14 test-case failures

$ go test -race -count=1 ./...
7 packages ok, FAIL syncutils (TestSyncSendAndReceiveHead, orbit-sync_test.go:422 SyncedCh ordering), 0 DATA RACE warnings

After (this branch):

$ go test -count=50 ./keystore/ ./identities/... ./oplog/
ok    orbitdb/go-orbitdb/keystore                  15.812s
ok    orbitdb/go-orbitdb/identities                53.084s
ok    orbitdb/go-orbitdb/identities/identitytypes   0.007s
ok    orbitdb/go-orbitdb/identities/providers      30.768s
ok    orbitdb/go-orbitdb/oplog                     17.799s
=> 5 of 5 ok, 0 failures

$ go test -count=1 ./...            => all 8 packages ok
$ go test -race -count=1 ./...      => 7 packages ok, FAIL syncutils (same pre-existing TestSyncSendAndReceiveHead ordering flake; fails 4/5 runs on untouched upstream/main too), 0 DATA RACE warnings
$ go build ./... && go vet ./... && gofmt -l . && go mod tidy -diff   => clean
$ go run honnef.co/go/tools/cmd/staticcheck@latest ./...             => clean (was 4 findings)
$ go run golang.org/x/vuln/cmd/govulncheck@latest ./...              => 2 reachable (see below)

The TestSignVerifyManyFreshKeys log line from one run: 3000 keys: 30 had a leading-zero coordinate, 25 signatures had a leading-zero r or s (all verified).

Known red in the new CI until follow-ups land

  • govulncheck (lint job): GO-2026-5073 and GO-2026-5684 in go-ipld-prime v0.21.0, reachable from oplog.Decode. The fix is go get github.com/ipld/go-ipld-prime@v0.23.0 && go mod tidy, but v0.23.0 declares go 1.25.7, so it raises the module's Go floor from 1.23 (and the CI matrix version). Left out of this PR as a separate decision.
  • syncutils TestSyncSendAndReceiveHead under -race: pre-existing ordering flake (peer-join event arrives on SyncedCh before the Add payload), unrelated to this change.

Note: the P-256 FillBytes fix is an explicit stopgap; the planned move to secp256k1 via go-libp2p core/crypto (DER signatures, compressed keys) replaces this code entirely.


Opened as a draft from the amirradjou fork; identical to amirradjou#1 (merged there). Happy to split the CI commit into its own PR if preferred.

haydenyoung and others added 30 commits October 30, 2024 04:15
Add CID Generation and Unit Tests for Entry Encoding
Refactor: Capitalize Log and Clock Structs for Exported Access
Added ID, clock, and Version to the entry and update the tests
Add Identity Support and Digital Signature to Entry Structure
amirradjou and others added 28 commits December 2, 2024 17:19
* Create go.yml

* update go version format
big.Int.Bytes() strips leading zero bytes, so about 1 in 128 P-256
signatures (r || s) and 1 in 128 public keys (X || Y) came out 63 bytes
long. VerifyMessage split those at len/2 and ReconstructPublicKeyFromHex
rejected them outright, so ~1.5% of freshly created identities could not
verify their own signatures and every Log read path silently dropped the
affected entries ("Skipping entry with invalid signature", "Expected 3
Entries, got 2", "invalid public key length: 63").

- keystore: encode r/s and X/Y with big.Int.FillBytes into fixed
  CoordinateSize (32-byte) halves; add EncodePublicKey as the single
  encoder and make VerifyMessage / ReconstructPublicKeyFromHex require
  the exact SignatureSize / PublicKeySize length instead of ">= 64,
  split in the middle".
- identities, providers: use keystore.EncodePublicKey and
  keystore.ReconstructPublicKeyFromHex instead of the ad-hoc
  X.Bytes()||Y.Bytes() encoder and midpoint-split parsers.
- tests: 3000-iteration Create -> Sign -> Verify loops in keystore,
  identities and providers, a 1000-identity oplog entry loop plus a Log
  append/read-back loop, and deterministic vectors with a known
  leading-zero X, Y, r and s (fixed-width verifies, legacy 63-byte form
  is rejected). All of them fail against the old encoding.

Before: go test -count=50 ./keystore/ ./identities/... ./oplog/ -> FAIL
in identities, identities/providers and oplog (14 test failures).
After: all five packages ok, 0 failures.
- databases/database.go, syncutils/orbit-sync.go: drop the unused `mu`
  mutex fields (U1000); the peerMap guard is tracked separately.
- databases/keyvalue.go: remove a no-op `.(interface{})` assertion (S1040).
- storage/ipfs_test.go: delete the unused encodeCBORMap helper (U1000).

`go run honnef.co/go/tools/cmd/staticcheck@latest ./...` is now clean so
it can gate in CI.
…, tidy

The workflow declared a three-OS matrix but ran everything on
`runs-on: ubuntu-latest`, ran `go mod tidy` as the install step (so a
dirty go.mod could never fail), used the deprecated upload-artifact@v3
and had no lint at all; the org repo has 0 recorded runs.

- lint job (Go stable): go mod verify, `go mod tidy -diff`, gofmt -l,
  go vet, staticcheck and govulncheck via `go run ...@latest` so there is
  no tool-install step to maintain.
- test job: `runs-on: ${{ matrix.os }}` on ubuntu/macos/windows at the
  go.mod floor (1.23) plus ubuntu on stable, `go mod download`, build,
  `go test -race -count=1 -coverprofile`, and a `-count=20` loop over
  the keystore/identities/oplog packages so the fixed-width ECDSA
  encoding cannot regress into a flake; upload-artifact@v4 with a
  per-matrix artifact name.
- least-privilege `permissions: contents: read`, cancel superseded runs.
- .github/dependabot.yml for gomod (grouped) and github-actions, weekly.

Known red until follow-ups land: govulncheck reports GO-2026-5073 and
GO-2026-5684 (go-ipld-prime v0.21.0, reachable from oplog.Decode); the
fix, `go get github.com/ipld/go-ipld-prime@v0.23.0`, raises the go
directive to 1.25.7 and is left for a separate decision. The syncutils
TestSyncSendAndReceiveHead ordering flake under -race is tracked
separately as well.
@amirradjou amirradjou closed this Sep 17, 2026
@amirradjou
amirradjou force-pushed the hq/fix-leading-zero-encoding branch from 0180f05 to 4a5cd7f Compare September 17, 2026 18:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants