Repository navigation
crypto: accelerate OpenSSL GHASH on C906 - #527
Conversation
Build outputsCompared with the build of main at 6c78048; codex/openssl-aes is at 1db3b62, so outputs changed since then are listed too. 4 changed, 1 added, 0 removed of 595 outputs.
|
dormancygrace
left a comment
dormancygrace
left a comment
There was a problem hiding this comment.
Reviewed at ce4fbf5. I found no correctness defect.
What I checked
- ABI: 144/208-byte frames are 16-byte aligned;
raand s0–s11 are saved. - Key buffer: the 896-byte
kp(320 + 576) is filled exactly, and entry 8 = {H[0], H[1]} matchesgcm_init_4bit. - Length guard:
done != lenkeepsgcm_ghash_4bit(a do-while loop) from getting a zero length. - Multiplier: every
muloperand is masked below 2^32, so the C906 multiplier stays on its fixed short path. - Dispatch: the same vendor-key-11 /
secure_getenv/ per-callprctlscheme as ChaCha.native-path-results.jsonshows hits only with the opt-in, and none under generic, vectors off or seccomp. imageCurlEnv(): keeps operator overrides for both variables.- Data: I recomputed the 320 medians from
native-results.jsonland found no mismatch. GHASH at 16 KiB goes 28.55 → 42.79 MiB/s. - End to end: I ran the real
gcm_c906.cand the generated assembly against a bit-serial SP800-38D GHASH:- 40 keys including all-ones and H=1, lengths 3072–16400, offsets 0–7: 1980/1980 pass.
- Two deliberately broken variants fail.
- Caveat: QEMU has no XTheadVector, so I mapped the
th.v*words to RVV 1.0. This checks the logic, not the 0.7.1 encodings; those rest on the generator's binutils check.
Requests
-
Where it runs.
curlImageArgs(..., true)pins ChaCha on the first attempt, so this path runs only in the second, generic attempt. With the opt-in default off, most GHASH work also stays on the 4-bit tables:- every
gmult(tags, single blocks); - every input under 3072 B (handshake, SSH and SRTP records, and small TLS records);
- every process without the opt-in;
- the
len % 80leftover after each vector prefix.
Please state in the README and the PR description that GHASH stays table-based (not constant-time) outside this path. A cheap improvement:
kpis already expanded invector_prefix, so the leftover blocks could go throughossl_ghash_blocks_c906before the cleanse instead ofgcm_ghash_4bit. The scalar C906 GHASH (no vector unit, noprctl) could also become a constant-time default. Per the go-crypto-c906 README it runs at about the speed of the 4-bit code, but that is a kernel figure, not one measured inside OpenSSL. - every
-
CI.
generate-ghash.py --checkruns the Python model and checks encodings, andtest-dispatch.pybuilds the glue for x86 with stub assembly. Nothing checks the key built from the 4-bit table, the Xi byte swaps or the 320/576 layout.verify-path.ccounts entries with one all-zero key and never checks output. A harness like mine, run under qemu with the generated file, would close this. I can share it. -
Expectation. The gain is 1.5×, not the ~3× I projected on #511: OpenSSL's 4-bit baseline (28.5 MiB/s) is about twice the kernel's generic figure I assumed. Software GCM open improves +10.0% / +8.2% (AES-128/256), and with AES DMA +3.4% / +3.7%. The README table has no small-record row.
native-medians.csvshows 1400 B at −2.0% / −1.4% (AES-128 open / AES-256 seal), within the trial spread. Please add that row and say there is no small-record regression. -
Nit. The key is rebuilt on every eligible call, which with
GHASH_CHUNKmeans once per 3 KiB: 4 multiplies, a 576-byte table and oneprctl. This is probably why 3072 B reaches 39.7 MiB/s against 42.8 at 16 KiB. It is a reasonable trade for no context growth, but please document it.
|
Proposed change for the leftover blocks (request 1 of my review), tested.
--- gcm_c906.c 2026-10-27 11:39:30.579465213 +0200
+++ gcm_tail.c 2026-10-27 12:09:10.396073109 +0200
@@ -60,7 +60,7 @@
{
u64 kp[112], powers[5][2], acc[2];
unsigned int pos = 320, i, q, j, lane;
- size_t groups = len / 80;
+ size_t groups = len / 80, blocks = len % 80 / 16;
u64 h0 = table[8].hi, h1 = table[8].lo, carry = h0 >> 63;
u64 lo = (h1 << 1) ^ carry;
@@ -89,12 +89,16 @@
acc[0] = __builtin_bswap64(xi[1]);
acc[1] = __builtin_bswap64(xi[0]);
ossl_ghash_blocks_c906_v(acc, kp, in, groups);
+ /* The scalar key in kp covers the remaining whole blocks, so they stay
+ * constant-time instead of returning to the 4-bit table. */
+ if (blocks != 0)
+ ossl_ghash_blocks_c906(acc, kp, in + groups * 80, blocks);
xi[0] = __builtin_bswap64(acc[1]);
xi[1] = __builtin_bswap64(acc[0]);
OPENSSL_cleanse(kp, sizeof(kp));
OPENSSL_cleanse(powers, sizeof(powers));
OPENSSL_cleanse(acc, sizeof(acc));
- return groups * 80;
+ return groups * 80 + blocks * 16;
}
size_t ossl_gcm_c906_prefix(u64 xi[2], const u128 *table,
Test. The harness below runs the real
Results:
QEMU has no XTheadVector, so for this run the harness.c and stubs.c#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
typedef uint64_t u64; typedef uint32_t u32; typedef uint8_t u8;
typedef struct { u64 hi, lo; } u128;
/* reference: SP800-38D bit-serial multiply on byte strings */
static void gfmul(u8 x[16], const u8 h[16]) {
u8 z[16] = {0}, v[16]; memcpy(v, h, 16);
for (int i = 0; i < 128; i++) {
if (x[i/8] & (0x80 >> (i%8))) for (int k=0;k<16;k++) z[k]^=v[k];
int lsb = v[15] & 1;
for (int k=15;k>0;k--) v[k] = (v[k]>>1) | (v[k-1]<<7);
v[0] >>= 1; if (lsb) v[0] ^= 0xe1;
}
memcpy(x, z, 16);
}
static void ref_ghash(u8 xi[16], const u8 h[16], const u8 *in, size_t len) {
for (size_t i=0;i+16<=len;i+=16){ for(int k=0;k<16;k++) xi[k]^=in[i+k]; gfmul(xi,h); }
}
static u64 be64(const u8 *p){u64 v=0;for(int i=0;i<8;i++)v=v<<8|p[i];return v;}
size_t ossl_gcm_c906_prefix(u64 xi[2], const u128 *table, const u8 *in, size_t len);
int ossl_riscv_has_xtheadvector = 1;
static uint64_t rs=88172645463325252ULL; static u64 rnd(void){rs^=rs<<13;rs^=rs>>7;rs^=rs<<17;return rs;}
int main(void){
static u8 buf[20000+64]; int fails=0, n=0;
size_t lens[]={3072,3200,3216,4096,16384,16400,3136, 80*40, 6144};
for (int key=0; key<40; key++){
u8 h[16]; for(int k=0;k<16;k++) h[k]=rnd();
if (key==0) memset(h,0xff,16);
if (key==1) { memset(h,0,16); h[15]=1; }
if (key==2) { memset(h,0,16); h[0]=0x80; } /* H = 1 */
u128 table[16]; memset(table,0,sizeof table); table[8].hi=be64(h); table[8].lo=be64(h+8);
for (size_t li=0; li<sizeof lens/sizeof*lens; li++) for (int off=0; off<8; off+= (key&1)?3:1) {
size_t len=lens[li]; u8 *in=buf+off; for(size_t i=0;i<len;i++) in[i]=rnd();
if (key==3) memset(in,0xff,len);
u8 x0[16]; for(int k=0;k<16;k++) x0[k]=rnd(); if(key==4) memset(x0,0,16);
u64 xi[2]; memcpy(xi,x0,16);
size_t done = ossl_gcm_c906_prefix(xi,table,in,len);
u8 ref[16]; memcpy(ref,x0,16); ref_ghash(ref,h,in,done);
n++;
if (done != len/16*16 || memcmp(ref,xi,16)) { fails++; if (fails<5) printf("FAIL key=%d len=%zu off=%d done=%zu\n",key,len,off,done); }
}
}
printf("%d cases, %d failures\n", n, fails); return fails!=0;
}#include <stdarg.h>
char *secure_getenv(const char *n){ (void)n; return "auto"; }
int prctl(int op, ...){ (void)op; return 2; }python3 to_rvv.py ghash-c906.S ghash-rvv.S # .4byte th.v* -> RVV 1.0 mnemonics, +v in .option arch
CC="riscv64-linux-gnu-gcc -O2 -static -march=rv64gc_xtheadbb -Iinc" # inc: crypto/modes.h, crypto/riscv_arch.h, openssl/crypto.h
$CC -c gcm_c906.c && $CC -c ghash-rvv.S && $CC harness.c gcm_c906.o ghash-rvv.o stubs.c -o harness
qemu-riscv64 -cpu max,vlen=128,xtheadbb=true ./harness |
84de85e to
9e3de10
Compare
ce4fbf5 to
cf3cbbd
Compare
dormancygrace
left a comment
dormancygrace
left a comment
There was a problem hiding this comment.
Restacked head cf3cbbddaa6c98222c3a1acd7655375b30360ce3 onto the cleaned AES #511 (9e3de102), directly above SHA-256 #508. SHA-1 offload commits and CI are absent; AES verification and DMA helper cleanup come from the parent.
All 15 final-head checks pass. The clean C906/O3 build applies every patch with zero fuzz, retains all 5808 public exports, and has no SHA-1 offload or legacy-provider DMA objects. GHASH generator reference/encoding checks, dispatcher sanitizer/negative controls and downloader tests pass. scope-cleanup-validation.json pins this build.
0010-ghash-c906.patch is byte-identical to the previously qualified patch. Native timing and compatibility results remain attributed to their recorded pre-cleanup builds; no native rerun is claimed here. The scalar-remainder proposal in comment 6439303613 remains a separate pending change and is not included in this restack.
dormancygrace
left a comment
dormancygrace
left a comment
There was a problem hiding this comment.
Reviewed the final head 0d12f28d1c45f1c309eb7b36d272a35e02d9d75c. The scalar-remainder proposal in comment 6439303613 is implemented, and the four requests in review 5665138577 are addressed.
- Eligible calls process their remaining 0–4 blocks with the existing scalar C906 key before cleansing. No persistent context growth, threshold change or broader opt-in is introduced. Short calls, gmult/tag work, disabled vectors and processes without opt-in still use the data-indexed 4-bit implementation; the documentation no longer implies whole-library constant-time GHASH. The downloader reaches this opt-in on its second, generic attempt after ChaCha.
- Fast CI now applies the complete patches with zero fuzz to SHA-256-verified OpenSSL source and runs the real glue/generated assembly against an independent bit-serial SP800-38D reference anchored by a known-answer test. All 3520 cases pass: 40 keys, 11 lengths, eight offsets, every scalar-tail count, initial Xi, key layout and input/context/table guards. All five deliberate arithmetic/glue corruptions are detected. QEMU maps the draft vector words to RVV 1.0 for arithmetic coverage; the separate encoding check and a physical C906 run of all 3520 cases with the original XTheadVector instructions also pass.
- The 1400 B results and their trial spread are documented, without a new small-record or HTTPS gain claim. Earlier throughput figures are explicitly historical; the scalar-tail change was qualified for correctness, not newly benchmarked.
- Per-call key construction (four scalar multiplies, a 576-byte vector table and prctl, repeated per 3 KiB GHASH chunk) and unchanged context sizes are documented.
A clean C906/O3 build retains exactly 5808 public definitions with no test exports. Native generic/vector/vector+DMA profiles pass the AES, AEAD, GCM, CTR, tail/IV and SHA suites; 28 additional independent EVP SHA tail answers pass. Final production patch hashes match this tested build. scalar-tail-validation.json records identities and results. No remaining correctness finding.
All 19 final-head CI checks pass, including the firmware build.
Add opt-in C906 GHASH using the shared BSD-3-Clause five-block generator. Four vector lanes plus a scalar block process the prefix; the remaining 0–4 whole blocks reuse the expanded scalar key. OpenSSL retains AAD, padding, lengths, tags and operation boundaries.
Dispatch requires at least 3072 B,
NANOKVM_OPENSSL_GHASH=auto, the startup all-CPU XTheadVector probe, and current-thread vector permission. This does not make all OpenSSL GHASH constant-time: single-blockgmult(including tag work), short inputs, processes without the opt-in and unavailable vector paths retain the data-indexed 4-bit tables. Eligible complete calls, including their scalar remainder, use the C906 arithmetic. The downloader first requests ChaCha, so GHASH acceleration applies on its second generic-cipher attempt.Each eligible call rebuilds the key (four scalar multiplies and the 576-byte vector table) and queries thread permission; in
GHASH_CHUNKprocessing this repeats per 3 KiB. Context sizes remain 448 B for GCM and 984 B for the AES-GCM provider, without a persistent expanded-key cache.Historical measurements from the pinned pre-cleanup library: 16 KiB GHASH 42.79 versus 28.55 MiB/CPU-s; full software AES-128/256-GCM opening improves 10.0%/8.2%, or 3.4%/3.7% with AES DMA already enabled. At 1400 B, AES-128 open / AES-256 seal software-only medians differ by −2.0%/−1.4%, within the recorded trial spread; no material small-record regression was established. No new GHASH throughput or HTTPS gain is claimed for the scalar-tail revision.
Current validation:
scalar-tail-validation.jsonrecords exact patch, library and helper identities.The previously recorded OpenSSH seccomp and TLS download qualifications retain their original identities; they were not repeated for this change. Stacked on AES #511 and SHA-256 #508; SHA-1/DES/TDES/SM3/SM4 offloads and a combined-record kernel GCM API are excluded.