[dormancygrace]

Speed up AES on C906 with a separately reviewable scalar patch and a bounded direct SG2002 DMA patch. OpenSSL keeps streaming, padding, GHASH, authentication and TLS; FIPS uses the unchanged fallback assembly. No cryptodev/ENGINE interface is added.

0005-aes-c906.patch uses byte extraction plus scaled indexed loads. Five native rotations show about 32–33% less CPU per AES block than upstream, with unchanged key setup and tables. Repeated EVP updates with an existing key improve even at 16 B: ECB 33–38%, CBC 17–27%, CTR 21–30% across key sizes/directions. Complete EVP calls that reinitialize a key for every 16–128 B operation can still regress; these results are published too.

0005-aes-dma.patch selects initial minima against that faster software: 3072 B for ECB/CBC, 2048 B for CTR/GCM, and 512 B for a continuation on an already open descriptor. A 6000 B request now submits 4096+1904 B; a 16 B tail stays in software because another RUN loses there. The CTR/GCM minimum has one shared definition. Calls below the initial minimum skip the DMA helper, and request initialization happens only after validation and successful open.

Recorded pre-cleanup three-rotation EVP results (MiB/CPU-s, baseline / patched software / patched DMA):

• AES-128-GCM open, 1400 B: 7.779 / 11.071 / 11.034.
• AES-128-GCM open, 16 KiB: 8.036 / 11.208 / 14.141.
• AES-256-GCM open, 16 KiB: 6.695 / 9.462 / 13.620.

These use upstream software GHASH, without #527. The complete GCM numbers include five 3072 B DMA callbacks and a 1024 B software tail. A combined-record kernel API/persistent descriptor is a separate unselected experiment, not part of this implementation.

The common DMA cipher helper accepts AES only and rejects other algorithms before device access. SHA-1, DES, TDES and SM4 offloads are absent; the legacy provider no longer links the SG2002 transport. AES block verification is self-contained in experimental/openssl-aes/verify.c: 972 cases, independent OpenSSL 3.5.5 fingerprint 019ff5988780130c, exercised by CI.

Current cleanup validation: 92,880 ASan/UBSan AES continuation cases, 960 dense tail/failure cases, 25 GCM counter-wrap cases and seven rejected non-AES IDs; three helper negative controls and an incorrect-fingerprint negative control. Host EVP checks pass 972 block, 288 CTR, 408 AEAD, 2160 GCM and 20,736 independent-output/IV tail cases. A clean C906/O3 build applies every patch with zero fuzz, removes the SHA-1 offload object and retains all 5808 public exports. Exact identities and results are in scope-cleanup-validation.json.

The earlier pinned native build passed DMA-on/off checks, upstream AES vectors, four real AES-128/256-GCM HTTPS downloads under TLS 1.2/1.3, and generic/FIPS fallback machine-code comparisons. Its original 1944-case verifier included the 972 AES cases now extracted. Native checks and timing were not repeated for this cleanup; raw historical reports retain their original source/library hashes. AES assembly, algorithms and thresholds are unchanged. Context sizes in those measurements are 472/984 B (AES/GCM), 24 B above pre-DMA contexts; no per-context output buffer.

Raw key-size × mode × direction trials from 16 B through 16 KiB, the experimental low-threshold sweep including 1024 B regressions, all source/binary identities and standalone harnesses are in firmware/crypto/experimental/openssl-aes/. Stacked directly on SHA-256 #508; scalar and DMA changes are separate OpenSSL patches within this AES PR.

The current head incorporates the SHA-256 continuation/CI fixes from #508. AES code and its thresholds are unchanged. Rechecked all five patches against the pinned source with zero fuzz, the real SHA assembly/wrapper, AES helper sanitizers and negative controls, and the independent AES/GCM/CTR/tail host verifiers. All fast final-head CI checks pass; the full firmware build is still running.

[dormancygrace]

The CBC and CTR IV logic and the GCM 32-bit wrap split are correct against both the kernel driver and OpenSSL.

There is one correctness assumption that nothing verifies yet, and the provider glue is not covered by any test. I'd treat both as gates for leaving draft.

[github-actions]

Build outputs

Compared with the build of main at 6c78048; codex/openssl-sha256 is at 06f3d60, so outputs changed since then are listed too.

4 changed, 1 added, 0 removed of 595 outputs.

	directory	files
changed	server	server/NanoKVM-Server
changed	tools	tools/openssl/usr/lib/libcrypto.so.3 tools/openssl/usr/lib/ossl-modules/legacy.so tools/openssl/usr/share/nanokvm/openssl/source.json
added	tools	tools/openssl/usr/share/licenses/nanokvm-openssl/C906-BSD-3-Clause.txt
Hashes: outputs-sha256 artifact of this run.

[dormancygrace]

A more efficient AES-GCM design for this offload

Splitting GCM into its parts shows where the time goes on the SG2002, and why offloading only the CTR stream through ctx->ctr gains little.

Part	Cost on SG2002/C906 (measured in this repository)
AES-CTR keystream + XOR	engine: about 35 µs per 4 KiB request (≈117 MB/s), see sg2002_aes_probe.c
GHASH (GF(2¹²⁸) multiply per 16 bytes)	CPU: 46–49 MB/s with the C906 GHASH of patch 0041, 14–15 MB/s with generic 64-bit code (firmware/cpu/go-crypto-c906/README.md)
H = E(0), J0, E(J0) ⊕ GHASH for the tag	one block each, negligible

So GHASH is the bottleneck, not AES. The current patch has three structural costs on top of that:

1. No overlap. SG2002_CRYPTO_RUN busy-waits in engine_wait() (readl_poll_timeout with 0 µs delay for up to 200 µs). On the single C906 core the engine time is pure CPU time for the caller. CRYPTO_gcm128_encrypt_ctr32 then hashes afterwards, so the cost per record is CTR + GHASH in series.
2. The call pattern is fixed by gcm128. The stream function sees at most GHASH_CHUNK/16 blocks (3 KiB), so a 16 KiB TLS record costs six device transactions. Each one does open + fstat + INFO + RUN + close and takes the driver mutex twice.
3. OpenSSL's GHASH on this target is the generic 4-bit table code. It is slower than the C906 GHASH, and its table lookups are data-dependent.

The kernel already contains the efficient version of this: crypto_gcm.inc starts a chunk of up to 4 KiB on the engine with crypto_start_locked(), hashes the AAD and the previous chunk's ciphertext with the 0041 GHASH while the engine runs, then calls crypto_finish_locked(). The engine time is hidden behind GHASH, and per record the cost is roughly GHASH alone. In-kernel gcm(aes) went from 13 to 33 MB/s at 64 KiB with this design and the C906 GHASH. Userspace cannot reach it today, because AF_ALG (CRYPTO_USER_API_AEAD) is off.

Proposal, in order of value

1. Use the C906 GHASH in OpenSSL first, independent of the device. Generate an OpenSSL-ABI variant of the 0041 GHASH (gcm_init/gcm_gmult/gcm_ghash for gcm128.c) from gen_ghash_linux.py, the same way #506 brings the ChaCha assembly into OpenSSL. Select it at runtime on XTheadVector, with the scalar 32×32 Karatsuba variant otherwise. This roughly triples GHASH, helps every AES-GCM user even without the module, and removes the table-based GHASH. It is also the prerequisite for anything below, since GHASH bounds the result.

2. Offload a whole record with a single ioctl that reuses the kernel's overlapped path. Add a SG2002_CRYPTO_GCM request: key, IV (96-bit only, otherwise fall back), AAD of at most a few hundred bytes, up to 16 KiB of data plus a 16-byte tag; it encrypts, or decrypts and verifies. Inside, run the existing sg2_gcm chunk loop (start chunk → GHASH while it runs → finish), with the tag compared in constant time in the kernel and the output wiped on failure. In OpenSSL, implement it as the oneshot member of a PROV_GCM_HW instead of overriding ctx->ctr. In 3.5.9, gcm_tls_cipher() passes each whole TLS record (AAD, payload, tag) to hw->oneshot in one call, so a record costs one syscall and the engine time disappears behind GHASH. cipherupdate/cipherfinal, used by streaming callers, keep the software path with the step 1 GHASH.

A split START/FINISH ioctl pair would also allow overlap, but it means holding the engine mutex across two syscalls from userspace, which a killed or stopped process could leave held. The single-call version keeps lock ownership inside one ioctl, as now.

3. Keep the device handle per process. Open once (O_CLOEXEC), cache INFO and ABI, and latch ENOENT/EACCES/ENOTTY. This is also asked in #507.

4. Choose the threshold from measurements of the final design. Compare it against step 1 alone at 1.4, 4 and 16 KiB, with and without concurrent H.265 video, and against the SRTP latency the engine mutex adds. With an overlapped record call, the crossover against software GCM should sit well below 1 KiB of data, but that needs measuring.

Expected result. Software AES-GCM in OpenSSL becomes bounded by the faster GHASH (about 3× on that part). The device path approaches the in-kernel gcm(aes) figure (about 33 MB/s at large sizes) minus copy and syscall costs. The current design pays engine time + generic GHASH + six transactions per record. These are estimates from the component numbers above, not measurements of this PR.

[dormancygrace]

The serialized CTR/GHASH path and the 3 KiB stream-call boundary are real limitations. The current transport now caches INFO/ABI and permanent errors (f5c9efe); descriptors remain operation-owned. The combined AES/GHASH path needs to address both TLS versions before its crossover can replace the current thresholds.

Two source/measurement details change the proposed implementation:

• In OpenSSL 3.5.9, gcm_tls_cipher() uses hw->oneshot for the TLS 1.2 explicit-IV record format. TLS 1.3 (ssl/record/methods/tls13_meth.c, lines 218-238) uses separate AAD/data EVP_CipherUpdate and EVP_CipherFinal_ex calls. Overriding oneshot alone would leave TLS 1.3 on the existing path. A bounded combined AES/GHASH bulk-update operation can preserve the upstream streaming/final/tag state while covering both; it must keep the complete operation and lock lifetime inside one ioctl.
• The 14-15 MB/s baseline is the kernel generic GHASH, not OpenSSL's 4-bit implementation. Three rotated physical-C906 component trials (CRYPTO_gcm128_aad, offset 5, 0.15 CPU seconds per size) measured OpenSSL at 28.535 MiB/CPU-s and the C906 vector adapter at 43.496 at 16 KiB: +52.4%. At 16/64/256 bytes the same adapter regressed: 23.888/27.012/28.242 to 19.972/25.547/27.593. A previous full AES-128-GCM comparison improved about 10.7% at 16 KiB, while increasing each context by 640 bytes and slowing fresh-key setup. The next software candidate retains upstream short-block multiplication and prepares compact vector powers lazily; it still requires native qualification.

The correctness questions now have native evidence as well. The physical engine carries the full 128-bit CTR counter for AES-128/192/256, including carries into byte 11, byte 7 and the all-ones-minus-two vector. The new boundary split still removes that hardware-width dependency. The current integrated build (7b8c071; libcrypto SHA-256 91c98df16c068a6103a10036191179e74ff557b02029fea47281efe0ee522b72) passes all 288 CTR cases and 2160 extended GCM cases, plus the existing 408 AEAD cases, with DMA enabled/disabled. GCM also passes with /dev hidden in the test process's private mount namespace, exercising actual ENOENT fallback. Driver completion counters advanced and poison remained zero. No projected whole-record speedup is being treated as a measured result.

[dormancygrace]

Thanks for b4efcf2.

• CTR carry: every CTR request is now split before the low 32-bit wrap, so correctness no longer depends on the engine's carry. verify-ctr.c --raw-only will also show what the engine actually does.
• GCM tests: verify-gcm.c adds context copy, in-place operation and rejected tag/ciphertext across the 1/2/3/4 KiB cuts.

I resolved those two threads. The native run of the new suites is still pending, as the README says.

The indexed-load change is correct. th.lrwu rd,tbl,idx,0 is the same zero-extended load as add + lwu, from the same table addresses, so the leakage profile is unchanged; it is still T-table AES. The measured +17–21% matches one instruction fewer per lookup.

The DMA cutoffs are mostly set by the transport's per-call cost (see #507). Concrete suggestions follow inline.

[dormancygrace]

Re-reviewed at 84de85e. No blocking issues.

• Separate patch: 0005-aes-c906.patch now holds the table-load change. I checked the fast path against upstream do_lookup: th.extu Ts,Qs,shamt+7,shamt + th.lrwu ...,2 is Table[(Q >> shamt) & 0xFF]. do_lookup_byte keeps the ×4 stride with th.lrbu. Td4 uses bytes [8i+7:8i] with stride 1. Register-ordering constraints are unchanged from upstream.
• cpp in the .S rename: the only two # if comments in aes-riscv64.pl (lines 816/824 of 3.5.9) are converted. No other comment line looks like a directive.
• continuation_minimum applies the session minimum only to the first chunk, as suggested. The GCM ctr32 path passes 2048/512, and verify-tails.c covers 4096+16…3072 tails, offsets and in-place for all three modes.
• Tail thread: accepted, see the reply there.

[dormancygrace]

Restack needed: the owner dropped the SHA-1 (#509), DES (#512) and TDES (#513) offloads, since nothing on the device uses them. #512 and #513 are closed.

1. Rebase this branch onto codex/openssl-sha256 without the SHA-1 commits, retarget the PR base to it, and remove the "OpenSSL SHA-1 partial DMA continuation" CI step. Then close crypto: offload OpenSSL SHA-1 compression to SG2002 #509.
2. crypto: optimize C906 AES and add measured SG2002 offload #511's qualification cites the shared 1944-case block-mode verifier in experimental/openssl-des/verify.c (fingerprint a37680a0…). Please move its AES cases into experimental/openssl-aes (new fingerprint) and point the CI step there.
3. ossl_sg2002_cipher can drop the DES/TDES block-size branch (block = … ? 8 : 16) and narrow the algorithm range check to AES (SM4 has no caller either).
4. crypto: accelerate OpenSSL GHASH on C906 #527 sits on this branch and needs the same rebase afterwards.

The environment fix that #512's 0006 carried is already in #507's 0002 (secure_getenv), so nothing else needs rescuing.

[dormancygrace]

Addressed the scope cleanup request in head 9e3de10228a9e15cb4276e6e39585a62872cd211.

• The branch targets SHA-256 #508 directly. The SHA-1 commits, patch, qualification directory and CI step are absent. GHASH #527 follows this cleaned AES parent.
• experimental/openssl-aes/verify.c preserves the 972 AES cases from the shared verifier, with independently pinned OpenSSL 3.5.5 fingerprint 019ff5988780130c. CI runs it under ASan/UBSan; it no longer requires the legacy provider.
• ossl_sg2002_cipher uses a fixed 16-byte block and accepts AES only. Non-AES IDs return before device access with output/IV unchanged. The legacy provider no longer links the DMA transport/helper.
• All 15 final-head checks pass. Local validation includes 92,880 AES continuation cases, 960 dense CBC tail/failure cases, 25 GCM wrap cases, seven non-AES rejection cases, three helper negative controls, and EVP block/CTR/GCM/tail checks. A clean C906/O3 build applies all patches with zero fuzz and retains the same 5808 public exports.

scope-cleanup-validation.json records the new build and host checks. The earlier native performance, HTTPS and SSH results retain their original source/binary identities; native runs were not repeated for this cleanup. AES assembly, provider algorithms and thresholds are unchanged.

[dormancygrace]

Reviewed 432a9b47, including the incorporated SHA-256 fixes. The AES patches remain unchanged: AES-only acceptance, CBC in-place chaining, 128-bit CTR versus GCM low-32-bit carry, partial failures and continuation limits remain correct. The CI conflict resolution retains both the new SHA tests and every AES test.

Local validation passes: all five patches apply to the pinned OpenSSL source with zero fuzz; real C906 SHA arithmetic/wrapper tests, 92,880 sanitizer continuation cases, 960 dense-tail cases, 25 GCM wrap cases, seven rejected non-AES IDs and three negative controls. Independent host verifiers pass 972 AES, 2160 GCM, 288 CTR and 20,736 tail/output/IV cases. All fast final-head CI checks pass. No new AES correctness finding.

[dormancygrace]

Reviewed the final head 1db3b623bd0c9a35660d37f59d27b000c239e115. The corrected SHA-256 parent and current main are integrated; the AES implementation is unchanged.

Rechecked CBC decrypt IV preservation, low-32-bit CTR splitting, GCM counter rollover, continuation thresholds, provider context copies and scalar indexed-load equivalence. Local tests pass 92880 continuation cases, 960 dense CBC-tail cases, 25 GCM wrap/fault cases and seven non-AES rejections, with three negative controls.

Physical-device verification of the identical AES patches in the final GHASH candidate passes 972 AES, 408 AEAD, 2160 GCM split/copy/in-place/rejected-tag, 288 CTR-carry and 20736 tail/IV cases in each of generic, vector-only and vector-plus-DMA modes. No legacy SHA-1/DES/TDES offload was reintroduced. Existing resolved findings remain addressed; no new correctness issue found.

All 19 final-head CI checks pass, including the firmware build.