Repository navigation
Fix detection on onnxruntime 1.27.0 / arm64 (KleidiAI batch-1 conv bug) - #91
Merged
Merged
Conversation
onnxruntime 1.27.0 on arm64 (e.g. Apple silicon) runs batch-1 convolutions through its KleidiAI kernels, which return wrong results. The speech-embedding model runs one window per call, so every score collapsed to ~0: the reference positive.wav scored 0.004 (0.98 on 1.20-1.26). Every session the package opens now sets mlas.disable_kleidiai=1 unless the caller set it; older versions accept and ignore it (checked 1.17-1.27). Adds tests/test_pretrained.py (the shipped hey_livekit model on the Rust/Swift reference clips, no torch needed) and runs it on macos-latest (arm64) in CI, since the x86 runner can't reproduce this.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On onnxruntime 1.27.0 on arm64 (Apple silicon), wake words are never detected: every score is ~0. The reference
positive.wav(the clip the Rust and Swift ports test against) scores 0.004 withexamples/resources/hey_livekit.onnx; on 1.20–1.26 it scores 0.98.1.27.0 is hard to avoid:
reachy-mini≥ 1.10, for example, pinsonnxruntime==1.27.0.Cause
onnxruntime 1.27.0 runs batch-1 convolutions on arm64 through its KleidiAI kernels, which return wrong results.
WakeWordModel.predictruns the speech-embedding model one 76-frame window per call (batch 1), so the embedding is garbage from the firstConvon. The mel spectrogram and the classifiers are unaffected.What I measured, on the embedding model with the positive clip's windows (max abs error against 1.26):
mlas.disable_kleidiai=1Graph optimization levels make no difference. A single standalone
Convreproduces it too (errors of 20–35 at batch 1). The Swift port batches its windows, which is why its tests pass on the same model.Fix
livekit.wakeword.session.session_options()returns the caller'sSessionOptions(or new defaults) withmlas.disable_kleidiai=1, unless the caller already set that entry. It is used by every session the package opens: mel, embedding, classifiers and eval. Older onnxruntime versions accept the entry and ignore it (checked 1.17.3, 1.20.1, 1.27.0).tests/test_pretrained.py: the shippedhey_livekitmodel ontests/fixtures/{positive,negative}.wav(copied from the Swift fixtures). It needs only numpy + onnxruntime, so it runs with--noconftest.inference-arm64job onmacos-latestruns that test, because the existing Ubuntu x86 job can't reproduce the bug.Tested (macOS arm64)
test_positive_clip_detectedfails (0.004).ruff check/ruff format --checkare clean on the touched files. (Two long lines ininference/model.pypredate this change.)The upstream bug belongs in an onnxruntime issue too. The standalone repro is a single
Conv,(1,1,76,32)input, 24 output channels, 3×3 kernel,pads=[0,1,0,1], on arm64 CPU.