Support for Qwen/Qwen3-Embedding-8B

[iilyak]

The new /v1/embeddings doesn't work correctly with Qwen/Qwen3-Embedding-8B

While experimenting with pmetal SDK I noticed that embeddings produced by the Qwen/Qwen3-Embedding-8B are not correct. I used something along these lines.

let hidden = model.forward_hidden(&ids, None)?;
let pooled = pooling::pool(&hidden, &mask, mode)?;

Logs:

[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 12 into shape (1,12,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 12 into shape (1,12,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 12 into shape (1,12,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.

I wanted to debug wether it is related to my code or pmetal implementation. So I decided to try talkin to the http endpoint.

1. Run with Qwen/Qwen3-Embedding-8B
   Command:

curl http://127.0.0.1:11436/v1/embeddings \
                                                -H "Content-Type: application/json" \
                                                -d '{
                                              "input": "Your text string here",
                                              "model": "qwen/Qwen3-Embedding-8B"
                                            }'
{"object":"list","data":[{"index":0,"object":"embedding","embedding":[0.0]}],"model":"qwen/Qwen3-Embedding-8B","usage":{"prompt_tokens":4,"total_tokens":4}}⏎

Logs:

pmetal serve --model qwen/Qwen3-Embedding-8B --port 11436 --max-seq-len 10000
2026-06-14T23:54:17.807772Z  INFO pmetal::commands::serve: Resolving model: qwen/Qwen3-Embedding-8B
2026-06-14T23:54:17.810629Z  INFO pmetal_hub::download: Model qwen/Qwen3-Embedding-8B found in cache: /Users/iilyak2/.cache/huggingface/hub/models--qwen--Qwen3-Embedding-8B/snapshots/1d8ad4ca9b3dd8059ad90a75d4983776a23d44af
2026-06-14T23:54:17.810713Z  INFO pmetal::commands::serve: Loading tokenizer...
2026-06-14T23:54:17.944139Z  INFO pmetal::commands::serve: Loading model from "/Users/iilyak2/.cache/huggingface/hub/models--qwen--Qwen3-Embedding-8B/snapshots/1d8ad4ca9b3dd8059ad90a75d4983776a23d44af"...
2026-06-14T23:54:18.124152Z  INFO pmetal::commands::serve: Model loaded successfully
2026-06-14T23:54:18.124389Z  INFO pmetal_data::chat_templates: Chat template: Qwen (from tokenizer_config.json, jinja attached)
2026-06-14T23:54:18.125667Z  INFO pmetal_serve::engine: Inference engine ready: model_id=qwen/Qwen3-Embedding-8B, stop_tokens=[151643, 151645]
2026-06-14T23:54:18.125755Z  INFO pmetal_serve::engine: Selected serving generation backend model=/Users/iilyak2/.cache/huggingface/hub/models--qwen--Qwen3-Embedding-8B/snapshots/1d8ad4ca9b3dd8059ad90a75d4983776a23d44af backend=Gpu ane_enabled=false ane_max_seq_len=1024 ane_real_time=false
2026-06-14T23:54:18.125990Z  INFO pmetal_serve::server: Starting PMetal inference server on 127.0.0.1:11436
2026-06-14T23:54:18.125997Z  INFO pmetal_serve::server: OpenAI-compatible API available at http://127.0.0.1:11436/v1
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
2026-06-14T23:56:27.147787Z  INFO pmetal_metal::context: Initialized Metal device: Apple M1 Max
2026-06-14T23:56:27.154771Z  INFO pmetal_metal::context: Loaded Metal library (606520 bytes, 103 kernels available)
2026-06-14T23:56:27.154983Z  INFO pmetal_metal::context: GPU: Apple7 Max (gen 14), 24 cores, 280 GB/s [MeasuredGpuCopy], NAX: false, ANE: 16 cores
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,32,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 4 into shape (1,4,8,128).
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [rms_norm]: [rms_norm] Input must have at least 1 dimension but got input with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [rope]: [rope] Input must have at least 3 dimensions but got input with 0 dimensions.
[pmetal-bridge] exception in [sdpa]: [scaled_dot_product_attention] input with shape () expected to be rank 4
[pmetal-bridge] exception in [transpose_axes]: [transpose] Recived 4 axes for array with 0 dimensions.
[pmetal-bridge] exception in [reshape]: [reshape] Cannot reshape array of size 1 into shape (1,4,4096).
[pmetal-bridge] exception in [matmul]: [matmul] Got 0 dimension input. Inputs must have at least one dimension.

2. Run with qwen/Qwen3.5-9B
   Command:

curl http://127.0.0.1:11436/v1/embeddings \
                                                -H "Content-Type: application/json" \
                                                -d '{
                                              "input": "Your text string here",
                                              "model": "qwen/Qwen3-0.6B"
                                            }'
{"object":"list","data":[{"index":0,"object":"embedding","embedding":[1.4957438,-0.24979623,1.4551576,0.1484651,0.0921154,0.19227195,-2.0051706,0.29821873,-0.119668916,...]}],"model":"qwen/Qwen3.5-9B","usage":{"prompt_tokens":4,"total_tokens":4}}

Logs:

pmetal serve --model qwen/Qwen3.5-9B --port 11436 --max-seq-len 10000
2026-06-15T00:00:04.335651Z  INFO pmetal::commands::serve: Resolving model: qwen/Qwen3.5-9B
2026-06-15T00:00:04.335931Z  INFO pmetal_hub::download: Model qwen/Qwen3.5-9B found in cache: /Users/iilyak2/.cache/huggingface/hub/models--qwen--Qwen3.5-9B/snapshots/c202236235762e1c871ad0ccb60c8ee5ba337b9a
2026-06-15T00:00:04.335950Z  INFO pmetal::commands::serve: Loading tokenizer...
2026-06-15T00:00:04.507439Z  INFO pmetal::commands::serve: Loading model from "/Users/iilyak2/.cache/huggingface/hub/models--qwen--Qwen3.5-9B/snapshots/c202236235762e1c871ad0ccb60c8ee5ba337b9a"...
2026-06-15T00:00:04.519677Z  INFO pmetal_models::loader: Qwen3Next weight loading: all 427 weights matched successfully
2026-06-15T00:00:29.928358Z  INFO pmetal::commands::serve: Model loaded successfully
2026-06-15T00:00:29.929832Z  INFO pmetal_data::chat_templates: Chat template: Qwen (from tokenizer_config.json, jinja attached)
2026-06-15T00:00:29.930467Z  INFO pmetal_serve::engine: Inference engine ready: model_id=qwen/Qwen3.5-9B, stop_tokens=[248044, 248046]
2026-06-15T00:00:29.930809Z  INFO pmetal_serve::engine: Selected serving generation backend model=/Users/iilyak2/.cache/huggingface/hub/models--qwen--Qwen3.5-9B/snapshots/c202236235762e1c871ad0ccb60c8ee5ba337b9a backend=Gpu ane_enabled=false ane_max_seq_len=1024 ane_real_time=false
2026-06-15T00:00:29.930865Z  INFO pmetal_serve::server: Starting PMetal inference server on 127.0.0.1:11436
2026-06-15T00:00:29.930872Z  INFO pmetal_serve::server: OpenAI-compatible API available at http://127.0.0.1:11436/v1
2026-06-15T00:00:35.420999Z  INFO pmetal_metal::context: Initialized Metal device: Apple M1 Max
2026-06-15T00:00:35.427229Z  INFO pmetal_metal::context: Loaded Metal library (606520 bytes, 103 kernels available)
2026-06-15T00:00:35.427582Z  INFO pmetal_metal::context: GPU: Apple7 Max (gen 14), 24 cores, 280 GB/s [MeasuredGpuCopy], NAX: false, ANE: 16 cores

[iilyak]

The problem is in the naming of layers weights

jq '.weight_map | to_entries[:5] | from_entries' ~/.cache/huggingface/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218/model.safetensors.index.json

{
  "lm_head.weight": "model-00005-of-00005.safetensors",
  "model.embed_tokens.weight": "model-00001-of-00005.safetensors",
  "model.layers.0.input_layernorm.weight": "model-00001-of-00005.safetensors",
  "model.layers.0.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
  "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00005.safetensors"
}

jq '.weight_map | to_entries[:5] | from_entries' ~/.cache/huggingface/hub/models--Qwen--Qwen3-Embedding-8B/snapshots/1d8ad4ca9b3dd8059ad90a75d4983776a23d44af/model.safetensors.index.json

{
  "embed_tokens.weight": "model-00001-of-00004.safetensors",
  "layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
  "layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
  "layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
  "layers.0.mlp.up_proj.weight": "model-00001-of-00004.safetensors"
}

I confirmed it with following patch

diff --git i/crates/pmetal-models/src/loader.rs w/crates/pmetal-models/src/loader.rs
index f9b907d..8437758 100644
--- i/crates/pmetal-models/src/loader.rs
+++ w/crates/pmetal-models/src/loader.rs
@@ -92,9 +92,12 @@ fn assign_loaded_weights<M: ModuleParameters + ModuleParametersExt>(
     loaded: HashMap<String, Array>,
 ) {
     let mut params = model.flatten_params_mut();
+    println!("Available model parameters: {:?}", params.keys().collect::<Vec<_>>());
     for (key, value) in loaded {
         if let Some(param) = params.get_mut(&key) {
             **param = value;
+        } else {
+            println!("Warning: Loaded weight '{}' not found in model parameters", key);
         }
     }
 }

Here is the output [truncated]

Available model parameters: ["model.layers.25.mlp.down_proj.weight", "model.layers.17.self_attn.k_proj.weight", "model.layers.18.self_attn.q_norm.weight", "model.layers.12.mlp.up_proj.weight", "model.layers.22.mlp.gate_proj.weight", "model.layers.14.post_attention_layernorm.weight", "model.layers.33.input_layernorm.weight", "model.layers.6.mlp.up_proj.weight", "model.layers.15.self_attn.v_proj.weight", "model.layers.31.post_attention_layernorm.weight", "model.layers.35.self_attn.q_proj.weight", "model.layers.9.self_attn.v_proj.weight", "model.layers.13.mlp.down_proj.weight", "model.layers.23.self_attn.k_norm.weight", "model.layers.5.mlp.up_proj.weight", "model.layers.7.mlp.down_proj.weight".....
....
Warning: Loaded weight 'layers.17.self_attn.q_proj.weight' not found in model parameters
Warning: Loaded weight 'layers.21.post_attention_layernorm.weight' not found in model parameters
Warning: Loaded weight 'layers.14.post_attention_layernorm.weight' not found in model parameters
Warning: Loaded weight 'layers.13.mlp.down_proj.weight' not found in model parameters
Warning: Loaded weight 'layers.21.self_attn.v_proj.weight' not found in model parameters
Warning: Loaded weight 'layers.21.self_attn.q_norm.weight' not found in model parameters
Warning: Loaded weight 'layers.15.input_layernorm.weight' not found in model parameters
Warning: Loaded weight 'layers.16.mlp.gate_proj.weight' not found in model parameters
...

[iilyak]

The following patch fixes the problem. I didn't open PR because I don't like having model specifics in generic weights loader. There might be a better place to plug this in.

diff --git i/crates/pmetal-models/src/loader.rs w/crates/pmetal-models/src/loader.rs
index f9b907d..c898c56 100644
--- i/crates/pmetal-models/src/loader.rs
+++ w/crates/pmetal-models/src/loader.rs
@@ -36,6 +36,8 @@ pub enum LoadError {
     SafeTensors(String),
     #[error("Missing weight: {0}")]
     MissingWeight(String),
+    #[error("Unexpected layer key: {0}")]
+    UnexpectedLayerKey(String),
     #[error("Shape mismatch for {key}: expected {expected:?}, got {actual:?}")]
     ShapeMismatch {
         key: String,
@@ -71,7 +73,13 @@ fn load_shard(path: &std::path::Path) -> Result<HashMap<String, Array>, LoadErro
         ))
     })?;
     pmetal_bridge::inline_array::load_safetensors_shard(path_str)
-        .map(|pairs| pairs.into_iter().collect::<HashMap<_, _>>())
+        .map(|pairs| pairs.into_iter().map(|(mut k, v)|{
+            // Qwen3-Embedding-specific hack: add "model." prefix if missing, since the `pmetal` expects it.
+            if !k.starts_with("model.") {
+                k.insert_str(0, "model.");
+            }
+            (k, v.into())
+        }).collect::<HashMap<_, _>>())
         .ok_or_else(|| LoadError::SafeTensors(format!("Failed to load safetensors: {}", path_str)))
 }
 
@@ -90,13 +98,16 @@ struct MlxQuantizationConfig {
 fn assign_loaded_weights<M: ModuleParameters + ModuleParametersExt>(
     model: &mut M,
     loaded: HashMap<String, Array>,
-) {
+) -> Result<(), LoadError> {
     let mut params = model.flatten_params_mut();
     for (key, value) in loaded {
         if let Some(param) = params.get_mut(&key) {
             **param = value;
+        } else {
+            return Err(LoadError::UnexpectedLayerKey(key));
         }
     }
+    Ok(())
 }
 
 fn eval_loaded_parameters<M: ModuleParameters + ModuleParametersExt>(
@@ -1014,7 +1025,7 @@ pub fn load_generic_weights<M: ModuleParameters + ModuleParametersExt>(
     let single_file = model_dir.join("model.safetensors");
     if single_file.exists() {
         let loaded = load_shard(&single_file)?;
-        assign_loaded_weights(model, loaded);
+        assign_loaded_weights(model, loaded)?;
         eval_loaded_parameters(model)?;
         return Ok(());
     }
@@ -1031,7 +1042,7 @@ pub fn load_generic_weights<M: ModuleParameters + ModuleParametersExt>(
     for shard_file in shard_files {
         let shard_path = validate_shard_path(model_dir, shard_file)?;
         let loaded = load_shard(&shard_path)?;
-        assign_loaded_weights(model, loaded);
+        assign_loaded_weights(model, loaded)?;
     }
     eval_loaded_parameters(model)?;
     Ok(())

With the above patch I am able to produce the same embeddings vector as with python based runner. Here is the example:

use pmetal::hub::resolve_model_path;

let model_path = resolve_model_path("Qwen/Qwen3-Embedding-8B", None, None).await?;
let tokenizer = Tokenizer::from_model_dir(&model_path)?;
let mut model = DynamicModel::load(&model_path)?;
let tokens = tokenizer.encode_with_special_tokens("Value: Virtuous").map_err(|e| e.to_string())?;
let input_ids = Array::from_slice(&tokens.iter().map(|&t| t as i32).collect::<Vec<i32>>(), &[1, tokens.len() as i32]);
let hidden = model.forward_hidden(&input_ids, None)?;
let mask = Array::from_slice(&vec![1.0f32; tokens.len()], &[1, tokens.len() as i32]);
let pooled = pooling::pool(&hidden, &mask, pooling::PoolingMode::LastToken)?;
let normalized = pooling::normalize_embeddings(&pooled).map_err(|e| e.to_string())?;
normalized.try_eval().map_err(|e|e.to_string())?;
let out = normalized.as_slice::<f32>().to_vec();
println!("Embedding for 'Value: Virtuous': {:?}", out);

[nicholasjpaterno]

@iilyak thanks for opening this issue and pushing it forward. If you'd like to PR this solution I will gladly merge it! I want to make sure you get credit for your contributions!

[nicholasjpaterno]

@iilyak Just taking a closer look, load_shard is the wrong layer for the following reasons:

1. It's the byte-level shard reader with zero model context. A blanket model. prepend there corrupts keys that are correctly top-level — e.g. lm_head.weight would become model.lm_head.weight (wrong for untied-embedding
   checkpoints) — and breaks architectures that don't use the model. namespace at all (BERT uses bert.encoder.… and already has its own load_bert_weights remapper).
2. It can't know the target namespace, so any transform there is a guess.

Better home: assign_loaded_weights / load_generic_weights (loader.rs:88 / :1009) — the one layer that holds both the model and the loaded tensors, so it can reconcile against the model's actual param namespace instead
of hardcoding model.. Two parts:

(a) Make unmatched keys loud — this is the higher-value half of your patch. Today a fully-misnamed checkpoint "loads" successfully into an all-random model (exactly what bit you). pmetal already has a LoadReport {
loaded, skipped } pattern (used by the gemma4 / diffusion_gemma loaders); have load_generic_weights produce one and error when loaded == 0 (or when a large fraction of model params stayed unassigned).

(b) Reconcile the prefix from the model, not a hardcoded string — so you don't bake model specifics into the loader (your stated concern):

fn assign_loaded_weights(model, loaded) -> LoadReport {
    let mut params = model.flatten_params_mut();
    let mut report = LoadReport::default();
    for (key, value) in loaded {
        if let Some(p) = params.get_mut(&key) {
            **p = value; report.loaded += 1;
        } else if let Some(p) = params.get_mut(&format!("model.{key}")) {
            **p = value; report.loaded += 1;        // Qwen3-Embedding: `layers.…` ↔ `model.layers.…`
        } else {
            report.skipped.push(key);
        }
    }
    report
}

This leaves correctly-named keys untouched, fixes the embedding-style layers.… checkpoints generically, and surfaces mismatches.

[nicholasjpaterno]

Happy to make those changes if you're busy or not up for it. Otherwise I would be more than happy to merge your PR so your contributions are noted in the commit history!

[iilyak]

@nicholasjpaterno I created a PR #20, could you take a look?

[nicholasjpaterno]

@iilyak Thanks for the PR, taking a look now!