Skip to content

单机8块P800运行Qwen3-235B-A22B总是loadlayers开始的时候崩溃, xpu_ipc_get_memhandle failed 。 #8102

Description

@zziawan

INFO 2026-07-16 14:43:22,869 1748 engine.py[line:151] Waiting for worker processes to be ready...
Loading Weights: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [01:43<00:00, 1.04s/it]
Loading Layers: 0%| | 0/100 [00:11<?, ?it/s]
ERROR 2026-07-16 14:45:22,612 1748 engine.py[line:160] Failed to launch worker processes, check log/workerlog.* for more details.

ERROR 2026-07-16 14:45:31,391 1748 engine.py[line:447] Error extracting sub services: [Errno 3] No such process, Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/engine/engine.py", line 444, in _exit_sub_services
pgid = os.getpgid(self.worker_proc.pid) Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/engine/../worker/worker_process.py", line 1296, in
run_worker_proc()
File "/usr/local/lib/python3.10/dist-packages/paddle/base/dygraph/base.py", line 406, in _decorate_function
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/engine/../worker/worker_process.py", line 1281, in run_worker_proc
worker_proc.initialize_kv_cache()
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/engine/../worker/worker_process.py", line 659, in initialize_kv_cache
available_kv_cache_memory = self.worker.determine_available_memory()
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/worker/xpu_worker.py", line 130, in determine_available_memory
self.model_runner.profile_run()
File "/usr/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/worker/xpu_model_runner.py", line 1711, in profile_run
self.initialize_kv_cache(profile=True)
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/worker/xpu_model_runner.py", line 1270, in initialize_kv_cache
set_data_ipc(key_cache, key_cache_name)
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/model_executor/ops/xpu/fastdeploy_ops/init.py", line 2000, in set_data_ipc
outs = _C_ops._run_custom_op("set_data_ipc", tmp_input,shm_name)
OSError: xpu_ipc_get_memhandle failed

使用的是fastdeploy 2.5镜像,启动参数及模型启动参数均参考了文档的参数。请问有没有哪位碰到过类似问题。

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions