Removal of CLBlast backend - Discussion

[LostRuins]

As of v1.104, the CLBlast backend may be removed soon, as it is very outdated and no longer receives and updates, fixes or improvements. It can be considered superceded by the Vulkan backend.

Put any concerns, comments or suggestions here.

[jabberjabberjabber]

What are the downsides of keeping it?

[LostRuins]

It's mainly bloat at this point to keep a big chunk of code that very few people use. The main downsides are maintainability and file size, especially since people are request more vulkan variants

[alan-l]

If either you or ggernanov can fix the problem I'm having with vulkan wrt the out of memory allocation issues, CLBlast wouldn't be needed. But for now, even if CLBlast is slower than Vulkan, it doesn't crap out with that error. That's the only reason why I need to use it.

[alan-l]

Being GPU-poor and VRAM-limited, there's some models I cannot run if I don't use CLBlast option. I've tried it on an old NVidia card and an Intel Iris Xe iGPU setup and if I use Vulkan, I get a "Device memory allocation of size ... failed" error. With CLBlast (instead of Vulkan or CUDA), I'm able to run these models successfully.

[alan-l]

So running the above config on the Xe, if you simply change the Use CLBlast to "Use CPU" (not with oldpc), its instantly slower by 2x?

I ran the CPU mode twice. Here's my results:

---

Prompt: "Gandalf the Wizard casting a fireball spell at a Kobold."
Steps: 4
Width: 512
Height: 512
Guidance: 1

KoboldCPP.exe v1.104 - CLBlast
Step times: 101.51, 93.31, 92.78, 92.78

KoboldCPP.exe v1.104 - CPU
Step times: 258.05, 247.04, 239.99, 237.75
Step times: 267.43, 227.25, 207.66, 195.71

---

Let me know what else you need! Really appreciate the great work you're doing for the community!

[MerijnHendriks]

Heyo! Quick question @alan-l ;
which driver version are you running, and have you tried updating them using intel's official drivers?
https://www.intel.com/content/www/us/en/search.html#sortCriteria=@lastmodifieddt%20descending&cf-tabfilter=Downloads&cf-downloadsppth=Graphics

Reason I ask is that updating drivers for my N5000 (Intel UHD 605) helped resolve issues I had with vulkan on koboldcpp-olderpc-nocuda.exe.
The windows default installed driver only supports Vulkan 1.2, drivers released in 2024 support Vulkan 1.3 and is to run Llama.cpp.
Can't speak for image generation, my laptop is far too weak for that.

[MerijnHendriks]

Another thing to add: CLBlast seems to handle (V)RAM usage differently than when using Vulkan;

• CLBlast seems to view iGPU VRAM as part of system RAM and only uses Compute of your iGPU.
• Vulkan only accepts it when everything fits into iGPU Shared VRAM (see Windows Task Manager for the actual size).

You might want to use no kv offload to move your context into RAM. On an iGPU doing this doesn't matter much for inference speed but it does for lowering iGPU VRAM usage. Other options include reducing batch size and moving embed/mmproj models to CPU.

Let me know if any of this helps ya, or if I can help out in other ways!

[LostRuins]

The current clblast backend is a hack, it doesn't have proper offloading, the weights are still stored in ram, only copied onto GPU (duplicated), that's why I want to remove it

[alan-l]

Heyo! Quick question @alan-l ; which driver version are you running, and have you tried updating them using intel's official drivers? https://www.intel.com/content/www/us/en/search.html#sortCriteria=@lastmodifieddt%20descending&cf-tabfilter=Downloads&cf-downloadsppth=Graphics

Reason I ask is that updating drivers for my N5000 (Intel UHD 605) helped resolve issues I had with vulkan on koboldcpp-olderpc-nocuda.exe. The windows default installed driver only supports Vulkan 1.2, drivers released in 2024 support Vulkan 1.3 and is to run Llama.cpp. Can't speak for image generation, my laptop is far too weak for that.

Hi @MerijnHendriks! My drivers are the latest version and the GPU is an Iris Xe.

I'm going to try your recommendations again, but from past experience, with or without your recommendations, I get those error under Vulkan, thus needing the change over to CLBlast to avoid OOM errors. Without CLBlast option, well, you can see how much of a slow-down I incur...

[Luro223]

@LostRuins I personally have major boost with CLBlast, even better compared to VULKAN; great boost for legacy devices:

1.104 test results:

VULKAN/GT-740m/i7-4500U

CLBlast/GT-740m/i7-4500U

[LostRuins]

hmm you are using a different batch size 1024 vs 256. Different layer count too. but similar speeds actually.

[LostRuins]

Notice: With the addition of the older pc support for vulkan in 1.107 oldpc, clblast has now been removed. If you wish to use it, you will need to use 1.106.2 or earlier.