Does not detect improper HWA setup on Linux for GPUs

[felix920506]

Classifications

• Running python ./jellybench_py/hwi.py manually (within the Poetry shell) works and produces correct results.
• This happens on a Virtual Hardware (VM, WSL, ...)
• The issue occurs during the benchmark section of the script (after Starting Benchmark...).
• A server issue or faulty server response is causing the problem.
• jellybench was run in --debug mode (recommended for issue reports).
• I am using the latest version (branch develop) of jellybench. I understand older versions are not supported.

Description

When running on Linux, the following things might happen:

• The system does not have the correct GPU firmware installed
• The system does not have a new enough version of the kernel, mesa etc.
• The user running the benchmark tool might not have permission to use the GPU

In many of these cases, the program can still detect the presence of the GPUs, but will fail to run benchmarks on them.
The program should run a test before starting benchmarks to detect if things are setup properly.
There are commands in the Jellyfin HWA docs on what ffmpeg commands can be used to test for such cases.

Error Code or Output

Steps to Reproduce

1. Start Jellybench on a Linux system with improper hardware acceleration setup (eg. user without permissions to use GPU)
2. Run benchmarks
3. No errors are reported, benchmarks silently fail

Operating System

Linux

System Architecture

x86_64 (64-bit)

Python Version

3.12

CPU[s]

Intel Core i5-10400F

GPU[s]

Intel Arc A750

Additional Context

No response

Possible Solution

1. Run a test command to test if HWA was actually properly setup (commands can be found in Jellyfin HWA docs)
2. Print out a message every time a test fails

[BotBlake]

I dont understand how we are supposed to follow the test from the Jellyfin Documentation.
My Idea would be to test if we can run at least one ffmpeg worker on the selected GPU (simmilar to how nvenc detection works).
If THAT doesnt work then we know the device will not be abled to handle anything else.

The HWA documentation does not always specifc ffmpeg commands to test for a correct setup.
For Nvidia on Windows it is

Play a video in the Jellyfin web client and trigger a video transcoding by setting a lower resolution or bitrate.

Which we certainly cannot do 😆

On linux we are just kind of checking if we can access nv-smi and what it says when we transcode.

The situation seems rather complex and a bit out of scope.
I'd say anything but verifying if we can make use of the GPU at all (by literally trying to make use of it) is out of scope.

[felix920506]

Why would that be out of scope? I don't think test running a single stream at the beginning of the benchmark and see if it fails is out of scope.

[BotBlake]

What I said is that everything more complex then just "trying" and saying "hmm, somehing seems to be broken" would be out of scope.

That was in reply to your Idea to follow the Jellyfin HWA docs which is not realy possible.
NVIDIA Cards already have a test command ready, for CPU, AMD and Intel I will have to cook up something.

[BotBlake]

I am working on refractoring compatibility checks into a seperate file.
That way compatibility tests can be done individualy and easier.

It will be responsible for NVenv Limit checks AND the test commands for selected devices.