Priority inversion: userInitiated demux consumer blocks on the utility read-ahead producer

[a1go3]

Summary

Thread Performance Checker reports a priority inversion on the software playback path: the demux consumer (.userInitiated) blocks on an NSCondition waiting for the read-ahead producer (.utility) to deliver packets.

Thread running at User-initiated quality-of-service class waiting on a lower QoS
thread running at Utility quality-of-service class. Investigate ways to avoid
priority inversions
  Sources/AetherEngine/Native/SoftwarePacketReadAhead.swift:231

The two sides

Producer — SoftwarePacketReadAhead.swift:38:

private let worker = DispatchQueue(label: "engine.sw.packet-prefetch", qos: .utility)

Consumer — SoftwarePlaybackHost.swift:120, which calls readAhead.read(...) at SoftwarePlaybackHost.swift:2336:

private let demuxQueue = DispatchQueue(label: "engine.sw.demux", qos: .userInitiated)

The block is the wait loop at SoftwarePacketReadAhead.swift:231:

while count == 0, !ended, failure == nil, !closed, !seeking,
      token == generation, isCurrent() {
    condition.wait()
}

NSCondition does not donate the waiter's priority, so whenever the FIFO runs dry the .userInitiated consumer is gated on a .utility producer with no priority boost.

When it shows up

Reproducible whenever reads are slow enough for the buffer to empty — for us that is 1080p MKV streamed over WebDAV, where the producer is network-bound rather than CPU-bound. It fires around startup and after seeks, i.e. exactly when the buffer is empty by construction.

The prefetch queue is genuinely elective while the FIFO has depth, so .utility is the right resting state; the inversion is only about the window where the consumer is actually blocked on it.

Possible directions

Whichever fits the design best:

1. Raise the read-ahead worker to .userInitiated — simplest, at the cost of the elective-work property when the FIFO is full.
2. Keep .utility at rest and boost only while a consumer is blocked: dispatch produce() as a DispatchWorkItem and have the blocked consumer wait() on it, which propagates the waiter's QoS, instead of waiting on the bare condition.
3. Re-dispatch the pending produce() at the consumer's QoS when the wait loop is about to be entered with count == 0.

Option 2 keeps the intent of #27 (the elective decode must yield to real-time playback) while removing the inversion in the one window where the work is not elective any more.

Environment

• AetherEngine 6.76.0
• iPhone 13 Pro / iOS 26.5, Debug build with Thread Performance Checker enabled
• Software path, remote source over HTTP (WebDAV), MKV / H.264

[superuser404notfound]

Fixed and released in 6.79.0 (00da761c). Thank you for the report, and for locating both sides of it and the wait loop rather than only the Thread Performance Checker line.

Your reading of the resting state was right and is what shipped: .utility while the FIFO has depth, responsive only in the window where the work is not elective any more. Option 2 was not what it turned into, for a reason worth writing down.

What the inversion actually costs here is throughput, not fairness, and it needs no contention for cycles. Measured on an idle 8-core M1 with the consumer's decode cost made deliberately negligible (360p content, seven cores free, the process itself under 30 % of one core), so nothing was competing for a scheduler slot at all:

producer	pulled in 60 s of an 80 Mbit/s source
.utility (before)	147 MB
responsive	438 MB
.utility with only the thread's disk-I/O policy lifted	528 MB

That third arm is the one that names the cause. setiopolicy_np(IOPOL_TYPE_DISK, IOPOL_SCOPE_THREAD, IOPOL_IMPORTANT) changes exactly one of the two things that travel with a QoS class, so the class was throttling the spool writes this producer makes for every packet, not its cores. On a 1080p 8.2 Mbit/s source capped at 10 Mbit/s the same arms pulled 60.7 MB against 77.5 MB with the consumer blocked 60.2 s against 27.3 s of a 61 s session; with the box saturated by eight software decodes the demoted arm fell to 35.4 MB. A healthy link is unaffected, because 8.2 Mbit/s fits under the throttled ceiling of about 19 Mbit/s on this machine, which is consistent with your case being a network-bound WebDAV source: the throttle is a ceiling, so it only shows when the link could have gone faster.

That also rules option 1 out on measurement rather than on taste. Over a 100 s steady state the adaptive producer keeps 2165 ms of efficiency-class CPU against 2145 ms for the demoted one and 1 ms for a permanently responsive one, at identical total CPU, so raising it flat throws the whole elective-residency property away for nothing.

What shipped: the producer owns its thread and moves its own class, following the shape AE#286 already uses for the HLS pump. Responsive at start, after every seek, while any consumer is parked in a read, and below a quarter of the forward window; elective again above half of it. The two depths differ so a source sitting on one threshold does not retune once per packet.

One deliberate difference from your option 3: the decision hangs on the reservoir depth, not on the presence of a blocked waiter. A gate that waits for a blocked consumer only reacts once someone is already waiting, which is one buffer too late for the case you describe, where the buffer is empty by construction at startup and after a seek.

Consumer starvation is a diagnostic line of its own now, rate-limited to one a second, so "the source cannot keep up" is readable instead of inferred:

[SoftwarePacketReadAhead] consumer starved: waits= blocked= reservoir= qos=

Also in 6.79.0, unrelated to this but on the same path: the read-ahead now reports its QoS transitions, and it reads the class back after setting it, because a thread that was opted out of the QoS system keeps the old one silently and the whole mechanism then looks configured while doing nothing.