Skip to content

PERFORMANCE.md

Latest commit

 

History

History
245 lines (219 loc) · 13.9 KB

PERFORMANCE.md

File metadata and controls

245 lines (219 loc) · 13.9 KB
Raw

LensLink — Performance Notes

Where the cycles and joules actually go, what has been optimized, and how to measure before optimizing further. The guiding rule: the codec hardware and the network are the floor — the app/plugin code around them should add as close to zero as possible.

Where the cost is (by design)

Stage Where Cost Avoidable?
Camera capture iPhone ISP/sensor fixed No — it's the product
H.264/HEVC encode iPhone media engine (VideoToolbox) fixed No (hardware block)
Send to socket iPhone CPU (memcpy + syscall) ~1 copy/frame Minimized (see below)
Receive + parse OBS-box CPU ~0 copies Already zero-copy into the decoder
H.264/HEVC decode OBS-box GPU (D3D11VA/VideoToolbox/VAAPI) fixed Software fallback only when GPU fails
GPU→CPU frame download OBS-box (av_hwframe_transfer_data) 1 copy/frame No — obs_source_output_video needs system memory
OBS async frame ingest OBS-box (obs_source_output_video) 1 copy/frame No — OBS API copies internally

What the code already does (don't regress these)

iOS app

  • Capture delivers NV12 video-range buffers (420YpCbCr8BiPlanarVideoRange) straight into VideoToolbox — no BGRA, no format conversion, and alwaysDiscardsLateVideoFrames sheds load instead of queueing it.
  • Encoder is tuned for live use: real-time mode, no frame reordering, PrioritizeEncodingSpeedOverQuality, MaxFrameDelayCount = 1.
  • One copy per frame out of the encoder (AVCC→Annex B into an owned Data, sized up-front with reserveCapacity). This copy is deliberate: wrapping VideoToolbox's block buffer zero-copy would pin encoder-pool buffers until TCP send completion — up to 12 in flight — and starve the encoder under congestion.
  • Header + payload are sent as two batched writes (NWConnection.batch), so the frame payload is not memcpy'd a second time just to gain a 20-byte header prefix (StreamClient.sendVideoOnQueue).
  • Backpressure drops frames rather than queueing them (bounded memory, bounded latency), and adaptive bitrate reacts to send-delay/drops.
  • Dimmed = GPU idle: the dim overlay disables the preview layer's connection, so no preview frames are rendered under the near-black cover (CameraPreviewView.previewEnabled). The outgoing stream is unaffected. On top of the existing OLED-black + brightness drop, this is the biggest saver for the "phone mounted behind the monitor" case.
  • Idle standby listener costs nothing measurable: no timers, no camera — just an accepting socket and 1 Hz timesync replies while OBS is connected.
  • The health overlay's @Published sample is only written while the overlay is on screen (off by default). A 1 Hz publish on the main actor invalidates the whole Live view once a second, which is exactly the wakeup "dimmed = GPU idle" is trying to avoid; the counters advance regardless, so switching it on still reads correctly a second later.

OBS plugin

  • The receive buffer is parsed in place; video packets go to libavcodec via pointer (pkt->data = recv_buffer + offset) — zero copies from socket to decoder. Compaction is one memmove of the unparsed tail per recv cycle (amortized, not per packet), and the buffer shrinks after keyframe spikes so memory isn't pinned.
  • Decode is single-threaded with LOW_DELAY on purpose: frame threading adds a frame of latency per thread and buys nothing for a live stream that arrives one access unit at a time.
  • One dial-loop thread per source, poll-driven at 200 ms when idle; timesync (1 Hz), control forwarding, and diagnostics all piggyback on that loop — no extra threads or timers.
  • The web panel is a 1 Hz poll of two tiny JSON endpoints on loopback.
  • Lip-sync cross-correlation runs on the dial-loop thread, so its cost is a hole in the video receive path, not spare CPU. Two things keep it small (lipsync.c): the mic window's energy comes from a prefix sum (successive lags overlap almost completely — recomputing it per lag was a second pass over the whole window), and the dot product uses four independent accumulators, because one double accumulator serializes the loop on FP-add latency regardless of how little else it does. Together: 1.51 ms → 0.90 ms per estimate, bit-identical results.
  • It also runs far less often. What it measures — the mic's latency — is a property of the audio gear, while the number that actually moves is the video latency timesync already tracks for free. So the mic figure is latched once confident and the offset is re-derived from latency alone; the correlation returns only every 90 s to confirm the latched figure still holds (lipsync-cal.h). Over a quiet ten-minute stretch that is 6 correlations instead of 120 — and, more importantly, 120 offset updates instead of none, since tracking no longer waits on someone talking.
  • Once locked, the plugin also tells the phone to stop the reference entirely ({"cmd":"reference","on":false}): a live mic capture and ~256 kbit/s that teach the correlation nothing more, gone — including iOS's persistent mic indicator. It's requested back briefly around each periodic verification, and while measuring/relocking.
  • Effect parameter handles are resolved once, with the effect, not per render (yuv_effect()). video_render runs per source per rendered frame on the graphics thread — the thread the whole compositor waits on — and gs_effect_get_param_by_name is a by-name lookup.

How to measure (before optimizing further)

  • iPhone: Instruments → Energy Log + Time Profiler while streaming 1080p60 and 4K30, once undimmed and once dimmed. The app's own CPU should be single-digit percent; the energy split should be dominated by camera + display (undimmed) and camera only (dimmed).
  • OBS box: OBS Studio → Tools → Stats (render/encode lag) plus the plugin's own log line (capture->decode latency: avg …) and, for CPU, a profiler over obs64/obs filtered to the ios-camera-server thread. That thread should be nearly all recv/memmove/decoder time.
  • The wire protocol's TIMESYNC gives an end-to-end capture→decode latency figure continuously — watch it in the source's Status field; ~60 ms over USB at 1080p is the expected baseline.

Pipeline benchmark (built-in, for the GPU-pipeline comparison)

Tools → LensLink Settings → "Log pipeline benchmark numbers every 5 s". Every ~5 s of live streaming, one tagged line lands in the OBS log:

[lenslink][bench] pipeline=standard | 1920x1080 ~60 fps |
  video-path cost/frame: avg 2.84 ms, max 4.10 ms |
  pixel copies: 186.4 MB/s | OBS process CPU: 9.8%
  • video-path cost/frame — CPU time the plugin spends moving each decoded frame toward the compositor: GPU download + OBS's frame copy on the standard pipeline; texture map/draw prep on the GPU pipeline. This is the work the two pipelines do differently, isolated.
  • pixel copies — decoded video crossing system memory (0 on a healthy GPU-pipeline run; nonzero there means the fallback engaged).
  • OBS process CPU — sampled the same way OBS's own Stats dock does.

While the toggle is on, the plugin also writes one CSV row per second of live video to bench-<pipeline>-<epoch>.csv in its config directory (the OBS log prints the exact path when the file opens).

The before/after recipe:

  1. Keep your normal settings, enable the benchmark toggle, and stream the scene for ~10 s or more. That's the before file.
  2. Flip the GPU pipeline setting, restart OBS, and stream the same phone/scene/resolution for the same length. That's the after file.
  3. Run python3 tools/bench-report.py in a terminal/command prompt. With no arguments it lists the benchmark files it finds in the plugin's config directory and asks you to pick the before and after runs (or pass the two CSV paths directly).
  4. It prints and writes lenslink-bench-report.md (the comparison table — mean/median/p95 per metric with % change) and lenslink-bench-report.html (the same plus per-second charts), ready to paste into the repo.

Sanity check built into the report: "pixels crossing system memory" must read 0.00 on the GPU-pipeline run — a nonzero value there means the automatic CPU fallback engaged and the comparison isn't measuring what you think. Those report numbers are what a performance claim in this repo should cite.

Any future performance PR should quote at least one of these numbers before/after.

Measured results: standard vs GPU pipeline

Field measurements from the benchmark above — 12 configurations (720p/1080p/4K × 30/60 fps × H.264/HEVC), ~25 s of live video each, same PC (Ryzen 7 7800X3D, NVIDIA, D3D11) and same iPhone over Wi-Fi. Pixel copies read 0.00 MB/s on every GPU-pipeline run: the zero-copy path engaged in all 12 configurations.

Config Cost/frame, mean (ms) Pixel copies (MB/s) OBS CPU (%)
720p 30 H.264 1.14 → 0.09 (−92%) 82 → 0 1.7 → 1.4
720p 30 HEVC 0.81 → 0.09 (−89%) 83 → 0 2.3 → 1.5
720p 60 H.264 1.09 → 0.10 (−91%) 166 → 0 2.1 → 1.4
720p 60 HEVC 0.81 → 0.09 (−89%) 163 → 0 2.2 → 1.4
1080p 30 H.264 2.13 → 0.11 (−95%) 187 → 0 1.7 → 1.6
1080p 30 HEVC 1.48 → 0.10 (−93%) 183 → 0 2.3 → 1.4
1080p 60 H.264 2.11 → 0.10 (−95%) 370 → 0 2.3 → 1.6
1080p 60 HEVC 1.42 → 0.09 (−93%) 373 → 0 2.6 → 1.4
4K 30 H.264 7.71 → 0.10 (−99%) 734 → 0 3.3 → 1.5
4K 30 HEVC 4.96 → 0.14 (−97%) 721 → 0 2.9 → 1.5
4K 60 H.264 7.54 → 0.10 (−99%) 1221 → 0 4.1 → 1.8
4K 60 HEVC 4.75 → 0.10 (−98%) 1439 → 0 4.1 → 1.6

Reading: per-frame video-path cost drops 89–99% (the copy work simply disappears), and the win scales with resolution — at 4K60 HEVC the standard pipeline was pushing ~1.4 GB/s of decoded video through system memory that the GPU pipeline eliminates outright, roughly halving OBS process CPU. Capture→decode latency and decoded fps are measured before the pipelines diverge and showed no systematic difference — those are set by the network and the phone's encoder, not by the render path.

The table above is 8-bit. 10-bit streams (HLG / Apple Log) gained the same zero-copy path (D3D11 shared P010, VAAPI R16/GR1616 dmabuf) a release later; the sanity check applies unchanged — pixel copies must read 0.00 with the GPU pipeline on, and at 10 bits the eliminated copy traffic is 2× the 8-bit figures. A before/after bench pair for a 10-bit config still needs to be captured and pasted here (macOS is excluded: VideoToolbox converts 10-bit to 8-bit BGRA itself, so HDR there renders via the RGBA path).

Apple capture guidance: what we follow, what we deliberately don't

The iOS capture/encode path was audited against Apple's AVFoundation and VideoToolbox guidance for real-time capture. Followed: RealTime + AllowFrameReordering=false (no B-frames) + PrioritizeEncodingSpeed OverQuality + MaxFrameDelayCount=1 on the encoder, average bitrate with a hard data-rate cap, 2-second keyframes, alwaysDiscardsLate VideoFrames, session interruption/runtime-error observers with restart, and — per the systemPressureState docs — thermal/power mitigation: at .serious the bitrate is halved, at .critical it's quartered and the frame rate halves (60→30, 30→15), restored when pressure abates.

Deliberate divergences (don't "fix" these without reading this):

  • Frame durations are locked (min = max = 1/fps). Apple's default lets auto-exposure sag the frame rate in low light, like the Camera app's Auto FPS. A sagging cadence hurts the encoder's rate control, OBS timing, and the lip-sync/latency math; we hold cadence and let the image get noisier instead. Two sanctioned exceptions: the thermal throttle above, and the opt-in Allow system video effects toggle (Options), which leaves the max duration at the format default — the Control Center effects appear to require that downward flexibility, and the toggle exists to prove or disprove exactly that. The thermal path keeps max unlocked too while the toggle is on.
  • Video stabilization stays off (the AVCaptureVideoDataOutput default). Every stabilization mode adds frames of latency; this is a latency-first product. Revisit only as an opt-in.
  • The wire defaults to 8-bit 4:2:0 video-range (420v). The opt-in colour modes switch the camera path to 10-bit end-to-end — HLG captures x420, Apple Log captures x422 (its formats are the 10-bit-422 class; VideoToolbox does the 4:2:0 downsample inside the Main10 encode) — but 10-bit only ever enters the pipeline through that setting. SDR capture, the screen mirror, and the H.264 path stay 8-bit; keep the 8-bit paths free of 10-bit branches (the decoder maps formats per-frame, so SDR costs nothing extra).
  • Rotation is sensor-native (.landscapeRight, an effective 0°). The AVCaptureConnection docs warn per-frame rotation costs; we never rotate the stream, only the on-phone preview.
  • Green screen OFF is free, ON pays exactly once. With the toggle off, the capture→encode path is the pre-feature code verbatim — no compositor branch on the hot path, no depth output attached. On, the budget is: Vision person segmentation per frame (.fast above 30 fps, .balanced at 30 — the dominant cost), one Metal compute pass writing green into a pool buffer (that pass is the single copy; the camera's own buffer is never written in place — it races the encoder's async read), and depth at ~15 Hz when assist is active. Depth delivery deactivates Center Stage and the system video effects by iOS policy, and overload sheds frames via the existing drop-don't-queue capture behaviour. On-device bench + thermal-soak numbers for green-screen-ON are still to be captured and pasted here; the OFF run must show the zero-cost claim holds.