roi_video_encoder

Re-encodes a video, giving extra bits (lower QP) to a detected region of interest (e.g. a person) via a per-frame ROI attached directly to each frame before libx264 encoding.

Two ROI detection modes are available via --roi-mode:

• pixel (default) - runs the ROI detector(s) on every decoded frame. Slower, but with no tracking drift between detections.
• compressed - only detects on I-frames; boxes are carried forward on the P/B-frames in between using the motion vectors the decoder already computes (via av's export_mvs), instead of re-running detection on every single frame. Much faster, at the cost of translate-only tracking accuracy between I-frames.

Setup

Requires Python 3.12, gcc, and curl (the last two only for the one-time native build below).

Install the Python dependencies:

pip install -r requirements.txt

If your server does not have GPU installed, change "inference==1.3.8" to "inference-core==1.3.8" in requirements.txt.

Build the native extension (only needed once per machine, or again after upgrading the av package - native/build.sh downloads matching FFmpeg headers for whatever av version is installed, no sudo required):

native/build.sh

If you want text detection (--roi text, see below), download the EAST model once (~92MB, not checked into the repo):

models/download_east.sh

Run

python3 roi_encoder.py <input_video> -o <output_video> --roi person --conf 0.4 --crf 40 --qoffset=-0.5

• <input_video> - path to the input video file (required, positional).
• -o, --output - path to write the output video (default: output.mp4).
• --roi - comma-separated ROI specifier(s), e.g. person or person,dog,car (default: person). Each must be either a COCO class the detector model knows, or the reserved word text to also detect on-screen text via OpenCV's EAST detector (requires models/download_east.sh to have been run), e.g. person,text. Every matching detection in a frame is boosted simultaneously - e.g. person,text boosts every detected person AND every detected text region in the same frame, each with the same --qoffset - not just the single largest match.
• --conf - detector confidence threshold (default: 0.4).
• --roi-mode - pixel (default) detects on every frame; compressed detects only on I-frames and propagates boxes across the following P/B-frames via motion vectors, trading tracking accuracy for speed.
• --crf - libx264 Constant Rate Factor for the overall encode; lower is higher quality/bitrate (default: 43).
• --qoffset - QP offset applied inside the ROI, e.g. -0.5 or -1/2; negative values improve quality inside the ROI. Use --qoffset=<value> (with =) since a leading - is otherwise misread as another flag.

Example:

python3 roi_encoder.py cooking.mp4 -o out.mp4 --roi person --conf 0.4 --crf 40 --qoffset=-0.5