gst-nvmm-cpp

GStreamer plugin suite for NVMM-native video processing on NVIDIA Jetson platforms. Wraps the Tegra-native NvBufSurface / NVMM memory model in proper GStreamer elements, enabling hardware-accelerated crop, scale, format conversion, and inter-process video sharing with no CPU copies on the data path. Intra-process transforms are zero-copy (VIC-side); cross-process IPC does a single GPU-side copy into a shared pool, which consumers then import without further copies.

Building

Prerequisites

• GStreamer >= 1.16 development libraries
• CMake >= 3.16 or Either Meson >= 0.62 + Ninja
• C++14 compiler (GCC 7+ or Clang 5+)
• On Jetson: JetPack 5 (L4T 35.x) or JetPack 6 (L4T 36.x)

Native build (Jetson)

CMake:

cd build-cmake
cmake .. -DCMAKE_BUILD_TYPE=RelWithDebInfo
make -j$(nproc)
sudo make install

Pipeline Examples

Inter-process video sharing

Process A (producer):

gst-launch-1.0 \
  nvv4l2decoder ! nvmmsink shm-name=/video_feed

Process B (consumer):

gst-launch-1.0 \
  nvmmappsrc shm-name=/video_feed ! videoconvert ! autovideosink

Multi-camera fan-out to multiple consumers

The motivating use case for nvmmsink / nvmmappsrc: one producer stream published once, consumed concurrently by as many processes as you want, all staying on the GPU. Each nvmmsink pool is written once per frame; every consumer just imports the fds and reads in place — adding a second (or third) consumer does not add a second GPU copy.

Example given N ZED cameras publishing NVMM NV12 at 120 fps, and K processes that each need to encode all N streams to MP4 — every consumer gets every stream, no CPU copies.

Producer (N cameras → N shm segments, one process). Replace the serial numbers with your own (zedsrc camera-sn=...):

gst-launch-1.0 -e \
  zedsrc camera-sn=<SN1> camera-resolution=4 camera-fps=120 stream-type=7 \
    ! 'video/x-raw(memory:NVMM),format=NV12' \
    ! queue ! nvmmsink shm-name=/cam1 \
  zedsrc camera-sn=<SN2> camera-resolution=4 camera-fps=120 stream-type=7 \
    ! 'video/x-raw(memory:NVMM),format=NV12' \
    ! queue ! nvmmsink shm-name=/cam2 \
  zedsrc camera-sn=<SN3> camera-resolution=4 camera-fps=120 stream-type=7 \
    ! 'video/x-raw(memory:NVMM),format=NV12' \
    ! queue ! nvmmsink shm-name=/cam3

Consumers (each instance attaches to all three shm segments and records to its own files). Launch this pipeline in as many shells as you want — the producer above doesn't care:

timeout -s INT 120 gst-launch-1.0 -e \
  nvmmappsrc shm-name=/cam1 do-timestamp=true is-live=true \
    ! 'video/x-raw(memory:NVMM),format=NV12' \
    ! nvv4l2h264enc bitrate=20000000 ! h264parse ! qtmux \
    ! filesink location=/tmp/out_B_cam1.mp4 sync=false async=false \
  nvmmappsrc shm-name=/cam2 do-timestamp=true is-live=true \
    ! 'video/x-raw(memory:NVMM),format=NV12' \
    ! nvv4l2h264enc bitrate=20000000 ! h264parse ! qtmux \
    ! filesink location=/tmp/out_B_cam2.mp4 sync=false async=false \
  nvmmappsrc shm-name=/cam3 do-timestamp=true is-live=true \
    ! 'video/x-raw(memory:NVMM),format=NV12' \
    ! nvv4l2h264enc bitrate=20000000 ! h264parse ! qtmux \
    ! filesink location=/tmp/out_B_cam3.mp4 sync=false async=false

timeout -s INT 120 gst-launch-1.0 -e \
  nvmmappsrc shm-name=/cam1 do-timestamp=true is-live=true \
    ! 'video/x-raw(memory:NVMM),format=NV12' \
    ! nvv4l2h264enc bitrate=20000000 ! h264parse ! qtmux \
    ! filesink location=/tmp/out_C_cam1.mp4 sync=false async=false \
  nvmmappsrc shm-name=/cam2 do-timestamp=true is-live=true \
    ! 'video/x-raw(memory:NVMM),format=NV12' \
    ! nvv4l2h264enc bitrate=20000000 ! h264parse ! qtmux \
    ! filesink location=/tmp/out_C_cam2.mp4 sync=false async=false \
  nvmmappsrc shm-name=/cam3 do-timestamp=true is-live=true \
    ! 'video/x-raw(memory:NVMM),format=NV12' \
    ! nvv4l2h264enc bitrate=20000000 ! h264parse ! qtmux \
    ! filesink location=/tmp/out_C_cam3.mp4 sync=false async=false

...

The pool's per-slot ref_counts handle the fan-out: each consumer atomically increments its slot's count on read and decrements when done, and the producer only reuses a slot once its count is back to 0. Buffers stay GPU-resident through the whole pipeline — decode to encode in the consumer never leaves NVMM.

After sudo cmake --install these commands work as shown. If you're running from the build tree instead, export GST_PLUGIN_PATH to each plugin subdir as described in README_tests.md.

Decode and scale (Jetson)

gst-launch-1.0 \
  filesrc location=video.mp4 ! qtdemux ! h264parse ! nvv4l2decoder \
  ! 'video/x-raw(memory:NVMM)' \
  ! nvmmconvert \
  ! 'video/x-raw(memory:NVMM),width=640,height=480' \
  ! nvmmsink shm-name=/camera_feed

Crop a region of interest

gst-launch-1.0 \
  ... ! nvmmconvert crop-x=100 crop-y=50 crop-w=800 crop-h=600 ! ...

Flip video

# Rotate 180 degrees
gst-launch-1.0 ... ! nvmmconvert flip-method=2 ! ...

# Mirror horizontally
gst-launch-1.0 ... ! nvmmconvert flip-method=4 ! ...

The Problem This Solves

On Jetson, the native video buffer type is NvBufSurface (NVMM) — physically contiguous, DMA-coherent memory managed by the Tegra VIC hardware engine. The standard GStreamer nvcodec plugin targets discrete desktop GPUs via CUDA and doesn't understand NVMM.

This creates a gap:

• nvv4l2decoder outputs video/x-raw(memory:NVMM) but no upstream GStreamer element can consume it without a CPU copy
• Crop/scale on NVMM requires the proprietary nvvidconv element, which is tied to specific JetPack versions
• No standard GstAllocator exists for NvBufSurface, so every team writes their own

gst-nvmm-cpp fills this gap with open-source, tested, upstream-ready GStreamer elements.

Elements

nvmmconvert

Video crop, scale, and color format conversion using the Tegra VIC (Video Image Compositor) hardware engine. Zero CPU involvement.

Property	Type	Default	Description
crop-x	uint	0	Source crop X offset (pixels)
crop-y	uint	0	Source crop Y offset (pixels)
crop-w	uint	0	Source crop width (0 = full width)
crop-h	uint	0	Source crop height (0 = full height)
flip-method	int	0	0=none, 1=90CW, 2=180, 3=90CCW, 4=flipH, 5=transpose, 6=flipV, 7=inv-transpose

Supported formats: NV12, RGBA, I420, BGRA

Caps: video/x-raw(memory:NVMM), format={NV12,RGBA,I420,BGRA}, width=[1,8192], height=[1,8192]

nvmmsink

Shares NVMM video frames across processes via a GPU-copy pool: incoming buffers are copied GPU-to-GPU (via NvBufSurfaceCopy) into a fixed pool of NVMM buffers, whose DMA-buf fds are handed to consumers over a unix-domain socket (SCM_RIGHTS). Consumers (ROS2 nodes, inference engines, visualization tools) import the fds and read directly from GPU memory — no further copies, no CPU in the data path.

Property	Type	Default	Description
shm-name	string	/nvmm_sink_0	POSIX shared memory segment name
pool-size	int (3–16)	16	Number of NVMM buffers in the shared pool

Wire protocol (see gst/common/shm_protocol.h):

The shm segment holds only the header — frame data lives in a pool of NVMM buffers whose DMA-buf fds are passed over a unix-domain socket (SCM_RIGHTS). The header carries:

• Magic (0x4E564D4D = "NVMM"), version, width, height, pixel format
• Pool size, per-plane pitches and offsets
• socket_path for the fd-passing unix socket
• write_idx, monotonic frame_number, PTS timestamp_ns
• ready flag (set once the first frame is published)
• ref_counts[pool_size] so producer knows when a slot is safe to reuse

Producers copy each incoming NVMM buffer into the pool via NvBufSurfaceCopy (GPU-to-GPU, no CPU involvement). Consumers connect the socket, receive pool fds + NvBufSurfaceMapParams, import with NvBufSurfaceImport, and read directly from GPU memory.

nvmmappsrc

Reads NVMM video frames from a POSIX shared memory segment written by nvmmsink or an external producer. Pushes frames into a GStreamer pipeline.

Property	Type	Default	Description
shm-name	string	/nvmm_sink_0	POSIX shared memory segment name to read from
is-live	bool	true	Whether this source is a live source

Auto-detects video format, resolution, and plane layout from the ShmHeader on the first frame.

Tests

Unit, integration, and pipeline tests plus benchmarks and sanitizer runs are documented separately in README_tests.md. Quick start:

ctest --test-dir build-cmake --output-on-failure

Developer / contributor docs

See README_dev.md for architecture, C++ internals, the GstNvmmAllocator embedding API, Docker build recipes, the Meson build alternative, the ROS2 bridge handshake, and upstream GStreamer context.

License

LGPL-2.1-or-later. See COPYING.