Why Your Real-Time Capture System Drops Frames Under Load

If you run a system that captures sensor data in real time while post-processing it (transcoding, compressing, uploading), you have probably hit this: the first recording is fine, then back-to-back recordings start dropping frames, sometimes from 50fps to single digits. Here is why, and the playbook to fix it.

The pattern

• Capture side: multiple sensors (cameras, IMU, pose) writing at high rate, in real time.
• Post-processing: transcode / compress / upload, running async.

The classic symptom, first task clean and later tasks collapsing, is post-processing starving the capture path of a shared resource.

The usual culprits, in order

1. CPU contention: software encoding (CPU AV1/H.264) in post-processing pegs the cores, so capture threads cannot get a time slice.
2. Memory bandwidth: decode, format-convert and upload push large frames through system RAM repeatedly. Bandwidth saturates and cgroups/cpusets cannot isolate it.
3. Hardware encoder session limits: consumer NVIDIA NVENC caps concurrent sessions (often 8 or fewer). Capture plus post-processing together can exceed it and silently fall back to CPU, which makes things worse.
4. Background noise: monitoring, logging and dashboard containers quietly eating CPU.

How to actually find it, the method matters

1. Instrument, do not guess: sample CPU/mem/GPU/encoder utilization every second and align it with the frame-rate timeline. You will see exactly who is eating resources at the moment frames drop.
2. Control variables: change one thing at a time (codec, resolution, core allocation, GPU on/off, connection).
3. Invariance reasoning: if a metric is immune to every downstream variable you change, the bottleneck is upstream at the source, not where you are poking.
4. Refute your own hypotheses: let measurements kill your guesses. I once knew for certain it was bandwidth; the data said no.

Stability in a capture-and-process system is shared-resource budget management. Measure first, then isolate.

The fix toolbox

• Lower the load: drop capture resolution and fps to good enough.
• Kill background processes during capture.
• Push encode to hardware: NVDEC, in-VRAM convert, then NVENC, so raw frames never hit system RAM. This removes both CPU decode and the memory-bandwidth pressure.
• Budget encoder sessions: keep capture plus post-processing hardware-encode sessions at or under the cap, and serialize post-processing if needed.
• Pick the right format: per-frame independent codecs (MJPEG) can drop frames freely. GOP/B-frame codecs (H.264/AV1) cannot, and they back-pressure the write queue.
• Decouple capture from post-processing, and use a single latest-frame slot rather than an unbounded queue. A growing queue is a latency avalanche.

A real result

One continuous-capture system, three combined levers: lower resolution, kill background services, and keep post-processing off the GPU NVENC session cap. Capture CPU peak went from 98% to 40%, and frame rate went from collapsing back to stable full rate. The win was not a single trick. It was monitoring first to find the true cause (CPU saturation), then reducing load in a targeted way.

TL;DR

Stability in a capture-and-process system is shared-resource budget management. Measure by aligning resource use with frame rate, then isolate by reducing load, offloading to hardware, budgeting sessions and decoupling stages. Do not tune by intuition.

Written by an engineer who debugs robotics data pipelines by day and builds products with AI by night.