Saltar al contenido
Social robotic creatures

How Cámara follows a person

A person changes direction without warning, and Cámara always sees and reacts a little late: following them means aiming where they are going to be.

Cámara is a robotic creature built on a commercial PTZ camera (Dahua, controlled over ONVIF) and a Raspberry Pi 5. Its most basic job is to keep a person's face centered in the image while they walk around the room and, if their back is turned, to look where their face would be.

It sounds simple, but its body was not built for this: it obeys about 7 commands per second and takes time to start and stop. Every turn of the tracking loop goes through three parts.

Capture

What Cámara sees and where it is looking right now.

Processing

Where the person is in the frame, where they are in the world and how fast they are moving.

Tracking

What speed to ask of its motors to keep up with the person without overshooting.

Capture: see what is happening now, not a moment ago

Everything Cámara sees reaches it with a delay, and every order it sends to its motors takes time too. We call the sum dead time, and it is what limits most how well it can follow someone.

Video arrives over RTSP from the PTZ substream (704×480, MJPEG). A separate thread reads it nonstop and the loop always takes the latest frame, so it never decides by looking at queued images. Motor position is read on a different channel from the one used for commands.

Substream codec
formatvideo cost per callvideo latency
MJPEG+36 ms0.04 s
H.264 2 Mbps+9 ms0.30–0.34 s
H.264 Baseline, GOP 15—0.34 s

H.264 makes each call cheaper, but video arrives 7 times later: the loop's dead time goes from 0.25 to 0.52 s.

Substream bitrate (MJPEG)
kbpscommand p50position read p50
819280 ms140 ms
4096 · chosen57 ms95 ms
204856 ms91 ms

At 4096 kbps detector noise rises from σ 0.0013 to 0.0017, with 100% detection. Going lower buys nothing.

−20 msSkipping ONVIF credential checks (OnvifLoginCheck=false): −15% per call. In exchange, anyone on the local network can move the PTZ over ONVIF.
+132%A parallel channel to read position. The PTZ's HTTP server handles one request at a time and makes commands more expensive. Dropped.
0 msTurning off audio, multicast, SNMP or lowering the main stream. No measurable effect. Sending commands over CGI instead of ONVIF: +51%, worse.
~7 HzPosition refresh rate, fixed in firmware. Asking more often brings no new data.
Takeaway

Dead time is the number that rules

It limits how much correction the controller can take. A codec or transport is chosen by total loop latency, not by what each call costs.

Takeaway

Every actuator has a rate ceiling

You have to measure how many commands and reads it really absorbs. Asking for more does not speed it up: it saturates it.

Processing: where the person is and where they are going

Cámara uses two detectors. The face gives it fine framing. The body saves it when the person has their back turned or is in profile and there is no face to see. The two take turns:

Face visible: YuNet frames. No face: YOLO11n looks for the body on every frame and aims where the face was inside the box, so Cámara does not jump when switching. Following a body: YuNet runs every other turn, only to recover the face; that keeps the loop at ~22 Hz instead of ~15.

Then the person's position is converted to PTZ pan/tilt coordinates and their velocity is estimated there. Pixels do not work: the image error also changes when Cámara turns.

Body detector · Pi 5, 240 frames front, profile and back
modelp50framesfrom behind
YOLO11n-25633 ms100%27/27
YOLO11n-32048 ms99%27/27
NanoDet-Plus (onnxruntime)45 ms92%27/27
MobileNet-SSD54 ms75%27/27
Face detector
modelresult
YuNet21–73 ms depending on processing width (384–640 px). 5 landmarks.
SCRFD-500MSame coverage (80%), 12 ms more expensive.
Haar cascadeLoses the face every few frames.
25.8 HzFace, live, on the Pi with the full loop running.
22.5 HzBody, live. YOLO11n costs the same as YuNet and runs on every frame without a face.
94%Frames with a detected person over 3 minutes of real use.
σ 0.0016Detector noise, in normalized frame units, with a fixed face.
cv2.dnnChosen engine. onnxruntime does not improve YOLO11n on the Pi (32.5 vs 34.1 ms).
Takeaway

Detection rate matters more than precision

A light detector on every frame tracks better than a good one that looks in bursts, and combining two detectors covers the cases one alone misses.

Takeaway

Estimate the target in the world, not in the image

Pixel error mixes the person's motion with Cámara's own. Filtered there, the estimated velocity comes out contaminated.

Tracking: keep pace instead of chasing

Cámara does not ask the PTZ to go to a point: it asks for a speed and corrects it on the fly. That speed has two parts. The first is the person's own, so it moves along with them. The second is a correction that grows with how far they drifted from the center of the frame.

How does it predict? With a simple assumption: that in the next instant the person keeps the same speed. There is no model of how people walk and no Kalman filter. Cámara estimates the current speed and extrapolates it in a straight line 0.25 s ahead, which is how long it takes to see (0.05 s) plus how long it takes to start moving (0.20 s). That anticipation can never be larger than the error it is correcting: that way, with the person centered, noise does not move the camera on its own.

The person's speed is built from two measurements: the camera's own speed (the slope of its last positions, ~0.5 s) plus how fast the person moves within the frame. The correction is not fixed either: the gain grows with the error, so near the center it corrects gently and far from it, hard.

0.25 sPrediction horizon: 0.05 s of video plus 0.20 s of motor dead time.
27 °/sTop speed believed for the person in pan. Someone walking at 3 m moves at ~19 °/s; anything above the cap is noise.
25 °/sGate: if the camera turns faster than this, the image is not trusted and anticipation decays to zero. When in doubt, it does not anticipate.

Command speed: ContinuousMove changes speed on the fly without braking. Its useful range is 3 to 35 °/s, and a person walking at 3 m moves at ~19 °/s. Brake early: stopping is decided by the error that will remain after 0.20 s of dead time and 0.10 s of coasting, not by the current one. Jump when the error is large: an AbsoluteMove recovers someone who ended up far away in one go.

Three modes · fixed target, 18° offset
modecommandscentered infinal |dx|
position11.3 s0.067
pure speed92.7 s0.011
chase71.2 s0.019
Anticipation · person moving around the room at 1–3 m, n=3
setup|dx| p50travelp90 pause
no anticipation0.195470°1.25 s
estimator only0.227409°0.96 s
anticipation0.167622°0.48 s

With a still target, chase sends no commands in 30 s. With a fixed face, anticipation does not oscillate.

Takeaway

Keep pace instead of chasing

Commanding speed and adding the person's own gives continuity. It is paid for in travel (+32%), and that cost gets measured too.

Takeaway

Brake on the error that will remain

An actuator with delay and coasting overshoots if you brake on the current error. You have to discount what it will keep moving.

How we measured it

Every number in this post came from a dedicated test, run on the Pi and not on a desktop computer. Configurations are interleaved (A, B, A, B…) so light or network conditions do not favor one of them, and every comparison is repeated before we trust it.

The tests
testtoolstimuluswhat it measures
Cost per call and codecluf-002 plantStill PTZ, with and without videoms per command and read; video latency by cross-correlation
Substream bitrateplant substreamfixed facecommand p50 and detector noise
Detectorsluf-002 bench-person240 frames front, profile and backp50 ms and % of frames detected
Full loop, A/Bluf-002 abprinted face at 1 m (n=4); person around the room at 1–3 m (n=3, 30 s)|dx|, commands, travel and p90 pause
Real use--trace + luf-002 trace3 minutes in the roomHz per detector and % of frames with a person

Comparisons use stimuli that do not change between runs: a still printed face or a looping video of a face on an iPad. The real person comes in last, to confirm that what won with the fixed stimulus also wins in the room.

Tests that change the PTZ configuration save a backup, restore it at the end and read back every value written: the camera answers OK even to parameters that do not exist.

What carries over to other robotic creatures

Beyond this camera, these are the method lessons that came out of measuring each part.

1

Measure both sides of every trade-off

Almost no improvement is free. If you only measure what you gain, the decision can come out backwards.

2

Compare with repeatable stimuli

A fixed target or a looping video gives the same test on every run and lets you compare medians. The real person comes last, to confirm.

3

Verify the instrument before trusting it

A measurement harness has bugs too: an image that was never flipped upright, a setting that answers OK without being applied, a feature that shows as enabled and does nothing.