Manual / Tracking
Person Detection Experimental
Person Detection is an optional, YOLO-based AI that finds people in the camera frame and can automatically steer your markers, or fire zone events without a marker. It ships ready to run: the quality-tier models are pre-installed, and detection starts as soon as you pick a tracking mode.
Person Detection is experimental: accuracy, model defaults, and the settings may change between releases. It is compute-heavy and does not run efficiently on a Raspberry Pi 5 today – a more powerful workstation is recommended for it – and it is not suited to very crowded scenes. Where it shines is tracking an individual performer: it reacts faster than a human operator once you have parked the assist anchor on the right person. Don't rely on it for show-critical paths; see Hardware.
What it does
- Steer a marker – detection writes a position to a marker, broadcast on PSN and driving zones like any operator-driven marker. Two modes, AI Assisted and Fully Automatic, are described below.
- Drive zones – detected people can act as occupants and fire the same OSC events as marker-driven zones, so you can automate zone occupancy with no operator. See Zone Occupancy Detection.
Turning it on
The Person Detection tab is hidden until you enable experimental features. On the pre-built image and .deb, the engine and the tier models are already installed, so there is nothing to add.
- Under General → Station Settings, turn on Show experimental features. The Person Detection tab appears (see Web Interface).
- Set Tracking to AI Assisted or Fully Automatic. Detection turns on with the mode; Off turns it off and costs no CPU.
On a source or base-only build, add the engine first – see Installing Person Detection. A red Detection needs extra components banner means the engine isn't present.
Tracking Experimental
The Tracking control is the master switch for the whole feature:
- Off – no detection runs; no CPU cost.
- AI Assisted – the default. Detection refines all of your controlled markers at once. Each marker has a manual anchor – the solid marker, with its card, that you steer with keyboard, gamepad, or mouse – and an AI-corrected output – a dim crosshair and ground ring, the position actually broadcast on PSN and used for zones. The output continuously glides (it never snaps) onto the detected person nearest your anchor, within the Assist radius, and eases back toward the anchor when nobody is in range. You choose who each marker follows by parking its anchor near them; the AI supplies the precise, jitter-free position. Because the anchor is independent, you can move it away at any time to hand off. This works for every operator's marker at once.
- Fully Automatic – hands-off, for a single marker. Detection picks the largest visible person and sticks to that person frame to frame, holding through brief occlusions. If the track is briefly lost or re-numbered it re-locks onto the person nearest where it last followed them; only once they are gone longer than the Grace period (or at a cold start) does it fall back to the largest person. Follow marker chooses which marker it drives.
Under both modes the detector runs a tracking-by-detection pipeline: each detection is bound to a persistent track with a stable identity, and a motion model predicts every track forward using the real time elapsed between frames. That is what lets a performer who is briefly occluded, or who steps into shadow and dims, keep the same track instead of being dropped and re-numbered.
Tracking settings
| Field | Meaning | Default |
|---|---|---|
| Follow marker | Fully Automatic only. Which marker detection drives. Currently selected (controller) follows whichever marker the operator has selected; a specific marker ID pins detection to that marker regardless of selection. | – |
| Track | Which part of the bounding box gives the position: Head (top of person) or Feet (floor position). Choose by whether show geometry references head height or floor position. | – |
| Smoothing / glide (0–1) | Exponential smoothing on the position. In AI Assisted mode this is also the glide speed toward the target. Lower is smoother and laggier; higher is more responsive, but never instant. 0.1–0.2 feels natural. | 0.15 |
| Prediction | Velocity lookahead that extrapolates the trajectory to compensate for detection lag on fast movers (0–20). 0 disables it. Frame-rate-independent, so it feels the same on a workstation and a Pi. | 8.0 |
| Grace period (ms) | How long the marker holds its last position after detection is lost (0–10000) before it stops updating. Prevents snapping away on a single missed frame. | 500 |
| Assist radius (m) | AI Assisted only. How close a detection must be to a marker's anchor to be picked up. Detections outside are ignored, so the output never jumps to a performer across the stage. Each marker follows whichever in-range detection is nearest its anchor. Depends on accurate camera calibration. | 1.0 |
| Anchor pull (clip strength) (0–1) | AI Assisted only. Where the output sits when a person is in range. 1 clips it exactly onto the person (the anchor only chooses who); lower blends the output back toward the anchor. Sets the target only; the glide to it is governed by Smoothing. | 0.5 |
Smoothing, Prediction, and Grace period live under Advanced motion; Follow marker (Fully Automatic only) and Track stay visible.
Detection Model
Detection quality is a choice of tier, fastest to most accurate. Higher tiers detect people more reliably – especially small, distant, or silhouetted subjects under stage lighting – but need more compute.
| Tier | Use |
|---|---|
| Fastest | Lowest compute; the right choice on a Raspberry Pi. |
| Fast | Light, quick on modest hardware. |
| Balanced | Good accuracy and speed; the default on a workstation. |
| Accurate | Sharper; needs a workstation. |
| Most Accurate | Best accuracy, heaviest compute. |
All five tiers come pre-installed on the macOS app. On a Raspberry Pi the Fastest / Fast / Balanced tiers ship in the image; the heavier Accurate / Most Accurate tiers (which a Pi can't run well anyway) appear grayed out and can be fetched on a workstation.
Advanced models (collapsed by default) covers everything beyond the tiers: downloading other cataloged models, selecting a model you provided yourself, deleting models to free space, and a readout of free / total space. Storage is automatic – model files live on the NVMe drive when one is mounted, otherwise in the application's working directory – and the location stays on the station: it is never written into a config export or overwritten by an import or peer broadcast. Building or converting a model with Download Model needs the model-build toolchain and internet, so run it on a workstation (see Installing Person Detection).
Sensitivity & overlay
How sensitive detection is, and what it draws on the Operator Screen:
| Field | Meaning | Default |
|---|---|---|
| Detection sensitivity (0–1) | The score a detection must reach to start a new track or count as a confident sighting. Lower catches more people but more false positives; higher is stricter. Detections just below this still hold an already-tracked person, so someone stepping into shadow keeps their identity. | 0.2 |
| Detection rate (FPS) | How often the model runs (1, 2, 5, 10, 15, 30). Higher is more responsive but uses more CPU. | 15 |
| Maximum people | Most detections kept per frame, highest-confidence first (1–50). Reduce to limit CPU when many people are in view. | 10 |
| Show boxes | Draw a bounding rectangle around each detected person. | On |
| Show labels | Draw the confidence score on each box. | On |
| Box color | Colour of the boxes; click the swatch to pick. A box currently attached to a marker is drawn in that marker's colour instead, so you can see which detection drives each followspot. | Grey |
| Box thickness (px) | Line weight for the boxes (1–10). | 2 |
Each box saves on its own. Settings take effect immediately; Save persists them across restarts.
Hardware and performance
Detection is far more compute-heavy than marker-driven tracking and doesn't run efficiently on a Raspberry Pi 5 today – a more capable workstation is recommended. If the Operator Screen or web interface becomes sluggish, drop the Detection rate, pick a lower model tier, or reduce Maximum people. See Hardware.