MCMT — Real-time multi-camera multi-target tracking
Person-tracking engine across hundreds of cameras (up to 1,300 simultaneously), combining a geometric C++ core with a graph neural network to resolve ambiguous cases.
MCMT is a multi-camera multi-target tracking library designed to answer, in real time and at scale, the question 'is this person I see in camera A the same one I saw earlier in camera B?'. The system tracks each object within a single camera using an extended Kalman filter, fuses those local tracks into global identities through a cascade of geometric and appearance-based rules, and only falls back on a graph neural network for the small fraction of cases geometry can't resolve confidently. It's built as a Python/C++ hybrid: Python orchestrates business logic and design patterns, while C++ (via Eigen and nanobind) implements the performance-critical kernels, sharing memory with zero copies through Apache Arrow. The whole architecture targets free-threaded Python 3.13 (no-GIL), with lock-free data structures and a strict lock hierarchy to guarantee real parallelism without deadlocks.
The problem
In a venue covered by many overlapping cameras, the system must continuously decide which detection from one camera corresponds to a person already seen in another, under tough constraints: each camera only provides 2D coordinates that must be projected into real-world metric space; the same person looks different depending on angle and lighting; camera clocks drift out of sync; and everything has to be resolved within milliseconds for thousands of detections per second at a scale of up to 1,300 cameras. A single bad association can also permanently corrupt a person's visual identity if not handled carefully.
The solution
I designed a two-tier architecture: a geometric tier ('Glance') that resolves roughly 99% of associations through a five-gate cascade of increasing cost — camera field-of-view compatibility, ground-plane proximity, Mahalanobis distance over the Kalman filter's covariance, velocity coherence, and finally appearance similarity (Re-ID) — solved with the Hungarian algorithm. The remaining 1% of ambiguous cases is handed off, off the critical path and on background threads, to a graph neural network (MPGNN) with attention and Sinkhorn normalization. Every performance-critical kernel (Kalman filter, 2D→3D projection, lock-free temporal synchronization, double-buffered embedding pool, graph network inference) has both a C++ implementation and a pure-NumPy one, used as an automatic fallback and as a numerically validated reference in tests. The result is a multi-threaded system with four concurrent responsibilities (critical path, AI inference, identity lifecycle) coordinated through double buffering, bounded queues, and a documented lock hierarchy that eliminates deadlocks by design. The system was validated on the WILDTRACK dataset with MODA/MODP metrics and an ablation study confirming the core design thesis: geometry resolves almost everything, and the graph network delivers a measurable improvement on the hard cases without compromising real-time latency.