Dynamic Shotgun Microphone

Recording clean audio for several people usually means a lapel mic on everyone. We set out to capture each person’s voice separately from a single camera-mounted device.

As team and software lead I defined the product scope from our timeline and customer interviews, and wrote the Python backend that processes video in real time and talks to the React front-end and the audio stack.

The software runs as separate Python processes. A frame buffer captures camera images with OpenCV, a face-detection process runs an MTCNN network in TensorFlow, and the main application tracks each person with a multi-Kalman filter; they exchange data through shared memory and message queues. The main application talks to the audio process and the React interface over REST, and streams live thumbnails of each person to the interface over a WebSocket.

My teammates designed the electronics and enclosure alongside the software. The electronics centre on an Nvidia Jetson TX2, with an IMX185 camera, an array of eight MEMS microphones streamed in over USB through a MiniDSP, a DAC for audio out and a 4.3″ touchscreen.

The enclosure holds the Jetson module on its carrier board, the camera, a custom breakout board, a 1400 mAh LiPo and the touchscreen. The microphone breakouts sit in a curved bar beneath it, and a hot-shoe adapter mounts the unit on a camera.