Millimeter-Wave 3D Radar and IR Multimodal Sensing System Enabling AI-Based Event Recognition with Enhanced Privacy

Technologies for sensing and monitoring human-related activities and vehicles/objects are an important and increasing part of urban management and safety. It is desirable to enable these systems to operate without visible-domain cameras for enhanced privacy and robustness to varying illumination and weather conditions. We introduce a multimodal sensing system integrating a 60-GHz phased-array TX/RX, a passive IR camera, a COTS FMCW radar sub-system, mixed-signal data acquisition, and an FPGA, with AI-based event classification on a separate PC. The prototype system and its multi-modal DNN are evaluated in two use cases: (1) detecting concealed objects in motion and (2) classifying head gestures. Experimental results support the feasibility of this sensing approach, achieving > 99% head-gesture accuracy and > 70% accuracy for 5 of 6 objects concealed in a backpack while moving along a 3m path. In both cases, each classification uses only < 15 ms of data.