Millimeter-wave 3D Radar and IR Multimodal Sensing System Enabling AI-based Event Recognition with Enhanced Privacy

Sensing technologies are an increasing part of urban management and safety. It is desirable to implement urban sensing without cameras for enhanced privacy and robustness to illumination conditions. In response to these requirements, we introduce a multimodal sensing system that comprises 60-GHz phased array TX and RX modules, a passive IR camera, a COTS-based FMCW radar sub-system, a mixed-signal data acquisition module, and an FPGA. A separate PC is used to execute AI-based inferencing. The prototype system and an associated multi-modal DNN are evaluated in two use cases (1) detection of concealed objects in motion, (2) head gesture classification. Experimental results support the feasibility of this sensing approach, achieving > 99% classification accuracy for head gestures, and > 70% classification accuracy for 5 of 6 objects concealed inside a backpack while in motion along a path of about 3m. In both cases, each classification results requires < 15msec. of data.