Defending multimodal fusion models against single-source adversaries

A multimodal perception method comprising:

• receiving a first signal from a first sensor, a second signal from a second sensor, and a third signal from a third sensor;
• extracting a first feature vectors from the first signal, a second feature vectors from the second signal, and a third feature vectors from the third signal;
• determining an odd-one-out, vector from the first, second, and third feature vectors via an odd-one-out network of a machine learning network based on an inconsistent modality prediction;
• fusing, via a fusion network, the first, second, and third feature vectors and odd-one-out vector into a fused feature vector;
• and
• outputting the fused feature vector.

A multimodal perception system for an autonomous vehicle comprising:

• a first sensor that is one of a video, RADAR, LIDAR, or ultrasound sensor;
• and
• a controller configured to,
• receive a first signal from a first sensor, a second signal from a second sensor, and a third signal from a third sensor,
• extract a first feature vector from the first signal,
• extract a second feature vector from the second signal,
• extract a third feature vector from the third signal,
• determine an odd-one-out vector from the first, second, and third feature vectors via an odd-one-out network of a machine learning network, based on inconsistent modality prediction,
• fuse, via a fusion network, the first, second, and third feature vectors and odd-one-out vector into a fused feature vector,
• output the fused feature vector, and
• control the autonomous vehicle based on the fused feature vector.