Spatio-temporal cooperative learning for multi-sensor fusion

A method for processing data by a first device, the method comprising:

• extracting first features from first frames of first sensor data;
• extracting second features from second frames of second sensor data, the second sensor data being captured after the first sensor data;
• determining first weights associated with the first features to obtain first weighted features;
• determining second weights associated with the second features to obtain second weighted features;
• aggregating the first weighted features to determine a first feature vector;
• aggregating the second weighted features to determine a second feature vector;
• transforming the first feature vector into a coordinate space to obtain a first transformed feature vector;
• transforming the second feature vector into the coordinate space to obtain a second transformed feature vector;
• determining first transform weights associated with the first transformed feature vector to obtain first transformed weighted features;
• determining second transform weights associated with the second transformed feature vector to obtain second transformed weighted features;
• and
• aggregating the first transformed weighted features and the second transformed weighted features together to determine a fused feature vector.

A non-transitory computer-readable storage medium of a first device comprising instructions stored thereon which, when executed by at least one processor, causes the at least one processor to:

• extract first features from first frames of first sensor data;
• extract second features from second frames of second sensor data, the second sensor data being captured after the first sensor data;
• determine first weights associated with the first features to obtain first weighted features;
• determine second weights associated with the second features to obtain second weighted features;
• aggregate the first weighted features to determine a first feature vector;
• aggregate the second weighted features to determine a second feature vector;
• transform the first feature vector into a coordinate space to obtain a first transformed feature vector;
• transform the second feature vector into the coordinate space to obtain a second transformed feature vector;
• determine first transform weights associated with the first transformed feature vector to obtain first transformed weighted features;
• determine second transform weights associated with the second transformed feature vector to obtain second transformed weighted features;
• and
• aggregate the first transformed weighted features and the second transformed weighted features together to determine a fused feature vector.