Back to results
US 12,597,262 B2Granted
Detecting and identifying objects represented in sensor data generated by multiple sensor systems
AssigneeGM Cruise Holdings LLC
InventorsGeorg Kuschk, Marc Unzueta Canals, Sven Möller, Michael Meyer, Karl-Heinz Krachenfels
Life of this patent
Estimating the term…
Abstract
A system includes a first sensor system of a first modality and a second sensor system of a second modality. The system further includes a computing system that is configured to detect and identify objects represented in sensor signals output by the first and second sensor systems. The computing system employs a hierarchical arrangement of transformers to fuse features of first sensor data output by the first sensor system and second sensor data output by the second sensor system.
Claims20
A system comprising:
- a first sensor system that generates first sensor data, the first sensor data corresponding to a first modality;
- a second sensor system that generates second sensor data, the second sensor data corresponding to a second modality;
- a computing system that is in communication with the first sensor system and the second sensor system, wherein the computing system comprises:
- a processor;
- and
- memory that stores computer-executable instructions that, when executed by the processor, cause the processor to perform acts comprising:
- generating, by a first transformer, a first output based upon the first sensor data, wherein the first output comprises identities of objects determined by the first transformer to be represented in the first sensor data and corresponding locations of the objects in the first sensor data;
- generating, by a second transformer, a second output based upon the second sensor data, wherein the second output comprises identities of objects determined by the second transformer to be represented in the second sensor data and corresponding locations of the objects in the second sensor data;
- and
- generating, by a third transformer, a third output based upon the first output and the second output, wherein the third transformer comprises an encoder and a decoder, wherein the encoder processes the second output and the decoder, using cross-attention and self-attention, processes the first output, wherein the cross-attention correlates objects between the first output and the second output, wherein the self-attention evaluates consistency of relationships between the correlated objects, and wherein the third output comprises identities of objects determined by the third transformer to be in an environment of the system and corresponding locations of the objects in the environment of the system.
7 more elements
A method performed by a computing system, the method comprising:
- generating, by a first transformer, a first output based upon first sensor data generated by a first sensor system, wherein the first output comprises identities of objects determined by the first transformer to be represented in the first sensor data and corresponding locations of the objects in the first sensor data, and further wherein the first sensor data is in a first modality;
- generating, by a second transformer, a second output based upon second sensor data generated by a second sensor system, wherein the second output comprises identities of objects determined by the second transformer to be represented in the second sensor data and corresponding locations of the objects in the second sensor data, and further wherein the second sensor data is in a second modality;
- and
- generating, by a third transformer, a third output based upon the first output and the second output, wherein the third transformer comprises an encoder and a decoder, wherein the encoder processes the second output and the decoder, using cross-attention and self-attention, processes the first output, wherein the cross-attention correlates objects between the first output and the second output, wherein the self-attention evaluates consistency of relationships between the correlated objects, and wherein the third output comprises identities of objects determined by the third transformer to be in an environment of the first sensor system and the second sensor system and corresponding locations of the objects in the environment of the first sensor system and the second sensor system.
1 more element
A computer-readable storage device comprising instructions that, when executed by a processor, cause the processor to perform acts comprising:
- generating, by a first transformer, a first output based upon first sensor data generated by a first sensor system, wherein the first output comprises identities of objects determined by the first transformer to be represented in the first sensor data and corresponding locations of the objects in the first sensor data, and further wherein the first sensor data is in a first modality;
- generating, by a second transformer, a second output based upon second sensor data generated by a second sensor system, wherein the second output comprises identities of objects determined by the second transformer to be represented in the second sensor data and corresponding locations of the objects in the first second sensor data, and further wherein the second sensor data is in a second modality;
- and
- generating, by a third transformer, a third output based upon the first output and the second output, wherein the third transformer comprises an encoder and a decoder, wherein the encoder processes the second output and the decoder, using cross-attention and self-attention, processes the first output, wherein the cross-attention correlates objects between the first output and the second output, wherein the self-attention evaluates consistency of relationships between the correlated objects, and wherein the third output comprises identities of objects determined by the third transformer to be in an environment of the first sensor system and the second sensor system and corresponding locations of the objects in the environment of the first sensor system and the second sensor system.
1 more element
Dependencies are inferred from the claim text.