Processing sensor data for neural networks
AssigneeGM Cruise Holdings LLC
InventorPaul Carrington Foster
Life of this patent
Estimating the term…
Abstract
Systems and techniques are disclosed for processing sensor data using neural networks. An example method can include receiving, via a first sensor, a first set of sensor data, wherein the first set of sensor data represents an environment; receiving, via a second sensor, a second set of sensor data, wherein the second set of sensor data represents the environment; generating one or more multi-view projection (MVP) tokens based on a combination of the first set of sensor data and the second set of sensor data, wherein each MVP token of the one or more MVP tokens includes a representation of the first set of sensor data projected into space and/or the second set of sensor data projected into space; and processing, via a neural network with an attention mechanism, the one or more MVP tokens.
Claims20
A system comprising:
- at least one memory;
- and
- at least one processor coupled to the at least one memory, the at least one processor configured to:
- receive, via a first sensor, a first set of sensor data, wherein the first set of sensor data represents an environment;
- receive, via a second sensor, a second set of sensor data, wherein the second set of sensor data represents the environment;
- generate one or more multi-view projection (MVP) tokens based on a combination of the first set of sensor data and the second set of sensor data, wherein each MVP token of the one or more MVP tokens comprises a representation of at least one of the first set of sensor data projected into space and the second set of sensor data projected into space;
- and
- process, via a neural network with an attention mechanism, the one or more MVP tokens, wherein the attention mechanism determines an importance or relevance of the one or more MVP tokens.
A computer-implemented method comprising:
- receiving, via a first sensor, a first set of sensor data, wherein the first set of sensor data represents an environment;
- receiving, via a second sensor, a second set of sensor data, wherein the second set of sensor data represents the environment;
- generating one or more multi-view projection (MVP) tokens based on a combination of the first set of sensor data and the second set of sensor data, wherein each MVP token of the one or more MVP tokens comprises a representation of at least one of the first set of sensor data projected into space and the second set of sensor data projected into space;
- and
- processing, via a neural network with an attention mechanism, the one or more MVP tokens, wherein the attention mechanism determines an importance or relevance of the one or more MVP tokens.
A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors, cause the one or more processors to:
- receive, via a first sensor, a first set of sensor data, wherein the first set of sensor data represents an environment;
- receive, via a second sensor, a second set of sensor data, wherein the second set of sensor data represents the environment;
- generate one or more multi-view projection (MVP) tokens based on a combination of the first set of sensor data and the second set of sensor data, wherein each MVP token of the one or more MVP tokens comprises a representation of at least one of the first set of sensor data projected into space and the second set of sensor data projected into space;
- and
- process, via a neural network with an attention mechanism, the one or more MVP tokens, wherein the attention mechanism determines an importance or relevance of the one or more MVP tokens subsequent to the generation of the one or more MVP tokens.
Dependencies are inferred from the claim text.