# US 12,601,831 B2 · Radar and camera fusion for vehicle applications > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,601,831 B2 - Document ID: US12601831B2 - Status: Granted - Kind: B2 - Application number: 18/463,049 - Filed: 2023-09-07 - Published: 2025-03-13 - Granted: 2026-04-14 - Earliest claimed: 2023-09-07 - Assignees: QUALCOMM Incorporated - Applicants: QUALCOMM Incorporated - Inventors: Senthil Kumar Yogamani; Varun Ravi Kumar - Web page: https://patentgrep.com/patent/US12601831B2 ## CPC - G01S 13/867: {Combination of radar systems with cameras} - G01S 7/354: {Extracting wanted echo-signals (Doppler systems G01S13/50)} - G01S 7/417: {involving the use of neural networks} - G01S 13/89: for mapping or imaging - G01S 13/931: of land vehicles - G01S 13/87: Combinations of radar systems, e.g. primary radar and secondary radar - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES ## Abstract This disclosure provides systems, methods, and devices for vehicle driving assistance systems that support image processing. In a first aspect, a method of image processing includes receiving image BEV features and receiving first radio detection and ranging (RADAR) BEV features. The first RADAR BEV features that are received are determined based on first RADAR data associated with a first data type. First normalized RADAR BEV features are determined, which includes rescaling the first RADAR BEV features using a first attention mechanism based on the image BEV features and the first RADAR BEV features. Fused data is determined that combines the first normalized RADAR BEV features and the image BEV features. Other aspects and features are also claimed and described. ## Claims ### Claim 1 A method for image processing for use in a vehicle assistance system, comprising: receiving image Bird's Eye View (BEV) features; receiving first radio detection and ranging (RADAR) BEV features that are determined based on first RADAR data associated with a first data type; determining first normalized RADAR BEV features, which includes rescaling the first RADAR BEV features using a first attention mechanism based on the image BEV features and the first RADAR BEV features; and determining fused data that combines the first normalized RADAR BEV features and the image BEV features. ### Claim 2 (depends on 1) The method of claim 1, wherein the first attention mechanism is a self-attention mechanism. ### Claim 3 (depends on 1) The method of claim 1, wherein the first attention mechanism is a cross-attention mechanism. ### Claim 4 (depends on 1) The method of claim 1, wherein the first data type is one of the data types in the group consisting of: a range-doppler map, a range-azimuth map, a point cloud, a 2D RADAR image, a list of detected objects based on the first RADAR data, and a set of 3D bounding boxes based on the first RADAR data. ### Claim 5 (depends on 4) The method of claim 4, wherein if the first data type is the range-doppler map data type, the range-azimuth map data type, or the point cloud data type, then the first attention mechanism is a cross-attention mechanism, and if the first data type is the 2D RADAR image type, the list of detected objects based on the first RADAR data type, or the set of 3D bounding boxes based on the first RADAR data type, then the attention mechanism is a self-attention mechanism. ### Claim 6 (depends on 1) The method of claim 1, further comprising: receiving second RADAR BEV features that are determined based on second RADAR data associated with a second data type different than the first data type; and determining second normalized RADAR BEV features, which includes rescaling the second RADAR BEV features using a second attention mechanism based on the image BEV features and the second RADAR BEV features, wherein the fused data combines the first normalized RADAR BEV features, the second normalized RADAR BEV features, and the image BEV features. ### Claim 7 (depends on 6) The method of claim 6, wherein the first attention mechanism is a cross-attention mechanism and the second attention mechanism is a self-attention mechanism. ### Claim 8 (depends on 1) The method of claim 1, wherein the image BEV features are determined based on image data received from an image sensor. ### Claim 9 (depends on 1) The method of claim 1, wherein the attention mechanism includes learned attention weights based on ground-truth annotations of a camera that captured image data on which the image BEV features are based. ### Claim 10 (depends on 1) The method of claim 1, further comprising: detecting an object based on the fused data; and controlling a function of a vehicle based on the object detected. ### Claim 11 An apparatus, comprising: a memory storing processor-readable code; and at least one processor coupled to the memory, the at least one processor configured to execute the processor-readable code to cause the at least one processor to perform operations including: receiving image Bird's Eye View (BEV) features; receiving first radio detection and ranging (RADAR) BEV features that are determined based on first RADAR data associated with a first data type; determining normalized RADAR BEV features, which includes rescaling the first RADAR BEV features using a first attention mechanism based on the image BEV features and the first RADAR BEV features; and determining fused data that combines the normalized RADAR BEV features and the image BEV features. ### Claim 12 (depends on 11) The apparatus of claim 11, wherein the first attention mechanism is a self-attention mechanism. ### Claim 13 (depends on 11) The apparatus of claim 11, wherein the first attention mechanism is a cross-attention mechanism. ### Claim 14 (depends on 11) The apparatus of claim 11, wherein the first data type is one of the data types in the group consisting of: a range-doppler map, a range-azimuth map, a point cloud, a 2D RADAR image, a list of detected objects, and a set of 3D bounding boxes. ### Claim 15 (depends on 14) The apparatus of claim 14, wherein if the first data type is the range-doppler map data type, the range-azimuth map data type, or the point cloud data type, then the first attention mechanism is a cross-attention mechanism, and if the first data type is the 2D RADAR image type, the list of detected objects based on the first RADAR data type, or the set of 3D bounding boxes based on the first RADAR data type, then the attention mechanism is a self-attention mechanism. ### Claim 16 (depends on 11) The apparatus of claim 11, wherein the operations further include: receiving second RADAR BEV features that are determined based on second RADAR data associated with a second data type different than the first data type; and determining second normalized RADAR BEV features, which includes rescaling the second RADAR BEV features using a second attention mechanism based on the image BEV features and the second RADAR BEV features, wherein the fused data combines the first normalized RADAR BEV features, the second normalized RADAR BEV features, and the image BEV features. ### Claim 17 (depends on 16) The apparatus of claim 16, wherein the first attention mechanism is a cross-attention mechanism and the second attention mechanism is a self-attention mechanism. ### Claim 18 (depends on 11) The apparatus of claim 11, wherein the image BEV features are determined based on image data received from an image sensor. ### Claim 19 (depends on 11) The apparatus of claim 11, wherein the attention mechanism includes learned attention weights based on ground-truth annotations of a camera that captured image data on which the image BEV features are based. ### Claim 20 (depends on 11) The apparatus of claim 11, wherein the operations further include: detecting an object based on the fused data; and controlling a function of a vehicle based on the object detected. ### Claim 21 A non-transitory computer-readable medium storing instructions that, when executed by a processing system that includes one or more processors, cause the processing system to perform operations comprising: receiving image Bird's Eye View (BEV) features; receiving first radio detection and ranging (RADAR) BEV features that are determined based on first RADAR data associated with a first data type; determining normalized RADAR BEV features, which includes rescaling the first RADAR BEV features using a first attention mechanism based on the image BEV features and the first RADAR BEV features; and determining fused data that combines the normalized RADAR BEV features and the image BEV features. ### Claim 22 (depends on 21) The non-transitory, computer-readable medium of claim 21, wherein the first attention mechanism is a self-attention mechanism or a cross-attention mechanism. ### Claim 23 (depends on 21) The non-transitory, computer-readable medium of claim 21, wherein the first data type is one of the data types in the group consisting of: a range-doppler map, a range-azimuth map, a point cloud, a 2D RADAR image, a list of detected objects, and a set of 3D bounding boxes. ### Claim 24 (depends on 21) The non-transitory, computer-readable medium of claim 21, wherein the operations further include: receiving second RADAR BEV features that are determined based on second RADAR data associated with a second data type different than the first data type; and determining second normalized RADAR BEV features, which includes rescaling the second RADAR BEV features using a second attention mechanism based on the image BEV features and the second RADAR BEV features, wherein the fused data combines the first normalized RADAR BEV features, the second normalized RADAR BEV features, and the image BEV features. ### Claim 25 (depends on 21) The non-transitory, computer-readable medium of claim 21, wherein the attention mechanism includes learned attention weights based on ground-truth annotations of a camera that captured the image data. ### Claim 26 A vehicle, comprising: an image sensor; a radio detection and ranging (RADAR) system; and a memory storing processor-readable code; and at least one processor coupled to the memory, the at least one processor in communication with the image sensor and the RADAR system and configured to execute the processor-readable code to cause the at least one processor to perform operations including: receiving image Bird's Eye View (BEV) features that are determined based on image data received from the image sensor; receiving first RADAR BEV features that are determined based on first RADAR data associated with a first data type, the first RADAR data received from the RADAR system; determining normalized RADAR BEV features, which includes rescaling the first RADAR BEV features using a first attention mechanism based on the image BEV features and the first RADAR BEV features; and determining fused data that combines the normalized RADAR BEV features and the image BEV features. ### Claim 27 (depends on 26) The vehicle of claim 26, wherein the first attention mechanism is a self-attention mechanism or a cross-attention mechanism. ### Claim 28 (depends on 26) The vehicle of claim 26, wherein the RADAR system is a first RADAR system, the vehicle further comprising a second RADAR system in communication with the processing system, the operations including: receiving second RADAR BEV features that are determined based on second RADAR data associated with a second data type different than the first data type, the second RADAR data received from the second RADAR system; and determining second normalized RADAR BEV features, which includes rescaling the second RADAR BEV features using a second attention mechanism based on the image BEV features and the second RADAR BEV features, wherein the fused data combines the first normalized RADAR BEV features, the second normalized RADAR BEV features, and the image BEV features. ### Claim 29 (depends on 28) The vehicle of claim 28, wherein the first data type is one of the data types in the group consisting of: a range-doppler map, a range-azimuth map, and a point cloud, and wherein the second data type is one of the data types in the group consisting of: a 2D RADAR image, a list of detected objects, and a set of 3D bounding boxes. ### Claim 30 (depends on 26) The vehicle of claim 26, wherein the attention mechanism includes learned attention weights based on ground-truth annotations of a camera that captured the image data.