# US 12,602,915 B2 · Feature fusion for near field and far field images for vehicle applications > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,915 B2 - Document ID: US12602915B2 - Status: Granted - Kind: B2 - Application number: 18/313,287 - Filed: 2023-05-05 - Published: 2024-11-07 - Granted: 2026-04-14 - Earliest claimed: 2023-05-05 - Assignees: QUALCOMM Incorporated - Applicants: QUALCOMM Incorporated - Inventors: Varun Ravi Kumar; Senthil Kumar Yogamani - Web page: https://patentgrep.com/patent/US12602915B2 ## CPC - G06V 10/806: {of extracted features} - G06F 18/253: {of extracted features} - G06V 10/764: using classification, e.g. of video objects - G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods} - G06V 10/82: using neural networks - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract This disclosure provides systems, methods, and devices for vehicle driving assistance systems that support image processing. In a first aspect, a method of fusing features from near-field images and far-field images is provided that includes determining feature vectors and spatial locations for received images from near-field and far-field image sensors. A first set of weighted feature vectors may be determined based on spatial locations of the features and a second set of weighted feature vectors may be determined based on corresponding features between the feature vectors. Fused feature vectors may then be determined based on the weighted feature vectors, such as using a transformer attention process trained to select and combine features from both sets of weighted feature vectors. Vehicle control instructions may be determined based on the fused feature vectors. Other aspects and features are also claimed and described. ## Claims ### Claim 1 A method for image processing for use in a vehicle assistance system, comprising: receiving a first plurality of images captured by near-field image sensors and a second plurality of images captured by far-field image sensors; determining a first set of feature vectors from the first plurality of images and a second set of feature vectors from the second plurality of images; determining a first set of weighted feature vectors based on spatial locations for features from the first set of feature vectors and the second set of feature vectors; determining a second set of weighted feature vectors based on corresponding features between the first set of feature vectors and the second set of feature vectors; determining fused feature vectors based on the first set of weighted feature vectors and the second set of weighted feature vectors; and controlling a vehicle based on the fused feature vectors. ### Claim 2 (depends on 1) The method of claim 1, wherein determining the first set of weighted feature vectors comprises: determining, for feature values from the first set of feature vectors, higher weights to features that are located closer to the vehicle; and determining, for feature values from the second set of feature vectors, higher weights to features that are located further from the vehicle. ### Claim 3 (depends on 1) The method of claim 1, wherein determining the second set of weighted feature vectors comprises determining sets of corresponding features between the feature vectors, wherein each set of corresponding features identifies at least two feature values from at least two feature vectors located in the same or similar location. ### Claim 4 (depends on 3) The method of claim 3, wherein each of at least a subset of the sets of corresponding features is used to determine a corresponding feature value of the second set of weighted feature vectors. ### Claim 5 (depends on 1) The method of claim 1, wherein determining the second set of weighted feature vectors comprises determining weighted feature values based on feature values for corresponding features. ### Claim 6 (depends on 5) The method of claim 5, wherein the weighted feature values are determined using a pixel adaptive convolution process. ### Claim 7 (depends on 1) The method of claim 1, wherein the fused feature vectors are determined based on a transformer attention process that receives the first set of weighted feature vectors and the second set of weighted feature vectors. ### Claim 8 (depends on 7) The method of claim 7, wherein the second set of weighted feature vectors are provided as query vectors to the transformer attention process and the first set of weighted feature vectors are provided as key vectors to the transformer attention process. ### Claim 9 (depends on 8) The method of claim 8, wherein the fused feature vectors are determined based on output values from the transformer attention process. ### Claim 10 (depends on 1) The method of claim 1, wherein the feature vectors from the first plurality of images and the second plurality of images are identified by an encoder model. ### Claim 11 (depends on 1) The method of claim 1, further comprising determining spatial locations of the first and second sets of feature vectors within a physical area surrounding a vehicle. ### Claim 12 (depends on 11) The method of claim 11, wherein the spatial locations are determined as part of a top view of the physical area surrounding the vehicle. ### Claim 13 (depends on 11) The method of claim 11, wherein the spatial locations specify a distance from the vehicle and an angular offset from a heading of the vehicle for corresponding features. ### Claim 14 (depends on 1) The method of claim 1, further comprising: determining, based on the fused feature vectors, a top view segmentation map of an area surrounding the vehicle, wherein the vehicle control instructions are determined based on the top view segmentation map. ### Claim 15 An apparatus, comprising: a memory storing processor-readable code; and at least one processor coupled to the memory, the at least one processor configured to execute the processor-readable code to cause the at least one processor to perform operations including: receiving a first plurality of images captured by near-field image sensors and a second plurality of images captured by far-field image sensors; determining a first set of feature vectors from the first plurality of images and a second set of feature vectors from the second plurality of images; determining a first set of weighted feature vectors based on spatial locations for features from the first set of feature vectors and the second set of feature vectors; determining a second set of weighted feature vectors based on corresponding features between the first set of feature vectors and the second set of feature vectors; determining fused feature vectors based on the first set of weighted feature vectors and the second set of weighted feature vectors; and controlling a vehicle based on the fused feature vectors. ### Claim 16 (depends on 15) The apparatus of claim 15, wherein determining the first set of weighted feature vectors comprises: determining, for feature values from the first set of feature vectors, higher weights to features that are located closer to the vehicle; and determining, for feature values from the second set of feature vectors, higher weights to features that are located further from the vehicle. ### Claim 17 (depends on 15) The apparatus of claim 15, wherein determining the second set of weighted feature vectors comprises determining weighted feature values based on feature values for corresponding features. ### Claim 18 (depends on 17) The apparatus of claim 17, wherein the weighted feature values are determined using a pixel adaptive convolution process. ### Claim 19 (depends on 15) The apparatus of claim 15, wherein the fused feature vectors are determined based on a transformer attention process that receives the first set of weighted feature vectors and the second set of weighted feature vectors. ### Claim 20 (depends on 19) The apparatus of claim 19, wherein the second set of weighted feature vectors are provided as query vectors to the transformer attention process and the first set of weighted feature vectors are provided as key vectors to the transformer attention process. ### Claim 21 (depends on 20) The apparatus of claim 20, wherein the fused feature vectors are determined based on output values from the transformer attention process. ### Claim 22 (depends on 15) The apparatus of claim 15, wherein the operations further comprise: determining, based on the fused feature vectors, a top view segmentation map of an area surrounding the vehicle, wherein the vehicle control instructions are determined based on the top view segmentation map. ### Claim 23 A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising: receiving a first plurality of images captured by near-field image sensors and a second plurality of images captured by far-field image sensors; determining a first set of feature vectors from the first plurality of images and a second set of feature vectors from the second plurality of images; determining a first set of weighted feature vectors based on spatial locations for features from the first set of feature vectors and the second set of feature vectors; determining a second set of weighted feature vectors based on corresponding features between the first set of feature vectors and the second set of feature vectors; determining fused feature vectors based on the first set of weighted feature vectors and the second set of weighted feature vectors; and controlling a vehicle based on the fused feature vectors. ### Claim 24 (depends on 23) The non-transitory, computer-readable medium of claim 23, wherein determining the first set of weighted feature vectors comprises: determining, for feature values from the first set of feature vectors, higher weights to features that are located closer to the vehicle; and determining, for feature values from the second set of feature vectors, higher weights to features that are located further from the vehicle. ### Claim 25 (depends on 23) The non-transitory, computer-readable medium of claim 23, wherein the fused feature vectors are determined based on a transformer attention process that receives the first set of weighted feature vectors and the second set of weighted feature vectors. ### Claim 26 (depends on 25) The non-transitory, computer-readable medium of claim 25, wherein the second set of weighted feature vectors are provided as query vectors to the transformer attention process and the first set of weighted feature vectors are provided as key vectors to the transformer attention process. ### Claim 27 A vehicle, comprising: near-field image sensors; far-field image sensors; a memory storing processor-readable code; and at least one processor coupled to the memory, the at least one processor configured to execute the processor-readable code to cause the at least one processor to perform operations including: receiving a first plurality of images captured by the near-field image sensors and a second plurality of images captured by the far-field image sensors; determining a first set of feature vectors from the first plurality of images and a second set of feature vectors from the second plurality of images; determining a first set of weighted feature vectors based on spatial locations for features from the first set of feature vectors and the second set of feature vectors; determining a second set of weighted feature vectors based on corresponding features between the first set of feature vectors and the second set of feature vectors; determining fused feature vectors based on the first set of weighted feature vectors and the second set of weighted feature vectors; and controlling the vehicle based on the fused feature vectors. ### Claim 28 (depends on 27) The vehicle of claim 27, wherein determining the first set of weighted feature vectors comprises: determining, for feature values from the first set of feature vectors, higher weights to features that are located closer to the vehicle; and determining, for feature values from the second set of feature vectors, higher weights to features that are located further from the vehicle. ### Claim 29 (depends on 27) The vehicle of claim 27, wherein the fused feature vectors are determined based on a transformer attention process that receives the first set of weighted feature vectors and the second set of weighted feature vectors. ### Claim 30 (depends on 29) The vehicle of claim 29, wherein the second set of weighted feature vectors are provided as query vectors to the transformer attention process and the first set of weighted feature vectors are provided as key vectors to the transformer attention process.