# US 2025/0156997 A1 · Stochastic dynamic field of view for multi-camera bird’s eye view perception in autonomous driving

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0156997 A1
- Document ID: US20250156997A1
- Status: Published application
- Kind: A1
- Application number: 18/505,923
- Filed: 2023-11-09
- Published: 2025-05-15
- Earliest claimed: 2023-11-09
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Varun Ravi Kumar; Kiran Bangalore Ravi; Senthil Kumar Yogamani
- Web page: https://patentgrep.com/patent/US20250156997A1

## CPC

- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06N 3/047: Probabilistic or stochastic networks
- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/40: Extraction of image or video features
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An apparatus for processing image data includes a memory for storing the image data, wherein the image data comprises a first set of image data collected by a first camera comprising a first field of view (FOV) and a second set of image data collected by a second camera comprising a second FOV; and processing circuitry in communication with the memory. The processing circuitry is configured to: apply an encoder to extract, from the first set of image data, a first set of perspective view features; apply the encoder to extract, from the second set of image data, a second set of perspective view features; and project the first set of perspective view features and the second set of perspective view features onto a grid to generate a set of bird's eye view (BEV) features.

## Claims

### Claim 1

An apparatus for processing image data, the apparatus comprising:
a memory for storing the image data, wherein the image data comprises a first set of image data collected by a first camera comprising a first field of view (FOV) and a second set of image data collected by a second camera comprising a second FOV; and
processing circuitry in communication with the memory, wherein the processing circuitry is configured to:
apply an encoder to extract, from the first set of image data based on a location of a first one or more objects within the first FOV, a first set of perspective view features;
apply the encoder to extract, from the second set of image data based on a location of a second one or more objects within the second FOV, a second set of perspective view features; and
project the first set of perspective view features and the second set of perspective view features onto a grid to generate a set of bird's eye view (BEV) features that provides information corresponding to the first one or more objects and the second one or more objects.

### Claim 2 (depends on 1)

The apparatus of claim 1,
wherein to apply the encoder to extract the first set of perspective view features, the processing circuitry is configured to use a first stochastic depth scaling function to extract the first set of perspective view features based on a distance of each object of the first one or more objects from the first camera, and
wherein to apply the encoder to extract the second set of perspective view features, the processing circuitry is configured to use a second stochastic depth scaling function to extract the second set of perspective view features based on a distance of each object of the second one or more objects from the second camera.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein the processing circuitry is further configured to:
generate a first set of input perspective view features based on the first set of image data;
generate a second set of input perspective view features based on the second set of image data;
determine a first camera pose corresponding to the first camera;
calculate, based on the first camera pose and the first FOV, a first FOV mask corresponding to the first camera;
determine a second camera pose corresponding to the second camera;
calculate, based on the second camera pose and the second FOV, a second FOV mask corresponding to the second camera;
extract the first set of perspective view features based on the first set of input perspective view features, the first stochastic depth scaling function, and the first FOV mask; and
extract the second set of perspective view features based on the second set of input perspective view features, the second stochastic depth scaling function, and the second FOV mask.

### Claim 4 (depends on 3)

The apparatus of claim 3,
wherein to extract the first set of perspective view features, the processing circuitry is configured to multiply, for each spatial location of a set of spatial locations, the first set of input perspective view features, the first stochastic depth scaling function, and the first FOV mask, and
wherein to extract the second set of perspective view features, the processing circuitry is configured to multiply, for each spatial location of a set of spatial locations, the second set of input perspective view features, the second stochastic depth scaling function, and the second FOV mask.

### Claim 5 (depends on 1)

The apparatus of claim 1, wherein the image data further comprises a third set of image data collected by a third camera comprising a third FOV and a fourth set of image data collected by a fourth camera comprising a fourth FOV, and wherein the processing circuitry is further configured to:
apply the encoder to extract, from the third set of image data based on a location of a third one or more objects within the third FOV, a third set of perspective view features;
apply the encoder to extract, from the fourth set of image data based on a location of a fourth one or more objects within the fourth FOV, a fourth set of perspective view features; and
project the third set of perspective view features and the fourth set of perspective view features onto the grid to generate the set of BEV features.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein the first FOV overlaps with the second FOV, and wherein to apply the encoder to extract the first set of perspective view features and extract the second set of perspective view features, the processing circuitry is configured to capture information in the first set of perspective view features and the second set of perspective view features corresponding to one or more objects of the first one or more objects and the second one or more objects located within both of the first FOV and the second FOV.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein to project the first set of perspective view features and the second set of perspective view features onto the grid to generate the set of BEV features, the processing circuitry is configured to:
perform one or more stochastic drop actions to drop one or more features of the first set of perspective view features and the second set of perspective view features; and
transform, using a least squares operation, the first set of perspective view features and the second set of perspective view features into the set of BEV features.

### Claim 8 (depends on 1)

The apparatus of claim 1, wherein the memory is further configured to store a set of training data comprising a plurality of sets of training image data, and wherein the processing circuitry is further configured to train, based on the plurality of sets of training image data, the encoder, wherein the encoder represents a residual network comprising a set of layers.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein to train the encoder, the processing circuitry is configured to:
perform a plurality of training iterations using the plurality of sets of training image data,
wherein during each training iteration of the plurality of training iterations, the processing circuitry is configured to cause the residual network to skip one or more layers of the set of layers; and
insert the skipped layers of the set of layers into the residual network to complete training of the encoder.

### Claim 10 (depends on 9)

The apparatus of claim 9, wherein to cause the residual network to skip one or more layers of the set of layers, the processing circuitry is configured to determine whether to skip each layer of the set of layers according to a probability of skipping the layer, wherein the probability of skipping the layer is greater when the layer corresponds to one or more objects and the probability of skipping the layer is smaller when the layer corresponds to a background region.

### Claim 11 (depends on 1)

The apparatus of claim 1, wherein the processing circuitry is further configured to apply a decoder to generate an output based on the set of BEV features.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein the processing circuitry is further configured to use the output generated by the decoder to control a device based on the first one or more objects and the second one or more objects.

### Claim 13 (depends on 12)

The apparatus of claim 12,
wherein the device is a vehicle,
wherein to apply the decoder to generate the output based on the set of BEV features, the processing circuitry is configured to cause the decoder to generate the output to include information identifying one or more characteristics corresponding to each object of the first one or more objects and the second one or more objects, and
wherein to use the output generated by the decoder to control the vehicle based on the first one or more objects and the second one or more objects, the processing circuitry is configured to use the output generated by the decoder to control the vehicle based on the one or more characteristics corresponding to each object of the first one or more objects and the second one or more objects.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein the one or more characteristics corresponding to each object of the first one or more objects and the second one or more objects may include an identity of the object, a location of the object relative to the vehicle, one or more characteristics of a movement of the object, one or more actions performed by the object, or any combination thereof.

### Claim 15 (depends on 1)

The apparatus of claim 1, wherein the processing circuitry is part of an advanced driver assistance system (ADAS).

### Claim 16

A method comprising:
storing image data in a memory, wherein the image data comprises a first set of image data collected by a first camera comprising a first field of view (FOV) and a second set of image data collected by a second camera comprising a second FOV;
applying an encoder to extract, from the first set of image data based on a location of a first one or more objects within the first FOV, a first set of perspective view features;
applying the encoder to extract, from the second set of image data based on a location of a second one or more objects within the second FOV, a second set of perspective view features; and
projecting the first set of perspective view features and the second set of perspective view features onto a grid to generate a set of bird's eye view (BEV) features that provides information corresponding to the first one or more objects and the second one or more objects.

### Claim 17 (depends on 16)

The method of claim 16,
wherein applying the encoder to extract the first set of perspective view features comprises using a first stochastic depth scaling function to extract the first set of perspective view features based on a distance of each object of the first one or more objects from the first camera, and
wherein applying the encoder to extract the second set of perspective view features comprises using a second stochastic depth scaling function to extract the second set of perspective view features based on a distance of each object of the second one or more objects from the second camera.

### Claim 18 (depends on 17)

The method of claim 17, further comprising:
generating a first set of input perspective view features based on the first set of image data;
generating a second set of input perspective view features based on the second set of image data;
determining a first camera pose corresponding to the first camera;
calculating, based on the first camera pose and the first FOV, a first FOV mask corresponding to the first camera;
determining a second camera pose corresponding to the second camera;
calculating, based on the second camera pose and the second FOV, a second FOV mask corresponding to the second camera;
extracting the first set of perspective view features based on the first set of input perspective view features, the first stochastic depth scaling function, and the first FOV mask; and
extracting the second set of perspective view features based on the second set of input perspective view features, the second stochastic depth scaling function, and the second FOV mask.

### Claim 19 (depends on 18)

The method of claim 18,
wherein extracting the first set of perspective view features comprises multiplying, for each spatial location of a set of spatial locations, the first set of input perspective view features, the first stochastic depth scaling function, and the first FOV mask, and
wherein extracting the second set of perspective view features comprises multiplying, for each spatial location of a set of spatial locations, the second set of input perspective view features, the second stochastic depth scaling function, and the second FOV mask.

### Claim 20 (depends on 16)

The method of claim 16, wherein the image data further comprises a third set of image data collected by a third camera comprising a third FOV and a fourth set of image data collected by a fourth camera comprising a fourth FOV, and wherein the method further comprises:
applying the encoder to extract, from the third set of image data based on a location of a third one or more objects within the third FOV, a third set of perspective view features;
applying the encoder to extract, from the fourth set of image data based on a location of a fourth one or more objects within the fourth FOV, a fourth set of perspective view features; and
projecting the third set of perspective view features and the fourth set of perspective view features onto the grid to generate the set of BEV features.

### Claim 21 (depends on 16)

The method of claim 16, wherein the first FOV overlaps with the second FOV, and wherein applying the encoder to extract the first set of perspective view features and extract the second set of perspective view features comprises capturing information in the first set of perspective view features and the second set of perspective view features corresponding to one or more objects of the first one or more objects and the second one or more objects located within both of the first FOV and the second FOV.

### Claim 22 (depends on 16)

The method of claim 16, wherein projecting the first set of perspective view features and the second set of perspective view features onto the grid to generate the set of BEV features comprises:
performing one or more stochastic drop actions to drop one or more features of the first set of perspective view features and the second set of perspective view features; and
transforming, using a least squares operation, the first set of perspective view features and the second set of perspective view features into the set of BEV features.

### Claim 23 (depends on 16)

The method of claim 16, further comprising:
storing a set of training data comprising a plurality of sets of training image data; and
training, based on the plurality of sets of training image data, the encoder, wherein the encoder represents a residual network comprising a set of layers.

### Claim 24 (depends on 23)

The method of claim 23, wherein training the encoder comprises:
performing a plurality of training iterations using the plurality of sets of training image data,
wherein during each training iteration of the plurality of training iterations, the method comprises causing the residual network to skip one or more layers of the set of layers; and
inserting the skipped layers of the set of layers into the residual network to complete training of the encoder.

### Claim 25 (depends on 24)

The method of claim 24, wherein causing the residual network to skip one or more layers of the set of layers comprises determining whether to skip each layer of the set of layers according to a probability of skipping the layer, wherein the probability of skipping the layer is greater when the layer corresponds to one or more objects and the probability of skipping the layer is smaller when the layer corresponds to a background region.

### Claim 26 (depends on 16)

The method of claim 16, further comprising applying a decoder to generate an output based on the set of BEV features.

### Claim 27 (depends on 26)

The method of claim 26, further comprising using the output generated by the decoder to control a device based on the first one or more objects and the second one or more objects.

### Claim 28 (depends on 27)

The method of claim 27,
wherein the device is a vehicle,
wherein applying the decoder to generate the output based on the set of BEV features comprises causing the decoder to generate the output to include information identifying one or more characteristics corresponding to each object of the first one or more objects and the second one or more objects, and
wherein using the output generated by the decoder to control the vehicle based on the first one or more objects and the second one or more objects comprises using the output generated by the decoder to control the vehicle based on the one or more characteristics corresponding to each object of the first one or more objects and the second one or more objects.

### Claim 29

A computer-readable medium storing instructions that, when applied by processing circuitry, causes the processing circuitry to:
store image data in a memory, wherein the image data comprises a first set of image data collected by a first camera comprising a first field of view (FOV) and a second set of image data collected by a second camera comprising a second FOV;
apply an encoder to extract, from the first set of image data based on a location of a first one or more objects within the first FOV, a first set of perspective view features;
apply the encoder to extract, from the second set of image data based on a location of a second one or more objects within the second FOV, a second set of perspective view features; and
project the first set of perspective view features and the second set of perspective view features onto a grid to generate a set of bird's eye view (BEV) features that provides information corresponding to the first one or more objects and the second one or more objects.
