# US 2025/0095344 A1 · Image interpolation for multi-sensor training of feature detection models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0095344 A1
- Document ID: US20250095344A1
- Status: Published application
- Kind: A1
- Application number: 18/469,213
- Filed: 2023-09-18
- Published: 2025-03-20
- Earliest claimed: 2023-09-18
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Andreas EIDEHALL
- Web page: https://patentgrep.com/patent/US20250095344A1

## CPC

- G06T 7/38: Registration of image sequences
- G06V 10/80: Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10)
- G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403)
- G06T 3/4046: using neural networks
- G06T 7/13: Edge detection
- G06T 7/174: involving the use of two or more images
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30261: Obstacle
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Image interpolation techniques for multi-sensor training of feature detection models are disclosed. The techniques can include obtaining a detection-and-ranging (DAR) frame captured by a DAR sensor, obtaining a first image frame captured at a first time by a camera and a second image frame captured at a second time by the camera, wherein at a capture time of the DAR frame, a field of view (FOV) of the DAR sensor overlaps an FOV of the camera, and wherein the capture time of the DAR frame is between the first time and the second time, interpolating based on the first image frame and the second image frame to create an interpolated image frame, wherein a nominal capture time of the interpolated image frame corresponds to the capture time of the DAR frame, and training a feature detection model using the DAR frame and the interpolated image frame.

## Claims

### Claim 1

A method for multi-sensor training of a feature detection model, comprising:
obtaining a detection-and-ranging (DAR) frame captured by a DAR sensor;
obtaining a first image frame captured at a first time by a camera and a second image frame captured at a second time by the camera, wherein at a capture time of the DAR frame, a field-of-view (FOV) of the DAR sensor overlaps an FOV of the camera, and wherein the capture time of the DAR frame is between the first time and the second time;
interpolating based on the first image frame and the second image frame to create an interpolated image frame, wherein a nominal capture time of the interpolated image frame corresponds to the capture time of the DAR frame; and
training the feature detection model using the DAR frame and the interpolated image frame.

### Claim 2 (depends on 1)

The method of claim 1, wherein the interpolating based on the first image frame and the second image frame includes interpolating using a generative machine-learning model.

### Claim 3 (depends on 2)

The method of claim 2, wherein the generative machine-learning model is a generative adversarial network (GAN) model.

### Claim 4 (depends on 1)

The method of claim 1, wherein the interpolating based on the first image frame and the second image frame includes implementing an image capture timing gradient across a dimension in the interpolated image frame based on a DAR capture timing gradient associated with the dimension in the DAR frame.

### Claim 5 (depends on 4)

The method of claim 4, wherein the DAR capture timing gradient is associated with a motion rate of the FOV of the DAR sensor with respect to the dimension.

### Claim 6 (depends on 1)

The method of claim 1, wherein the feature detection model is a DAR-based detection model, and training the feature detection model using the DAR frame and the interpolated image frame includes:
detecting a feature in the interpolated image frame using an image-based detection model;
annotating the interpolated image frame to indicate a location of the feature in the interpolated image frame; and
training the DAR-based detection model based on the DAR frame and the annotated interpolated image frame.

### Claim 7 (depends on 1)

The method of claim 1, wherein the feature detection model is an image-based detection model, and training the feature detection model using the DAR frame and the interpolated image frame includes:
detecting a feature in the DAR frame using a DAR-based detection model;
annotating the DAR frame to indicate a location of the feature in the DAR frame; and
training the image-based detection model based on the interpolated image frame and the annotated DAR frame.

### Claim 8 (depends on 1)

The method of claim 1, further comprising detecting a feature in a joint FOV of the camera and the DAR sensor based on the DAR frame and the interpolated image frame, using a multi-sensor fusion (MSF) feature detection model.

### Claim 9 (depends on 1)

The method of claim 1, wherein the DAR sensor is a radio detection and ranging (radar) sensor.

### Claim 10 (depends on 1)

The method of claim 1, wherein the DAR sensor is a light detection and ranging (lidar) sensor.

### Claim 11 (depends on 1)

The method of claim 1, wherein the camera and the DAR sensor are sensors of a vehicle.

### Claim 12

An apparatus for multi-sensor training of a feature detection model, comprising:
at least one processor; and
at least one memory communicatively coupled with the at least one processor and storing processor-readable code that, when executed by the at least one processor, is configured to:
obtain a detection-and-ranging (DAR) frame captured by a DAR sensor;
obtain a first image frame captured at a first time by a camera and a second image frame captured at a second time by the camera, wherein at a capture time of the DAR frame, a field-of-view (FOV) of the DAR sensor overlaps an FOV of the camera, and wherein the capture time of the DAR frame is between the first time and the second time;
interpolate based on the first image frame and the second image frame to create an interpolated image frame, wherein a nominal capture time of the interpolated image frame corresponds to the capture time of the DAR frame; and
train the feature detection model using the DAR frame and the interpolated image frame.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein to interpolate based on the first image frame and the second image frame, the processor-readable code is, when executed by the at least one processor, configured to interpolate using a generative machine-learning model.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein the generative machine-learning model is a generative adversarial network (GAN) model.

### Claim 15 (depends on 12)

The apparatus of claim 12, wherein to interpolate based on the first image frame and the second image frame, the processor-readable code is, when executed by the at least one processor, configured to implement an image capture timing gradient across a dimension in the interpolated image frame based on a DAR capture timing gradient associated with the dimension in the DAR frame.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the DAR capture timing gradient is associated with a motion rate of the FOV of the DAR sensor with respect to the dimension.

### Claim 17 (depends on 12)

The apparatus of claim 12, wherein the feature detection model is a DAR-based detection model, and wherein to train the feature detection model using the DAR frame and the interpolated image frame, the processor-readable code is, when executed by the at least one processor, configured to:
detect a feature in the interpolated image frame using an image-based detection model;
annotate the interpolated image frame to indicate a location of the feature in the interpolated image frame; and
train the DAR-based detection model based on the DAR frame and the annotated interpolated image frame.

### Claim 18 (depends on 12)

The apparatus of claim 12, wherein the feature detection model is an image-based detection model, and wherein to train the feature detection model using the DAR frame and the interpolated image frame, the processor-readable code is, when executed by the at least one processor, configured to:
detect a feature in the DAR frame using a DAR-based detection model;
annotate the DAR frame to indicate a location of the feature in the DAR frame; and
train the image-based detection model based on the interpolated image frame and the annotated DAR frame.

### Claim 19 (depends on 12)

The apparatus of claim 12, wherein the processor-readable code is, when executed by the at least one processor, further configured to detect a feature in a joint FOV of the camera and the DAR sensor based on the DAR frame and the interpolated image frame, using a multi-sensor fusion (MSF) feature detection model.

### Claim 20 (depends on 12)

The apparatus of claim 12, wherein the DAR sensor is a radio detection and ranging (radar) sensor.

### Claim 21 (depends on 12)

The apparatus of claim 12, wherein the DAR sensor is a light detection and ranging (lidar) sensor.

### Claim 22 (depends on 12)

The apparatus of claim 12, wherein the camera and the DAR sensor are sensors of a vehicle.

### Claim 23

An apparatus for multi-sensor training of a feature detection model, comprising:
means for obtaining a detection-and-ranging (DAR) frame captured by a DAR sensor;
means for obtaining a first image frame captured at a first time by a camera and a second image frame captured at a second time by the camera, wherein at a capture time of the DAR frame, a field-of-view (FOV) of the DAR sensor overlaps an FOV of the camera, and wherein the capture time of the DAR frame is between the first time and the second time;
means for interpolating based on the first image frame and the second image frame to create an interpolated image frame, wherein a nominal capture time of the interpolated image frame corresponds to the capture time of the DAR frame; and
means for training the feature detection model using the DAR frame and the interpolated image frame.

### Claim 24 (depends on 23)

The apparatus of claim 23, wherein the means for interpolating based on the first image frame and the second image frame includes means for interpolating using a generative machine-learning model.

### Claim 25 (depends on 24)

The apparatus of claim 24, wherein the generative machine-learning model is a generative adversarial network (GAN) model.

### Claim 26 (depends on 23)

The apparatus of claim 23, wherein the means for interpolating based on the first image frame and the second image frame includes means for implementing an image capture timing gradient across a dimension in the interpolated image frame based on a DAR capture timing gradient associated with the dimension in the DAR frame.

### Claim 27 (depends on 23)

The apparatus of claim 23, wherein the feature detection model is a DAR-based detection model, and the means for training the feature detection model using the DAR frame and the interpolated image frame includes:
means for detecting a feature in the interpolated image frame using an image-based detection model;
means for annotating the interpolated image frame to indicate a location of the feature in the interpolated image frame; and
means for training the DAR-based detection model based on the DAR frame and the annotated interpolated image frame.

### Claim 28 (depends on 23)

The apparatus of claim 23, wherein the feature detection model is an image-based detection model, and the means for training the feature detection model using the DAR frame and the interpolated image frame includes:
means for detecting a feature in the DAR frame using a DAR-based detection model;
means for annotating the DAR frame to indicate a location of the feature in the DAR frame; and
means for training the image-based detection model based on the interpolated image frame and the annotated DAR frame.

### Claim 29 (depends on 23)

The apparatus of claim 23, further comprising means for detecting a feature in a joint FOV of the camera and the DAR sensor based on the DAR frame and the interpolated image frame, using a multi-sensor fusion (MSF) feature detection model.

### Claim 30

A non-transitory computer-readable medium storing instructions for multi-sensor training of a feature detection model, the instructions including code to:
obtain a detection-and-ranging (DAR) frame captured by a DAR sensor;
obtain a first image frame captured at a first time by a camera and a second image frame captured at a second time by the camera, wherein at a capture time of the DAR frame, a field-of-view (FOV) of the DAR sensor overlaps an FOV of the camera, and wherein the capture time of the DAR frame is between the first time and the second time;
interpolate based on the first image frame and the second image frame to create an interpolated image frame, wherein a nominal capture time of the interpolated image frame corresponds to the capture time of the DAR frame; and
train the feature detection model using the DAR frame and the interpolated image frame.
