# US 2025/0148757 A1 · Self-improving data engine for autonomous vehicles

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0148757 A1
- Document ID: US20250148757A1
- Status: Published application
- Kind: A1
- Application number: 18/931,681
- Filed: 2024-10-30
- Published: 2025-05-08
- Earliest claimed: 2023-11-02
- Assignees: NEC Laboratories America, Inc.
- Applicants: NEC Laboratories America, Inc.
- Inventors: Jong-Chyi Su; Sparsh Garg; Samuel Schulter; Manmohan Chandraker; Mingfu Liang
- Web page: https://patentgrep.com/patent/US20250148757A1

## CPC

- G06V 10/764: using classification, e.g. of video objects
- B60W 60/001: {Planning or execution of driving tasks}
- G06T 11/00: Two-dimensional [2D] image generation
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems and methods for a self-improving data engine for autonomous vehicles is presented. To train the self-improving data engine for autonomous vehicles (SIDE), multi-modality dense captioning (MMDC) models can detect unrecognized classes from diversified descriptions for input images. A vision-language-model (VLM) can generate textual features from the diversified descriptions and image features from corresponding images to the diversified descriptions. Curated features, including curated textual features and curated image features, can be obtained by comparing similarity scores between the textual features and top-ranked image features based on their likelihood scores. Generate annotations, including bounding boxes and labels, can be generated for the curated features by comparing the similarity scores of labels generated by a zero-shot classifier and the curated textual features. The SIDE can be trained using the curated features, annotations, and feedback.

## Claims

### Claim 1

A computer-implemented method for training a self-improving data engine for autonomous vehicles (SIDE), comprising:
detecting unrecognized classes from diversified descriptions for input images generated using a multi-modality dense captioning (MMDC) model;
generating, with a vision-language-model (VLM), textual features from the diversified descriptions and image features from corresponding images to the diversified descriptions;
obtaining curated features, including curated textual features and curated image features, by comparing similarity scores between the textual features and top-ranked image features based on their likelihood scores;
generating annotations, including bounding boxes and labels, for the curated features by comparing the similarity scores of labels generated by a zero-shot classifier and the curated textual features; and
training the SIDE using the curated features, annotations, and feedback.

### Claim 2 (depends on 1)

The computer-implemented of claim 1, further comprising generating trajectories within a traffic scene simulation to control an autonomous vehicle using the trained SIDE.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising generating diverse traffic simulations using the VLM to verify that the trained SIDE can detect previously unrecognized classes.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein the diverse traffic simulations further includes new unrecognized classes to be detected by the trained SIDE.

### Claim 5 (depends on 1)

The computer-implemented of claim 1, wherein generating the textual features and the image features further comprises generating photorealistic synthetic images that align with the textual features using a generative model.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein generating the annotations further comprises combining a base label space including the curated features with existing datasets to include objects likely present on road scenes for zero-shot classification.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein training the SIDE further comprises retaining previously learned knowledge by employing pseudo-labels for known categories to continuously train the SIDE.

### Claim 8

A system for training a self-improving data engine for autonomous vehicles (SIDE), comprising:
a memory device;
one or more processor devices operatively coupled with the memory device to:
detect unrecognized classes from diversified descriptions for input images generated using a multi-modality dense captioning (MMDC) model;
generate, with a vision-language-model (VLM), textual features from the diversified descriptions and image features from corresponding images to the diversified descriptions;
obtain curated features, including curated textual features and curated image features, by comparing similarity scores between the textual features and top-ranked image features based on their likelihood scores;
generate annotations, including bounding boxes and labels, for the curated features by comparing the similarity scores of labels generated by a zero-shot classifier and the curated textual features; and
training the SIDE using the curated features, annotations, and feedback.

### Claim 9 (depends on 8)

The system of claim 8, further comprising to generate trajectories within a traffic scene simulation to control an autonomous vehicle using the trained SIDE.

### Claim 10 (depends on 8)

The system of claim 8, further comprising to generate diverse traffic simulations using the VLM to verify that the trained SIDE can detect previously unrecognized classes.

### Claim 11 (depends on 10)

The system of claim 10, wherein the diverse traffic simulations further includes new unrecognized classes to be detected by the trained SIDE.

### Claim 12 (depends on 8)

The system of claim 8, wherein to generate the textual features and the image features further comprises generating photorealistic synthetic images that align with the textual features using a generative model.

### Claim 13 (depends on 8)

The system of claim 8, wherein to generate the annotations further comprises to combine a base label space including the curated features with existing datasets to include objects likely present on road scenes for zero-shot classification.

### Claim 14 (depends on 8)

The system of claim 8, wherein to train the SIDE further comprises retaining previously learned knowledge by employing pseudo-labels for known categories to continuously train the SIDE.

### Claim 15

A non-transitory computer program product comprising a computer-readable storage medium including program code for training a self-improving data engine for autonomous vehicles (SIDE), wherein the program code when executed on a computer causes the computer to:
detect unrecognized classes from diversified descriptions for input images generated using a multi-modality dense captioning (MMDC) model;
generate, with a vision-language-model (VLM), textual features from the diversified descriptions and image features from corresponding images to the diversified descriptions;
obtain curated features, including curated textual features and curated image features, by comparing similarity scores between the textual features and top-ranked image features based on their likelihood scores;
generate annotations, including bounding boxes and labels, for the curated features by comparing the similarity scores of labels generated by a zero-shot classifier and the curated textual features; and
training the SIDE using the curated features, annotations, and feedback.

### Claim 16 (depends on 15)

The non-transitory computer program product of claim 15, further comprising to generate trajectories within a traffic scene simulation to control an autonomous vehicle using the trained SIDE.

### Claim 17 (depends on 15)

The non-transitory computer program product of claim 15, further comprising to generate diverse traffic simulations using the VLM to verify that the trained SIDE can detect previously unrecognized classes.

### Claim 18 (depends on 15)

The non-transitory computer program product of claim 15, wherein to generate the textual features and the image features further comprises generating photorealistic synthetic images that align with the textual features using a generative model.

### Claim 19 (depends on 15)

The non-transitory computer program product of claim 15, wherein to generate the annotations further comprises to combine a base label space including the curated features with existing datasets to include objects likely present on road scenes for zero-shot classification.

### Claim 20 (depends on 15)

The non-transitory computer program product of claim 15, wherein to train the SIDE further comprises retaining previously learned knowledge by employing pseudo-labels for known categories to continuously train the SIDE.
