US 2024/0203085 A1Published application

System and method of bridging the gap between object and image-level representations for open-vocabulary detection

AssigneeMohamed bin Zayed University of Artificial Intelligence

InventorsHanoona Abdul Rasheed BANGALATH, Muhammad MAAZ, Muhammad Uzair KHATTAK, Salman KHAN, Fahad Shahbaz KHAN

Life of this patent

Estimating the term…

Loading patent…

Abstract

An object detection system and method in which a machine learning engine is configured with a region-based knowledge distillation stage that generates region embeddings from a training image having bounding boxes. A linear layer learns a region-level vision-language mapping for projecting feature embeddings from the training image to a common feature space shared by text embeddings to obtain the region embeddings. An image-level supervision stage generates pseudo-box labels for a classification training image and region embeddings from the training image having bounding boxes and corresponding class labels and the classification training image having an image-level label as input. Pseudo-box labels are determined on the classification training image as an image-level vision-language mapping. A weight transfer function conditions the image-level vision-language mapping on the learned region-level vision-language mapping. A trained object detector outputs a newly captured image annotated with a bounding box for a novel object.

Claims20

Dependent claims

An object detection system, comprising:

at least one camera configured to capture at least one image;
processing circuitry of a machine learning engine configured with
an object vocabulary training component having a region-based knowledge distillation stage and an image-level supervision stage joined by a weight transfer function, and an object detection inference component determined with the object vocabulary training component;
the region-based knowledge distillation stage configured to generate region embeddings from a training image having at least one bounding box for an object as input, wherein the region-based knowledge distillation stage includes:
a linear layer configured to learn a region-level vision-language mapping for projecting feature embeddings from the training image to a common feature space shared by text embeddings to obtain the region embeddings, and
a loss function configured to optimize alignment between the region embeddings and the text embeddings;
the image-level supervision stage configured to generate pseudo-box labels for the classification training image and region embeddings with the training image having bounding boxes and corresponding class labels and a classification training image having an image-level label as input, wherein the image-level supervision stage includes:
a computing component configured to determine a plurality of pseudo-box labels on the classification training image as an image-level vision-language mapping, and to compute region embeddings for the plurality of pseudo-box labels;
the weight transfer function conditions the image-level vision-language mapping on the learned region-level vision-language mapping;
and
the object detection inference component configured to output the captured image annotated with a bounding box for a novel object that was not included in the training image having bounding boxes by performing object detection on the captured image.

8 more elements

A non-transitory computer readable storage medium storing program instructions, which when executed by processing circuitry of a machine learning engine including an object vocabulary training component having a region-based knowledge distillation stage and an image-level supervision stage, which are joined by a weight transfer function, and an object detection inference component determined with the object vocabulary training component, the processing circuitry performs a method comprising:

generating region embeddings, by the region-based knowledge distillation stage, from a training image having at least one bounding box for an object as input, including
learning a region-level vision-language mapping for projecting feature embeddings from the training image to a common feature space shared by text embeddings to obtain the region embeddings, and
optimizing, by a loss function, alignment between the region embeddings and the text embeddings:
generating pseudo-box labels for a classification training image and the region embeddings, by the image-level supervision stage, from the training image having bounding boxes and corresponding class labels and the classification training image having an image-level label as input, including
determining a plurality of pseudo-box labels on the classification training image as an image-level vision-language mapping, and computing region embeddings for the plurality of pseudo-box labels;
conditioning the image-level vision-language mapping on the learned region-level vision-language mapping by the weight transfer function;
and
outputting the captured image annotated with a bounding box for a novel object that was not included in the training image having bounding boxes by performing object detection on the captured image.

5 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/085,048Filed 20 Dec 2022
Published: 20 Jun 2024
Term ends: Not estimatedThe source records don’t support an estimate.

Classification9 codes

G06V 10/25
Image preprocessing: determination of region of interest [ROI] or a volume of interest [VOI]
G06T 1/0021
General purpose image data processing: image watermarking
G06V 10/70
Arrangements for image or video recognition or understanding: using pattern recognition or machine learning
G06V 10/82
Using pattern recognition or machine learning: using neural networks

Family

US 12,288,372 B2Related patent
US 2024/0203085 A1This patent

From the US filing history. Foreign family members aren’t listed.

Timeline

Citations

Open elsewhere

Building an agent or pulling many patents? Use the API or MCP server instead of scraping pages. Get an API key