Back to results
US 12,694,697 B2Granted
Object characterization using one or more neural networks
AssigneeNVIDIA Corporation
InventorsDaguang Xu, Xiaosong Wang, Lickkong Tam, Riddhish Bhalodia, Kevin Lu, Yuhong Wen, Ali Hatmizadeh
Life of this patent
Estimating the term…
Abstract
Apparatuses, systems, and techniques are presented to detect one or more objects in one or more images. In at least one embodiment, one or more neural networks can be used to detect one or more objects in one or more images based, at least in part, on textual descriptions of the one or more objects.
Claims31
One or more processors, comprising circuitry to use one or more neural networks to:
- determine, from a report comprising text and one or more images:
- one or more image feature embeddings from at least one image of the one or more images, at least one portion of the at least one image depicting at least a portion of at least one object of one or more objects;
- and
- one or more text feature embeddings from at least a portion of the text, wherein the at least a portion of the text describes at least one object of the one or more objects;
- determine, based on a measure of similarity between the one or more image feature embeddings and the one or more text feature embeddings, that the one or more text features-feature embeddings and the one or more image feature embeddings correspond to a same object of the at least one object, wherein the measure of similarity is a loss calculation between the one or more image feature embeddings and the one or more text feature embeddings;
- and
- identify the at least one portion of the at least one image with the one or more text feature embeddings.
4 more elements
A system comprising:
- one or more processors to use one or more neural networks to:
- determine, from a report comprising text and one or more images:
- one or more modified image feature embeddings from at least one image of the one or more images, at least one portion of the at least one image depicting at least a portion of at least one object of one or more objects;
- and
- one or more text feature embeddings from at least a portion of the text, wherein the at least a portion of the text describes at least one object of the one or more objects;
- determine, based on a measure of similarity between the one or more image feature embeddings and the one or more text feature embeddings, that the one or more text feature embeddings and the one or more image feature embeddings correspond to a same object of the at least one object, wherein the measure of similarity is a loss calculation between the one or more image feature embeddings and one or more text feature embeddings;
- and
- identify the at least one portion of the at least one image with the one or more text feature embeddings.
5 more elements
A method comprising:
- using one or more neural networks to:
- determine, from a report comprising text and one or more images:
- one or more modified image feature embeddings from at least one image of the one or more images, at least one portion of the at least one image depicting at least a portion of at least one object of one or more objects;
- and
- one or more text feature embeddings from at least a portion of the text, the at least a portion of the text describing at least one object of the one or more objects;
- determine, based on a measure of similarity between the one or more image feature embeddings and the one or more text feature embeddings, that the one or more text feature embeddings and the one or more image feature embeddings correspond to a same object of the at least one object, wherein the measure of similarity is a loss calculation between the one or more image feature embeddings and one or more text feature embeddings;
- and
- identify the at least one portion of the at least one image with the one or more text feature embeddings.
5 more elements
A machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least:
- use one or more neural networks to:
- determine, from a report comprising text and one or more images:
- one or more modified image feature embeddings from at least one image of the one or more images, at least one portion of the at least one image depicting at least a portion of at least one object of one or more objects;
- and
- one or more text feature embeddings from at least a portion of the text, wherein the at least a portion of the text describes at least one object of the one or more objects;
- determine, based on a measure of similarity between the one or more image feature embeddings and the one or more text feature embeddings, that the one or more text feature embeddings and the one or more image feature embeddings correspond to a same object of the at least one object, wherein the measure of similarity is a loss calculation between the one or more image feature embeddings and one or more text feature embeddings;
- and
- identify the at least one portion of the at least one image with the one or more text feature embeddings.
5 more elements
An object detection system, comprising:
- one or more processors to use one or more neural networks to:
- determine, from a report comprising text and one or more images:
- one or more modified image feature embeddings from at least one image of the one or more images, at least one portion of the at least one image depicting at least a portion of at least one object of one or more objects;
- and
- one or more text feature embeddings from at least a portion of the text, wherein the at least a portion of the text describes at least one object of the one or more objects;
- determine, based on a measure of similarity between the one or more image feature embeddings and the one or more text feature embeddings, that the one or more text feature embeddings and the one or more image feature embeddings correspond to a same object of the at least one object, wherein the measure of similarity is a loss calculation between the one or more image feature embeddings and one or more text feature embeddings;
- and
- identify the at least one portion of the at least one image with the one or more text feature embeddings;
- and memory for storing network parameters for the one or more neural networks.
6 more elements
Dependencies are inferred from the claim text.