Back to results
US 12,711,763 B2Granted
Methods and apparatus for visual-aware hierarchy-based object recognition
AssigneeEli Lilly and Company
InventorsEdward John Delp, III, Jiangpeng He, Runyu Mao, Zeman Shao, Sri Kalyan Yarlagadda, Fengqing Maggie Zhu
Life of this patent
Estimating the term…
Abstract
The techniques described herein relate to computerized methods and apparatus for grouping images of objects based on semantic and visual information associated with the objects. The techniques described herein further relate to computerized methods and apparatus for training a machine learning model for object recognition.
Claims21
A computerized method for training a machine learning model for recognizing objects based on semantic and visual information associated with the objects, the method comprising:
- accessing a set of digital images, each digital image comprising an object associated with a semantic category of the object, wherein the semantic category is one of a plurality of semantic categories associated with the set of digital images, and wherein each digital image is labeled with data indicative of the semantic category to which the digital image is associated;
- determining, using a trained machine learning model, a visual feature vector for each digital image of the set of digital images, wherein the visual feature vector comprises a value for each of a plurality of visual features;
- for each semantic category of the plurality of semantic categories, calculating, for each visual feature of the plurality of visual features, a separate probability distribution of the values of the visual feature in the visual feature vectors of each image in the set of digital images associated with the semantic category;
- determining, based on the probability distributions, data indicative of similarities between digital images associated with different semantic categories of the plurality of semantic categories;
- grouping, based on the data indicative of similarities between the digital images associated with different semantic categories, the plurality of semantic categories into clusters of semantic categories;
- classifying, using the machine learning model, each digital image of the set of digital images, said classifying comprising:
- determining, using a first output layer of the machine learning model, a predicted semantic category of the plurality of semantic categories for said digital image, and
- determining, using a second output layer of the machine learning model, a predicted cluster for said digital image;
- and
- training the machine learning model using a loss function, wherein the loss function incorporates data indicative of (i) whether the predicted semantic category for each digital image corresponds to the semantic category associated with said digital image and (ii) whether the predicted cluster for said digital image corresponds to the cluster into which the semantic category associated with said digital image was grouped.
7 more elements
A non-transitory computer-readable media comprising instructions for training a machine learning model for recognizing objects based on semantic and visual information associated with the objects, wherein the instructions, when executed by one or more processors on a computing device, are operable to cause the one or more processors to:
- access a set of digital images, each digital image comprising an object associated with a semantic category of the object, wherein the semantic category is one of a plurality of semantic categories associated with the set of digital images, and wherein each digital image is labeled with data indicative of the semantic category to which the digital image is associated;
- determine, using a trained machine learning model, a visual feature vector for each digital image of the set of digital images, wherein the visual feature vector comprises a value for each of a plurality of visual features;
- for each semantic category of the plurality of semantic categories, calculate, for each visual feature of the plurality of visual features, a separate probability distribution of the values of the visual feature in the visual feature vectors of each image in the set of digital images associated with the semantic category;
- determine, based on the probability distributions, data indicative of similarities between digital images associated with different semantic categories of the plurality of semantic categories;
- group, based on the data indicative of similarities between the digital images associated with different semantic categories, the plurality of semantic categories into clusters of semantic categories;
- classify, using the machine learning model, each digital image of the set of digital images, said classifying comprising:
- determining, using a first output layer of the machine learning model, a predicted semantic category of the plurality of semantic categories for said digital image, and
- determining, using a second output layer of the machine learning model, a predicted cluster for said digital image;
- and
- training the machine learning model using a loss function, wherein the loss function incorporates data indicative of (i) whether the predicted semantic category for each digital image corresponds to the semantic category associated with said digital image and (ii) whether the predicted cluster for said digital image corresponds to the cluster into which the semantic category associated with said digital image was grouped.
7 more elements
A system for training a machine learning model for recognizing objects based on semantic and visual information associated with the objects, the system comprising:
- a memory storing instructions;
- and
- a processor configured to execute the instructions to:
- access a set of digital images, each digital image comprising an object associated with a semantic category of the object, wherein the semantic category is one of a plurality of semantic categories associated with the set of digital images, and wherein each digital image is labeled with data indicative of the semantic category to which the digital image is associated;
- determine, using a trained machine learning model, a visual feature vector for each digital image of the set of digital images, wherein the visual feature vector comprises a value for each of a plurality of visual features;
- for each semantic category of the plurality of semantic categories, calculate, for each visual feature of the plurality of visual features, a separate probability distribution of the values of the visual feature in the visual feature vectors of each image in the set of digital images associated with the semantic category;
- determine, based on the probability distributions, data indicative of similarities between digital images associated with different semantic categories of the plurality of semantic categories;
- group, based on the data indicative of similarities between the digital images associated with different semantic categories, the plurality of semantic categories into clusters of semantic categories;
- classify, using the machine learning model, each digital image of the set of digital images, said classifying comprising:
- determining, using a first output layer of the machine learning model, a predicted semantic category of the plurality of semantic categories for said digital image, and
- determining, using a second output layer of the machine learning model, a predicted cluster for said digital image;
- and
- training the machine learning model using a loss function, wherein the loss function incorporates data indicative of (i) whether the predicted semantic category for each digital image corresponds to the semantic category associated with said digital image and (ii) whether the predicted cluster for said digital image corresponds to the cluster into which the semantic category associated with said digital image was grouped.
10 more elements
Dependencies are inferred from the claim text.