US 12,112,537 B2 · Contrastive captioning for image groups

Abstract

A group captioning system includes computing hardware, software, and/or firmware components in support of the enhanced group captioning contemplated herein. In operation, the system generates a target embedding for a group of target images, as well as a reference embedding for a group of reference images. The system identifies information in-common between the group of target images and the group of reference images and removes the joint information from the target embedding and the reference embedding. The result is a contrastive group embedding that includes a contrastive target embedding and a contrastive reference embedding with which to construct a contrastive group embedding, which is then input to a model to obtain a group caption for the target group of images.

Claims20

Dependent claims

A method comprising:

generating a dataset on which to train a model;
and
training the model on the dataset;
wherein generating the dataset comprises:
parsing each image caption, in a set of image captions corresponding to a set of training images, into a scene graph;
identifying target groups from within the set of training images, wherein each of the target groups comprises a subset of the set of training images having a shared scene graph;
identifying reference groups from within the set of training images, wherein each of the reference groups corresponds to a different one of the target groups and comprises a different subset of the set of training images having scene graphs that only partially overlap with the shared scene graph of a corresponding one of the target groups;
and
generating a group caption for each of the target groups based at least on the shared scene graph for a given target group.

6 more elements

One or more non-transitory computer-readable media storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:

generating a dataset on which to train a model by:
parsing each image caption, in a set of image captions corresponding to a set of training images, into a scene graph;
identifying target groups from within the set of training images, wherein each of the target groups comprises a subset of the set of training images having a shared scene graph;
identifying reference groups from within the set of training images, wherein each of the reference groups corresponds to a different one of the target groups and comprises a different subset of the set of training images having scene graphs that only partially overlap with the shared scene graph of a corresponding one of the target groups;
and
generating a group caption for each of the target groups based at least on the shared scene graph for a given target group;
and
training the model on the dataset to generate corresponding group captions for target images.

5 more elements

A computing system comprising:

a memory component;
and
one or more processors coupled to the memory component, the one or more processors configured to perform operations comprising:
generating a dataset on which to train a model by:
parsing each image caption, in a set of image captions corresponding to a set of training images, into a scene graph;
identifying target groups from within the set of training images, wherein each of the target groups comprises a subset of the set of training images having a shared scene graph;
identifying reference groups from within the set of training images, wherein each of the reference groups corresponds to a different one of the target groups and comprises a different subset of the set of training images having scene graphs that only partially overlap with the shared scene graph of a corresponding one of the target groups;
and
generating a group caption for each of the target groups based at least on the shared scene graph for a given target group;
and
training the model on the dataset to generate corresponding group captions for target images, the dataset comprising the target groups and the reference groups, and each of the target groups of the dataset comprises each corresponding group caption for each of the target groups.

8 more elements

Dependencies are inferred from the claim text.

Contrastive captioning for image groups

Life of this patent

Contrastive captioning for image groups

Life of this patent

Abstract

Claims20