# US 12,112,537 B2 · Contrastive captioning for image groups

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,537 B2
- Document ID: US12112537B2
- Status: Granted
- Kind: B2
- Application number: 18/487,183
- Filed: 2023-10-16
- Published: 2024-02-01
- Granted: 2024-10-08
- Earliest claimed: 2020-08-20
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Quan Hung Tran; Long Thanh Mai; Zhe Lin; Zhuowan Li
- Web page: https://patentgrep.com/patent/US12112537B2

## CPC

- G06V 20/30: in albums, collections or shared content, e.g. social network photos or video
- G06F 16/535: Filtering based on additional data, e.g. user or group profiles
- G06F 16/55: Clustering; Classification
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 40/205: Parsing
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 10/82: using neural networks
- G06V 20/35: {Categorising the entire scene, e.g. birthday party or wedding scene}
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A group captioning system includes computing hardware, software, and/or firmware components in support of the enhanced group captioning contemplated herein. In operation, the system generates a target embedding for a group of target images, as well as a reference embedding for a group of reference images. The system identifies information in-common between the group of target images and the group of reference images and removes the joint information from the target embedding and the reference embedding. The result is a contrastive group embedding that includes a contrastive target embedding and a contrastive reference embedding with which to construct a contrastive group embedding, which is then input to a model to obtain a group caption for the target group of images.

## Claims

### Claim 1

A method comprising:
generating a dataset on which to train a model; and
training the model on the dataset;
wherein generating the dataset comprises:
parsing each image caption, in a set of image captions corresponding to a set of training images, into a scene graph;
identifying target groups from within the set of training images, wherein each of the target groups comprises a subset of the set of training images having a shared scene graph;
identifying reference groups from within the set of training images, wherein each of the reference groups corresponds to a different one of the target groups and comprises a different subset of the set of training images having scene graphs that only partially overlap with the shared scene graph of a corresponding one of the target groups; and
generating a group caption for each of the target groups based at least on the shared scene graph for a given target group.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
generating captions for image groups using the model trained on the dataset by at least:
generating a target embedding for a group of target images;
generating a reference embedding for a group of reference images;
identifying joint information in-common between the group of target images and the group of reference images;
removing the joint information from the target embedding and the reference embedding, to produce a contrastive target embedding and a contrastive reference embedding;
generating a contrastive group embedding based at least on the contrastive target embedding and the contrastive reference embedding; and
inputting the contrastive group embedding to the model to obtain each corresponding group caption.

### Claim 3 (depends on 2)

The method of claim 2, wherein:
the model includes a self-attention layer configured to identify and promote prominent features of image groups;
generating the target embedding for the group of target images comprises submitting the group of target images to the model to obtain the target embedding; and
generating the reference embedding for the group of reference images comprises submitting the group of reference images to the model to obtain the reference embedding.

### Claim 4 (depends on 2)

The method of claim 2, further comprising:
identifying the joint information in-common between the group of target images and the group of reference images comprises generating a joint embedding for a joint group of images that includes the group of target images and the group of reference images; and
removing the joint information from the target embedding and the reference embedding comprises subtracting the joint embedding from the target embedding, to produce the contrastive target embedding, and subtracting the joint embedding from the reference embedding, to produce the contrastive reference embedding.

### Claim 5 (depends on 2)

The method of claim 2, wherein:
the group of target images comprises a corresponding subset of a corresponding set of images returned by a query;
the group of reference images comprises a different corresponding subset of the corresponding set of images returned by the query; and
the method further comprises generating a refinement to the query based on the group caption generated for the group of target images.

### Claim 6 (depends on 2)

The method of claim 2, wherein the contrastive group embedding comprises a concatenation of the contrastive target embedding and the contrastive reference embedding.

### Claim 7 (depends on 1)

The method of claim 1, wherein the dataset comprises the target groups and the reference groups, and each of the target groups of the dataset comprises each corresponding group caption for each of the target groups.

### Claim 8

One or more non-transitory computer-readable media storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
generating a dataset on which to train a model by:
parsing each image caption, in a set of image captions corresponding to a set of training images, into a scene graph;
identifying target groups from within the set of training images, wherein each of the target groups comprises a subset of the set of training images having a shared scene graph;
identifying reference groups from within the set of training images, wherein each of the reference groups corresponds to a different one of the target groups and comprises a different subset of the set of training images having scene graphs that only partially overlap with the shared scene graph of a corresponding one of the target groups; and
generating a group caption for each of the target groups based at least on the shared scene graph for a given target group; and
training the model on the dataset to generate corresponding group captions for target images.

### Claim 9 (depends on 8)

The one or more non-transitory computer-readable media of claim 8, the operations further comprising:
generating captions for image groups using the model trained on the dataset by at least:
generating a target embedding for a group of target images;
generating a reference embedding for a group of reference images;
identifying joint information in-common between the group of target images and the group of reference images;
removing the joint information from the target embedding and the reference embedding, to produce a contrastive target embedding and a contrastive reference embedding;
generating a contrastive group embedding based at least on the contrastive target embedding and the contrastive reference embedding; and
inputting the contrastive group embedding to the model to obtain each corresponding group caption.

### Claim 10 (depends on 9)

The one or more non-transitory computer-readable media of claim 9, wherein:
the model includes a self-attention layer configured to identify and promote prominent features of image groups;
generating the target embedding for the group of target images comprises submitting the group of target images to the model to obtain the target embedding; and
generating the reference embedding for the group of reference images comprises submitting the group of reference images to the model to obtain the reference embedding.

### Claim 11 (depends on 9)

The one or more non-transitory computer-readable media of claim 9, the operations further comprising:
identifying the joint information in-common between the group of target images and the group of reference images comprises generating a joint embedding for a joint group of images that includes the group of target images and the group of reference images; and
removing the joint information from the target embedding and the reference embedding comprises subtracting the joint embedding from the target embedding, to produce the contrastive target embedding, and subtracting the joint embedding from the reference embedding, to produce the contrastive reference embedding.

### Claim 12 (depends on 9)

The one or more non-transitory computer-readable media of claim 9, wherein:
the group of target images comprises a corresponding subset of a corresponding set of images returned by a query;
the group of reference images comprises a different corresponding subset of the corresponding set of images returned by the query; and
the operations further comprising generating a refinement to the query based on the group caption generated for the group of target images.

### Claim 13 (depends on 9)

The one or more non-transitory computer-readable media of claim 9, wherein the contrastive group embedding comprises a concatenation of the contrastive target embedding and the contrastive reference embedding.

### Claim 14 (depends on 8)

The one or more non-transitory computer-readable media of claim 8, wherein the dataset comprises the target groups and the reference groups, and each of the target groups of the dataset comprises each corresponding group caption for each of the target groups.

### Claim 15

A computing system comprising:
a memory component; and
one or more processors coupled to the memory component, the one or more processors configured to perform operations comprising:
generating a dataset on which to train a model by:
parsing each image caption, in a set of image captions corresponding to a set of training images, into a scene graph;
identifying target groups from within the set of training images, wherein each of the target groups comprises a subset of the set of training images having a shared scene graph;
identifying reference groups from within the set of training images, wherein each of the reference groups corresponds to a different one of the target groups and comprises a different subset of the set of training images having scene graphs that only partially overlap with the shared scene graph of a corresponding one of the target groups; and
generating a group caption for each of the target groups based at least on the shared scene graph for a given target group; and
training the model on the dataset to generate corresponding group captions for target images, the dataset comprising the target groups and the reference groups, and each of the target groups of the dataset comprises each corresponding group caption for each of the target groups.

### Claim 16 (depends on 15)

The system of claim 15, the one or more processors configured to perform operations further comprising:
generating captions for image groups using the model trained on the dataset by at least:
generating a target embedding for a group of target images;
generating a reference embedding for a group of reference images;
identifying joint information in-common between the group of target images and the group of reference images;
removing the joint information from the target embedding and the reference embedding, to produce a contrastive target embedding and a contrastive reference embedding;
generating a contrastive group embedding based at least on the contrastive target embedding and the contrastive reference embedding; and
inputting the contrastive group embedding to the model to obtain each corresponding group caption.

### Claim 17 (depends on 16)

The system of claim 16, wherein:
the model includes a self-attention layer configured to identify and promote prominent features of image groups;
generating the target embedding for the group of target images comprises submitting the group of target images to the model to obtain the target embedding; and
generating the reference embedding for the group of reference images comprises submitting the group of reference images to the model to obtain the reference embedding.

### Claim 18 (depends on 16)

The system of claim 16, the one or more processors configured to perform operations further comprising:
identifying the joint information in-common between the group of target images and the group of reference images comprises generating a joint embedding for a joint group of images that includes the group of target images and the group of reference images; and
removing the joint information from the target embedding and the reference embedding comprises subtracting the joint embedding from the target embedding, to produce the contrastive target embedding, and subtracting the joint embedding from the reference embedding, to produce the contrastive reference embedding.

### Claim 19 (depends on 16)

The system of claim 16, wherein:
the group of target images comprises a corresponding subset of a corresponding set of images returned by a query;
the group of reference images comprises a different corresponding subset of the corresponding set of images returned by the query; and
the one or more processors configured to perform operations further comprising generating a refinement to the query based on the group caption generated for the group of target images.

### Claim 20 (depends on 16)

The system of claim 16, wherein the contrastive group embedding comprises a concatenation of the contrastive target embedding and the contrastive reference embedding.