# US 12,585,865 B2 · Annotating a collection of media content items

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,585,865 B2
- Document ID: US12585865B2
- Status: Granted
- Kind: B2
- Application number: 18/677,674
- Filed: 2024-05-29
- Published: 2024-09-26
- Granted: 2026-03-24
- Earliest claimed: 2018-03-30
- Assignees: Snap Inc.
- Applicants: Snap Inc.
- Inventors: Newar Husam Al Majid; Wisam Dakka; Donald Giovannini; Andre Madeira; Seyed Reza Mir Ghaderi; Yaming Lin; Yan Wu; Ranveer Kunal; Aymeric Damien; Maryam Daneshi; Yi Liu
- Web page: https://patentgrep.com/patent/US12585865B2

## CPC

- G06F 40/169: Annotation, e.g. comment data or footnotes
- G06F 16/24578: {using ranking}
- G06F 16/41: Indexing; Data structures therefor; Storage structures
- G06F 16/48: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F 16/487: using geographical or spatial information, e.g. location
- G06F 16/489: {using time information}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Various embodiments provide for systems, methods, and computer-readable storage media for annotating a collection of media items, such as digital images. According to some embodiments, an annotation system automatically determines one or more annotations for a plurality of media content items, and generates a collection of media content items that associates the determined annotations with the plurality of media content items. Depending on the embodiment, annotations that may be determined for the plurality of media content (and associated with the collection for the media content items) can include, without limitation, a caption, a geographic location, a category, a novelty measurement, an event, and a highlight media content item representing the collection.

## Claims

### Claim 1

A method comprising:
determining, by one or more processors, an individual caption for a plurality of media content items, the determining of the individual caption comprising:
extracting, from the plurality of media content items, a set of captions associated with one or more individual media content items in the plurality of media content items;
determining a set of caption scores for the set of captions by assigning a select caption score for each select caption in the set of captions, at least one caption in the set of captions being assigned an individual caption score based on an independence of the at least one caption within the plurality of media content items by:
determining a distribution of coverage scores, the determining of the distribution of coverage scores comprising assigning, for each individual media content item in the plurality of media content items associated with a given caption that includes the at least one caption, a coverage score that measures how much of the given caption is covered by the at least one caption; and
determining the independence of the at least one caption based on the distribution of coverage scores, the independence indicating how sufficient the at least one caption is in describing a majority of the media content items in the plurality of media content items, the at least one caption comprising a phrase;
determining a ranking for the set of captions based on the set of caption scores; and
selecting, from the set of captions, the individual caption based on the ranking;
identifying, by the one or more processors, a set of visual labels for the plurality of media content items using a deep neural network model configured to identify objects or concepts appearing in visual content of the media content items;
determining, by the one or more processors, whether the plurality of media content items is associated with an event based on a similarity of at least one visual label in the set of visual labels to a given visual label of one or more other media content items associated with the event;
generating, by the one or more processors, a collection of media content items that comprises the plurality of media content items, the individual caption, and collection annotation data that associates the collection of media content items with the event in response to determining that the plurality of media content items is associated with the event; and
providing, by the one or more processors, the collection of media content items to a client device for access by a user at the client device, one or more media content items from the plurality of media content items being used to generate a graphical tile on the client device, the graphical tile being used to graphically represent the collection of media content items on the client device.

### Claim 2 (depends on 1)

The method of claim 1, wherein the individual caption score is a first caption score, and wherein at least another caption in the set of captions is assigned a second caption score based on at least one of:
a popularity of the select caption within the plurality of media content items; or
a uniqueness of the select caption within the plurality of media content items.

### Claim 3 (depends on 1)

The method of claim 1, comprising:
identifying, by the one or more processors, the plurality of media content items by grouping specific media content items based on at least one of proximity of geographic locations associated with the specific media content items, proximity of times associated with the specific media content items, topics associated with the specific media content items, media sources associated with the specific media content items, or media types associated with the specific media content items.

### Claim 4 (depends on 1)

The method of claim 1,
wherein the determining of whether the plurality of media content items is associated with the event is further based on a similarity of at least one caption, in a set of captions extracted from the plurality of media content items, to a given caption of one or more other media content items associated with the event.

### Claim 5 (depends on 1)

The method of claim 1, wherein the event is a select event, and wherein the method comprises:
determining, by the one or more processors, whether the plurality of media content items is associated with an ongoing event based on a trend of media content items being added to the plurality of media content items over a period of time, the collection of media content items comprising collection annotation data that associates the collection of media content items with the ongoing event in response to determining that the plurality of media content items is associated with the ongoing event.

### Claim 6 (depends on 1)

The method of claim 1, wherein the event is a select event, and wherein the method comprises:
determining, by the one or more processors, whether the collection of media content items is associated with a concluded event, the collection of media content items comprising collection annotation data that associates the collection of media content items with the concluded event in response to determining that the plurality of media content items is associated with the concluded event.

### Claim 7 (depends on 1)

The method of claim 1, wherein the assigning of the select caption score for the select caption is based on a popularity of the select caption within the plurality of media content items, the popularity being determined based on at least one of:
a number of media content items in the plurality of media content items that are associated with the select caption; or
a number of users providing one or more media content items in the plurality of media content items who have used the select caption with respect to media content items not in the plurality of media content items.

### Claim 8

A system comprising:
one or more processors; and
one or more machine-readable mediums storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
determining an individual caption for a plurality of media content items, the determining of the individual caption comprising:
extracting, from the plurality of media content items, a set of captions associated with one or more individual media content items in the plurality of media content items;
determining a set of caption scores for the set of captions by assigning a select caption score for each select caption in the set of captions, at least one caption in the set of captions being assigned an individual caption score based on an independence of the at least one caption within the plurality of media content items, the independence indicating how sufficient the at least one caption is in describing a majority of the media content items in the plurality of media content items by:
determining a distribution of coverage scores, the determining of the distribution of coverage scores comprising assigning, for each individual media content item in the plurality of media content items associated with a given caption that includes the at least one caption, a coverage score that measures how much of the given caption is covered by the at least one caption; and
determining the independence of the at least one caption based on the distribution of coverage scores, the at least one caption comprising a phrase;
determining a ranking for the set of captions based on the set of caption scores; and
selecting, from the set of captions, the individual caption based on the ranking;
identifying a set of visual labels for the plurality of media content items using a deep neural network model configured to identify objects or concepts appearing in visual content of the media content items;
determining whether the plurality of media content items is associated with an event based on a similarity of at least one visual label in the set of visual labels to a given visual label of one or more other media content items associated with the event;
generating a collection of media content items that comprises the plurality of media content items, the individual caption, and collection annotation data that associates the collection of media content items with the event in response to determining that the plurality of media content items is associated with the event; and
providing the collection of media content items to a client device for access by a user at the client device, one or more media content items from the plurality of media content items being used to generate a graphical tile on the client device, the graphical tile being used to graphically represent the collection of media content items on the client device.

### Claim 9 (depends on 8)

The system of claim 8, wherein the individual caption score is a first caption score, and wherein at least another caption in the set of captions is assigned a second caption score based on at least one of:
a popularity of the select caption within the plurality of media content items; or
a uniqueness of the select caption within the plurality of media content items.

### Claim 10 (depends on 8)

The system of claim 8, wherein the operations comprise:
identifying the plurality of media content items by grouping specific media content items based on at least one of proximity of geographic locations associated with the specific media content items, proximity of times associated with the specific media content items, topics associated with the specific media content items, media sources associated with the specific media content items, or media types associated with the specific media content items.

### Claim 11 (depends on 8)

The system of claim 8, wherein
the determining of whether the plurality of media content items is associated with the event is further based on
a similarity of at least one caption, in a set of captions extracted from the plurality of media content items, to a given caption of one or more other media content items associated with the event.

### Claim 12 (depends on 8)

The system of claim 8, wherein the event is a select event, and wherein the operations comprise:
determining whether the plurality of media content items is associated with an ongoing event based on a trend of media content items being added to the plurality of media content items over a period of time, the collection of media content items comprising collection annotation data that associates the collection of media content items with the ongoing event in response to determining that the plurality of media content items is associated with the ongoing event.

### Claim 13 (depends on 8)

The system of claim 8, wherein the event is a select event, and wherein the operations comprise:
determining whether the collection of media content items is associated with a concluded event, the collection of media content items comprising collection annotation data that associates the collection of media content items with the concluded event in response to determining that the plurality of media content items is associated with the concluded event.

### Claim 14 (depends on 8)

The system of claim 8, wherein the assigning of the select caption score for the select caption is based on a popularity of the select caption within the plurality of media content items, the popularity being determined based on at least one of:
a number of media content items in the plurality of media content items that are associated with the select caption; or
a number of users providing one or more media content items in the plurality of media content items who have used the select caption with respect to media content items not in the plurality of media content items.

### Claim 15

A non-transitory computer-readable medium storing instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform operations comprising:
determining an individual caption for a plurality of media content items, the determining of the individual caption comprising:
extracting, from the plurality of media content items, a set of captions associated with one or more individual media content items in the plurality of media content items;
determining a set of caption scores for the set of captions by assigning a select caption score for each select caption in the set of captions, at least one caption in the set of captions being assigned an individual caption score based on an independence of the at least one caption within the plurality of media content items, the independence indicating how sufficient the at least one caption is in describing a majority of the media content items in the plurality of media content items by:
determining a distribution of coverage scores, the determining of the distribution of coverage scores comprising assigning, for each individual media content item in the plurality of media content items associated with a given caption that includes the at least one caption, a coverage score that measures how much of the given caption is covered by the at least one caption; and
determining the independence of the at least one caption based on the distribution of coverage scores, the at least one caption comprising a phrase;
determining a ranking for the set of captions based on the set of caption scores; and
selecting, from the set of captions, the individual caption based on the ranking;
identifying a set of visual labels for the plurality of media content items using a deep neural network model configured to identify objects or concepts appearing in visual content of the media content items;
determining whether the plurality of media content items is associated with an event based on a similarity of at least one visual label in the set of visual labels to a given visual label of one or more other media content items associated with the event;
generating a collection of media content items that comprises the plurality of media content items, the individual caption, and collection annotation data that associates the collection of media content items with the event in response to determining that the plurality of media content items is associated with the event; and
providing the collection of media content items to a client device for access by a user at the client device, one or more media content items from the plurality of media content items being used to generate a graphical tile on the client device, the graphical tile being used to graphically represent the collection of media content items on the client device.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the individual caption score is a first caption score, and wherein at least another caption in the set of captions is assigned a second caption score based on at least one of:
a popularity of the select caption within the plurality of media content items; or
a uniqueness of the select caption within the plurality of media content items.

### Claim 17 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the operations comprise:
identifying the plurality of media content items by grouping specific media content items based on at least one of proximity of geographic locations associated with the specific media content items, proximity of times associated with the specific media content items, topics associated with the specific media content items, media sources associated with the specific media content items, or media types associated with the specific media content items.

### Claim 18 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein
the determining of whether the plurality of media content items is associated with the event is further based on
a similarity of at least one caption, in a set of captions extracted from the plurality of media content items, to a given caption of one or more other media content items associated with the event.

### Claim 19 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the event is a select event, and wherein the operations comprise:
determining whether the plurality of media content items is associated with an ongoing event based on a trend of media content items being added to the plurality of media content items over a period of time, the collection of media content items comprising collection annotation data that associates the collection of media content items with the ongoing event in response to determining that the plurality of media content items is associated with the ongoing event.

### Claim 20 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the event is a select event, and wherein the operations comprise:
determining whether the collection of media content items is associated with a concluded event, the collection of media content items comprising collection annotation data that associates the collection of media content items with the concluded event in response to determining that the plurality of media content items is associated with the concluded event.
