# US 9,438,947 B2 · Content annotation tool

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,438,947 B2
- Document ID: US9438947B2
- Status: Granted
- Kind: B2
- Application number: 13/874,541
- Filed: 2013-05-01
- Published: 2014-11-06
- Granted: 2016-09-06
- Earliest claimed: 2013-05-01
- Assignees: Google Inc.
- Applicants: Google Inc.
- Inventors: Henry Will Schneiderman; Michael Andrew Sipe; Marco Paglia; Mikkel Crone Köser
- Web page: https://patentgrep.com/patent/US9438947B2

## CPC

- H04N 21/278: Content descriptor database or directory service for end-user access {(details of content or meta data based information retrieval of video data in video databases G06F16/739)}
- G06F 16/748: {Hypervideo (linking data to content, e.g. by linking an URL to a video object in the context of video distribution systems H04N21/858)}
- H04N 21/4722: for requesting additional data associated with the content
- H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A content annotation tool is disclosed. In a configuration, a portion of a movie may be obtained from a database. Entities, such as an actor, background music, text, etc. may be automatically identified in the movie. A user, such as a content producer, may associate and/or provide supplemental content for an identified entity to the database. A selection of one or more automatically identified entities may be received. A database entry may be generated that links the identified entity with the supplemental content. The selected automatically identified one or more entities and/or supplemental content associated therewith may be presented to an end user.

## Claims

### Claim 1

A computer-implemented method comprising:
obtaining a portion of a movie;
automatically identifying an entity within the portion of the movie using a technique selected from the group consisting of: face recognition, voice recognition, object recognition, individual characteristic recognition, scene break recognition, object tracking, speech recognition, and audio recognition;
receiving supplemental content from a first user about the identified entity within the first movie;
subsequent to receipt of the supplemental content, receiving an update to the supplemental content about the identified entity within the movie, wherein the update to the supplemental content modifies or replaces the supplemental content;
receiving a request from a second user for information related to a location of the second user; and
responsive to the request, providing, to the second user, the information related to the location and at least one item of the supplemental content, from the update to the supplemental content, related to the location.

### Claim 2 (depends on 1)

The method of claim 1, further comprising storing a database entry that links the identified entity within the portion of the movie and the supplemental content about the identified entity wherein the database entry is based upon an identification of the portion of the movie, the supplemental content, and the identified entity to a database.

### Claim 3 (depends on 1)

The method of claim 1, further comprising tracking the entity in the movie throughout a duration that the entity exists within the portion of the movie.

### Claim 4 (depends on 1)

The method of claim 1, wherein the entity comprises an audio component of the movie.

### Claim 5 (depends on 4)

The method of claim 4, wherein the audio component is selected from the group consisting of: a song, a soundtrack, a voice, and a sound effect.

### Claim 6 (depends on 1)

The method of claim 1, wherein the entity comprises a visual component.

### Claim 7 (depends on 6)

The method of claim 6, wherein the visual component is selected from the group consisting of: a scene break, a geographic location, a face, a person, an object, a physical object, and a text.

### Claim 8 (depends on 1)

The method of claim 1, wherein the supplemental content is selected from the group consisting of: a text, an audio entity, a visual entity, a URL, a picture, an advertisement, and a location.

### Claim 9 (depends on 1)

The method of claim 1, further comprising associating a time reference with the supplemental content.

### Claim 10 (depends on 1)

The method of claim 1, wherein the supplemental content comprises a selection of one or more of the automatically identified entities.

### Claim 11

A computer-implemented method comprising:
automatically identifying an entity in a movie using a technique selected from the group consisting of: face recognition, voice recognition, object recognition, individual characteristic recognition, scene break recognition, object tracking, speech recognition, and audio recognition;
obtaining supplemental content for the entity identified in the movie;
subsequently, receiving an update to the supplemental content for the identified entity in the movie, wherein the update to the supplemental content modifies or replaces the supplemental content;
providing the movie;
receiving, from a user, a query for information related to a location of the user, wherein the query is received by an input method during playback of the movie;
retrieving, from a database, the information related to the location and at least one item of the supplemental content, of the update to the supplemental content about the identified entity, related to the location, wherein the database comprises a plurality of automatically identified entities for a plurality of movies and corresponding supplemental content; and
providing, to the user, a response to the query wherein the response includes the update to the supplemental content.

### Claim 12 (depends on 11)

The method of claim 11 further comprising tracking the entity while the entity is present in the content.

### Claim 13 (depends on 11)

The method of claim 11, wherein the query is received by an input device.

### Claim 14 (depends on 11)

The method of claim 11, wherein receipt of the query causes playback of the content to pause.

### Claim 15

A computer-implemented method comprising:
receiving supplemental content;
storing the supplemental content;
obtaining content;
automatically identifying an entity in the content using a technique selected from the group consisting of: face recognition, voice recognition, object recognition, individual characteristic recognition, scene break recognition, object tracking, speech recognition, and audio recognition;
subsequent to receipt of the supplemental content, receiving an update to the supplemental content about the identified entity within the content, wherein the update to the supplemental content modifies or replaces the supplemental content;
retrieving a portion of the update to the supplemental content based on the entity;
storing a database entry that links the identified entity with the portion of the update to the supplemental content while the entity is present in the content;
providing, to a user, a portion of the content, the portion of the content including the identified entity;
receiving, from the user, a request for information related to a location of the user and based upon the portion of the content and the identified entity; and
responsive to the request, providing, to the user, the information related to the location and at least one item of the supplemental content, of the update to the supplemental content, related to the location.

### Claim 16 (depends on 15)

The method of claim 15, wherein the supplemental content is selected from the group consisting of: a text, an audio entity, a visual entity, a URL, a picture, an advertisement, and a location.

### Claim 17 (depends on 15)

The method of claim 15, wherein the content comprises a movie or multimedia data.

### Claim 18 (depends on 15)

The method of claim 15, wherein entity comprises an audio component of the content.

### Claim 19 (depends on 17)

The method of claim 17, wherein the audio component is selected from the group consisting of: a song, a soundtrack, a voice, and a sound effect.

### Claim 20 (depends on 15)

The method of claim 15, wherein the entity comprises a visual component.

### Claim 21 (depends on 19)

The method of claim 19, wherein the visual component is selected from the group consisting of: a scene break, a geographic location, a face, a person, an object, a physical object, and a text.

### Claim 22 (depends on 15)

The method of claim 15, further comprising associating a time reference with the supplemental content.

### Claim 23 (depends on 15)

The method of claim 15, further comprising tracking the entity in the content throughout a duration that the entity exists within the content.