# US 2018/0322103 A1 · Extracting audiovisual features from digital components

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0322103 A1
- Document ID: US20180322103A1
- Status: Published application
- Kind: A1
- Application number: 15/638,304
- Filed: 2017-06-29
- Published: 2018-11-08
- Earliest claimed: 2011-11-14
- Assignees: Google Inc.
- Applicants: Google Inc.
- Inventors: Boon-Lock Yeo; Xuemei GU; Gangjiang Li; Gaurav Bhaya; Robert Stets
- Web page: https://patentgrep.com/patent/US20180322103A1

## CPC

- G06F 40/279: Recognition of textual entities
- G06F 16/434: {using image data, e.g. images, photos, pictures taken by a user}
- G06F 16/583: using metadata automatically derived from the content
- G06F 40/134: Hyperlinking
- G06V 40/1365: {Matching; Classification}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06V 10/40: Extraction of image or video features
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods for extracting audiovisual features from images and other digital components. A data processing system can extract image data and image features from an input image. The data processing system can match the image features to the image features of a plurality of image to identify candidate images. A second image can be selected from the candidate images based on a request that the data processing system received with the input image.

## Claims

### Claim at position 1

-20. (canceled)

### Claim 21

A system to extract image features from input requests, comprising:
a recognition engine executed by a data processing system to:
receive, from a computing device, a first request comprising a first image;
retrieve, from a data repository, content data for each of a plurality of content items, the content data comprising image features;
extract an image feature from the first image;
select candidate content items from the plurality of content items based on matches between the image features of the plurality of content items and the image feature from the first image;
select a content item from the candidate content items based on the request; and
a network interface of the data processing system to transmit the content item to the computing device.

### Claim 22 (depends on 21)

The system of claim 21, wherein the first image is captured by a camera associated with the first computing device.

### Claim 23 (depends on 21)

The system of claim 21, wherein the first request comprises an audio-based input signal.

### Claim 24 (depends on 21)

The system of claim 21, comprising the recognition engine to:
apply an image feature detection to the first image to extract the image feature; and
apply the image feature detection to each of the plurality of content items.

### Claim 25 (depends on 21)

The system of claim 21, comprising the recognition engine to:
extract a plurality of image features from the first image;
extract a second plurality of image features from each of the plurality of content items; and
select the candidate content items from the plurality of content items by matching a first predetermined number of the plurality of image features with a second predetermined number of the second plurality of image features.

### Claim 26 (depends on 21)

The system of claim 21, comprising:
a natural language processor component executed by the data processing system to:
receive data packets comprising the first request;
parse the request to identify a trigger keyword corresponding to the request; and
the recognition engine to select the content item from the candidate content items based on the trigger keyword.

### Claim 27 (depends on 21)

The system of claim 21, comprising:
a direct action application programming interface executed by the data processing system to:
generate an action data structure based on the request;
transmit the action data structure to a service provider computing device to cause the service provider computing device to invoke a conversational application programming interface and establish a communication session between the service provider computing device and the first computing device; and
receive an indication that the service provider computing device established the communication session with the first computing device.

### Claim 28 (depends on 21)

The system of claim 21, comprising:
a natural language processor component executed by the data processing system to:
parse the request to identify a trigger keyword corresponding to the request;
the data processing system to:
select a template based on the trigger keyword; and
populate a field of the template with the first image.

### Claim 29 (depends on 28)

The system of claim 28, comprising the data processing system to:
request a value from a sensor associated with the computing device; and
populate a second field of the template with the value.

### Claim 30 (depends on 28)

The system of claim 28, comprising:
the data processing system configured to generate, based on the template, an action data structure.

### Claim 31

A method to extract image features from input requests, comprising:
receiving, from a computing device by a data processing system, a first request comprising a first image;
retrieving, from a data repository by the data processing system, content data for each of a plurality of content items, the content data comprising image features;
extracting, by a recognition engine executed by the data processing system, an image feature from the first image;
selecting, by the data processing system, candidate content items from the plurality of content items by determining matches between the image features of the plurality of content items and the image feature from the first image;
selecting, by the data processing system, a content item from the candidate content items based on the request; and
transmitting, via a network interface, the content item to the computing device.

### Claim 32 (depends on 31)

The method of claim 31, wherein the first image is captured by a camera associated with the first computing device.

### Claim 33 (depends on 31)

The method of claim 31, wherein the first request comprises an audio-based input signal.

### Claim 34 (depends on 31)

The method of claim 31, comprising:
applying an image feature detection to the first image to extract the image feature; and
applying the image feature detection to each of the plurality of content items.

### Claim 35 (depends on 31)

The method of claim 31, comprising:
extracting, by the recognition engine, a plurality of image features from the first image;
extracting, by the recognition engine, a second plurality of image features from each of the plurality of content items; and
selecting, by the data processing system, the candidate content items from the plurality of content items by matching a first predetermined number of the plurality of image features with a second predetermined number of the second plurality of image features.

### Claim 36 (depends on 31)

The method of claim 31, comprising:
receiving, by a natural language processor component executed by the data processing system, data packets comprising the first request;
parsing, by the natural language processor component, the request to identify a trigger keyword corresponding to the request; and
selecting, by the data processing system, the content item from the candidate content items based on the trigger keyword.

### Claim 37 (depends on 31)

The method of claim 31, comprising:
generating, by a direct action application programming interface, an action data structure based on the request;
transmitting, by the direct action application programming interface, the action data structure to a service provider computing device to cause the service provider computing device to invoke a conversational application programming interface and establish a communication session between the service provider computing device and the first computing device; and
receiving, by the data processing system, an indication that the service provider computing device established the communication session with the first computing device.

### Claim 38 (depends on 31)

The method of claim 31, comprising:
parsing, by a natural language processor component, the request to identify a trigger keyword corresponding to the request;
selecting, by the data processing system, a template based on the trigger keyword; and
populating, by the data processing system, a field of the template with the first image.

### Claim 39 (depends on 38)

The method of claim 38, comprising:
requesting, by the data processing system from the computing device, a value from a sensor associated with the computing device; and
populating, by the data processing system, a second field of the template with the value.

### Claim 40 (depends on 38)

The method of claim 38, comprising:
generating, based on the template, an action data structure.