# US 2023/0409632 A1 · Systems and methods for using conjunctions in a voice input to cause a search application to wait for additional inputs

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0409632 A1
- Document ID: US20230409632A1
- Status: Published application
- Kind: A1
- Application number: 18/239,401
- Filed: 2023-08-29
- Published: 2023-12-21
- Earliest claimed: 2020-01-07
- Assignees: Rovi Guides, Inc.
- Applicants: Rovi Guides, Inc.
- Inventors: Susanto Sen; Charishma Chundi
- Web page: https://patentgrep.com/patent/US20230409632A1

## CPC

- G06F 16/632: Query formulation
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 16/433: {using audio data}
- G06F 16/438: Presentation of query results
- G06F 18/25: Fusion techniques
- G06V 10/80: Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10)
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G10L 15/19: Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G10L 2015/223: {Execution procedure of a spoken command}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A search is performed based on a voice input combined with user selection of entities displayed on a display screen as well as real-world entities. A voice input is received from the user by a media device, as well as a selection of a first entity being displayed on the media device. A conjunction spoken in the voice input triggers the media device to wait for selection of a second entity before performing the search. After receiving selection of the second entity, a search query is constructed based on the voice input, the first entity, and the second entity. The search query is transmitted to a database and, in response, the media device receives at least one identifier of a least one content item. The at least one identifier is then generated for display to the user.

## Claims

### Claim at position 1

-50. (canceled)

### Claim 51

A computer-implemented method, comprising:
receiving input from a user via a user input interface of a media device;
processing the input to identify a particular pronoun;
identifying a gesture made by the user;
determining a plurality of identities of a plurality of entities, respectively, in an image associated with the gesture;
determining, based on the plurality of identities, a respective pronoun for each entity of the plurality of entities;
determining, from among the plurality of entities, a particular entity having a pronoun that corresponds to the particular pronoun identified based on the input, wherein the particular entity is not being displayed on a display of the media device;
querying a database based on the input and the particular entity;
based on querying the database, receiving at least one identifier of at least one content item; and
generating for presentation, using the media device, the at least one identifier of the at least one content item.

### Claim 52 (depends on 51)

The method of claim 51, wherein the input is a voice input, the particular entity is a second entity, and the method further comprises:
receiving, at the media device, a selection of a first entity currently being displayed on the display of the media device; and
processing the voice input to identify a search operator,
wherein querying the database comprises constructing a search query based on the identified search operator, the first entity and the second entity.

### Claim 53 (depends on 51)

The method of claim 51, wherein the image is captured by an imaging sensor, and identifying the gesture made by the user comprises determining, based on the image, motion of the user.

### Claim 54 (depends on 51)

The method of claim 51, wherein the image is captured by an imaging sensor, and determining the particular entity associated with the gesture further comprises:
determining a direction of the gesture; and
determining that the direction of the gesture corresponds to the image, the image depicting a real-world scene proximate to the user.

### Claim 55 (depends on 54)

The method of claim 54, wherein determining the particular entity associated with the gesture further comprises:
performing image processing of the image to determine a plurality of portions of the image that respectively correspond to the plurality of entities;
based on the direction of the gesture, extrapolating a path of the gesture to a portion of the image; and
determining as the particular entity an entity of the plurality of entities associated with the portion of the image that the path intersects.

### Claim 56 (depends on 54)

The method of claim 54, wherein the imaging sensor is a first imaging sensor, the image captured by the first imaging sensor is a first image, and the method further comprises:
determining that a second image captured by a second imaging sensor depicts a different perspective of the real-world scene than a perspective of the real-world scene depicted in the first image;
extrapolating a first path from the direction of the gesture in the first image;
extrapolating a second path from the direction of the gesture in the second image;
identifying a point at which the first path crosses the second path; and
determining as the particular entity an entity of the plurality of entities associated with the point of the image at which the first path crosses the second path.

### Claim 57 (depends on 54)

The method of claim 54, wherein the imaging sensor is a first imaging sensor, the image captured by the first imaging sensor is a first image, and the method further comprises:
determining that a second image, captured by a second imaging sensor that is facing a second direction, depicts an area in which the user made the gesture;
performing image processing of the second image to identify the gesture;
extrapolating a first path from the direction of the gesture in the second image;
calculating, based on a position and an angle of the first imaging sensor and a position and an angle of the second imaging sensor, a second path in the first image corresponding to the first path;
performing image processing of the first image to determine a plurality of portions of the first image that respectively correspond to the plurality of entities; and
determining as the particular entity an entity of the plurality of entities associated with the portion of the image that the second path intersects.

### Claim 58 (depends on 51)

The method of claim 51, wherein determining, from among the plurality of entities, the particular entity further comprises comparing the respective pronoun of each respective entity of the plurality of entities with the identified particular pronoun.

### Claim 59 (depends on 51)

The method of claim 51, wherein the media device is a first media device, the method further comprising:
generating, for presentation at a second media device proximate to the first media device, a content item,
wherein the image associated with the gesture corresponds to a frame of the content item being presented at the second media device.

### Claim 60 (depends on 59)

The method of claim 59, wherein the particular entity is a first entity, the content item is a first content item, and the method further comprises:
generating, for presentation at the first media device while the first content item is being presented at the second media device, a second content item,
wherein the input is a voice input, and the voice input comprises a reference to a second entity depicted in a frame of the second media device, the querying of the database being further based on the second entity.

### Claim 61

A computer-implemented system, comprising:
input/output (I/O) circuitry configured to:
receive input from a user via a user input interface of a media device;
control circuitry configured to:
process the input to identify a particular pronoun;
identify a gesture made by the user;
determine a plurality of identities of a plurality of entities, respectively, in an image associated with the gesture;
determine, based on the plurality of identities, a respective pronoun for each entity of the plurality of entities;
determine, from among the plurality of entities, a particular entity having a pronoun that corresponds to the particular pronoun identified based on the input, wherein the particular entity is not being displayed on a display of the media device; and
query a database based on the input and the particular entity,
wherein the I/O circuitry is further configured to:
based on querying the database, receive at least one identifier of at least one content item; and
generate for presentation, using the media device, the at least one identifier of the at least one content item.

### Claim 62 (depends on 61)

The system of claim 61, wherein:
the input is a voice input, the particular entity is a second entity;
the I/O circuitry is further configured to receive, at the media device, a selection of a first entity currently being displayed on the display of the media device; and
the control circuitry is further configured to:
process the voice input to identify a search operator; and
query the database by constructing a search query based on the identified search operator, the first entity and the second entity.

### Claim 63 (depends on 61)

The system of claim 61, wherein the image is captured by an imaging sensor, and the control circuitry is configured to identify the gesture made by the user by determining, based on the image, motion of the user.

### Claim 64 (depends on 61)

The system of claim 61, wherein the image is captured by an imaging sensor, and the control circuitry is configured to determine the particular entity associated with the gesture by:
determining a direction of the gesture; and
determining that the direction of the gesture corresponds to the image, the image depicting a real-world scene proximate to the user.

### Claim 65 (depends on 64)

The system of claim 64, wherein the control circuitry is configured to determine the particular entity associated with the gesture by:
performing image processing of the image to determine a plurality of portions of the image that respectively correspond to the plurality of entities;
based on the direction of the gesture, extrapolating a path of the gesture to a portion of the image; and
determining as the particular entity an entity of the plurality of entities associated with the portion of the image that the path intersects.

### Claim 66 (depends on 64)

The system of claim 64, wherein the imaging sensor is a first imaging sensor, the image captured by the first imaging sensor is a first image, and the control circuitry is further configured to:
determine that a second image captured by a second imaging sensor depicts a different perspective of the real-world scene than a perspective of the real-world scene depicted in the first image;
extrapolate a first path from the direction of the gesture in the first image;
extrapolate a second path from the direction of the gesture in the second image;
identify a point at which the first path crosses the second path; and
determine as the particular entity an entity of the plurality of entities associated with the point of the image at which the first path crosses the second path.

### Claim 67 (depends on 64)

The system of claim 64, wherein the imaging sensor is a first imaging sensor, the image captured by the first imaging sensor is a first image, and the control circuitry is further configured to:
determine that a second image, captured by a second imaging sensor that is facing a second direction, depicts an area in which the user made the gesture;
perform image processing of the second image to identify the gesture;
extrapolate a first path from the direction of the gesture in the second image;
calculate, based on a position and an angle of the first imaging sensor and a position and an angle of the second imaging sensor, a second path in the first image corresponding to the first path;
perform image processing of the first image to determine a plurality of portions of the first image that respectively correspond to the plurality of entities; and
determine as the particular entity an entity of the plurality of entities associated with the portion of the image that the second path intersects.

### Claim 68 (depends on 61)

The system of claim 61, wherein the control circuitry is further configured to determine, from among the plurality of entities, the particular entity by comparing the respective pronoun of each respective entity of the plurality of entities with the identified particular pronoun.

### Claim 69 (depends on 61)

The system of claim 61, wherein the media device is a first media device, and the control circuitry is further configured to:
generate, for presentation at a second media device proximate to the first media device, a content item,
wherein the image associated with the gesture corresponds to a frame of the content item being presented at the second media device.

### Claim 70 (depends on 69)

The system of claim 69, wherein the particular entity is a first entity, the content item is a first content item, and the control circuitry is further configured to:
generate, for presentation at the first media device while the first content item is being presented at the second media device, a second content item,
wherein the input is a voice input, and the voice input comprises a reference to a second entity depicted in a frame of the second media device, the querying of the database being further based on the second entity.
