# US 12,002,471 B2 · Interruption detection and handling by digital assistants

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,471 B2
- Document ID: US12002471B2
- Status: Granted
- Kind: B2
- Application number: 16/950,397
- Filed: 2020-11-17
- Published: 2022-05-19
- Granted: 2024-06-04
- Earliest claimed: 2020-11-17
- Assignees: Rovi Guides, Inc.
- Applicants: Rovi Guides, Inc.
- Inventors: Prateek Varshney; Madhusudhan Seetharam; Vishwas Sharadanagar Panchaksharaiah
- Web page: https://patentgrep.com/patent/US12002471B2

## CPC

- G10L 15/222: {Barge in, i.e. overridable guidance for interrupting prompts}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 16/634: {Query by example, e.g. query by humming}
- G06F 16/90332: {Natural language query formulation or dialogue systems}
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G10L 25/51: for comparison or discrimination
- G10L 25/84: for discriminating voice from noise
- G16Y 40/10: Detection; Monitoring
- G10L 17/00: Speaker identification or verification techniques
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G16Y: INFORMATION AND COMMUNICATION TECHNOLOGY SPECIALLY ADAPTED FOR THE INTERNET OF THINGS [IoT]

## Abstract

Systems and methods are described for managing digital assistant interaction. A query is received from a user, and a reply to the query is generated for output. An interruption for the user is detected, and subsequently an end of the interruption is detected. In response to detecting the end of the interruption, a predicted query related to the initial query is identified, and a prompt to provide a reply to the predicted query may be generated for output.

## Claims

### Claim 1

A method comprising:
receiving, by a device executing an assistant application, a first query from a user;
determining, using a machine learning model, that the first query is related to a particular entity in a first context, wherein the first query comprises a first request to be provided with an indication of a first characteristic of the particular entity and related to the first context;
causing the device to generate for output a first reply to the first query, wherein the first reply provides the indication of the first characteristic to the user;
detecting an interruption for the user based on determining that the interruption persists for at least a threshold period of time;
detecting an end of the interruption; and
in response to detecting the interruption and subsequently detecting the end of the interruption:
generating a predicted second query comprising a plurality of words, wherein the predicted second query comprises a second request to be provided with a second characteristic of the particular entity of the first query in a second context and the predicted second query is a prediction of input that would have been received from the user after the first reply had the interruption not occurred, wherein the second context is different from the first context and the first characteristic is different than the second characteristic, and wherein the generating of the predicted second query comprises:
accessing, at a database storing a plurality of historical queries, historical queries previously received from a plurality of other users different from the user;
determining, using the machine learning model, a particular historical query of the accessed historical queries having one or more attributes that relate to an attribute of the particular entity of the first query; and
generating the predicted second query by identifying the plurality of words to be included in the predicted second query based on a plurality of words included in the particular historical query accessed via the database and having been received from one of the plurality of other users different from the user; and
causing the device to generate for output a prompt to provide a second reply to the predicted second query, the second reply comprising an indication of the second characteristic.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
receiving from the user a voice confirmation to provide the second reply; and
generating for output the second reply.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
refraining from generating for output replies and prompts during the interruption.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first query is a voice query, the first reply is a voice reply, the prompt is a voice prompt, and the second reply is a voice reply.

### Claim 5 (depends on 1)

The method of claim 1, wherein:
detecting the interruption for the user comprises:
detecting a voice other than a voice of the user; and
detecting the end of the interruption comprises:
determining that a predetermined period of time has passed since the detecting of the voice other than the voice of the user.

### Claim 6 (depends on 1)

The method of claim 1, wherein:
the device that receives the first query is a voice assistant device; and
detecting the interruption for the user comprises:
detecting that the user has moved to a location that is outside a predetermined distance from the voice assistant device; and
detecting the end of the interruption comprises:
detecting that the user has moved back to a location that is within the predetermined distance from the voice assistant device.

### Claim 7 (depends on 1)

The method of claim 1, wherein:
detecting the interruption for the user comprises:
listening for non-voice information, and detecting the non-voice information; and
determining that a sound level of the non-voice information exceeds a predetermined threshold;
detecting the end of the interruption comprises:
in response to detecting the non-voice information, continuing to listen for further non-voice information for a non-zero predetermined period of time; and
in response to determining, based on the continued listening for the further non-voice information, that the non-zero predetermined period of time has passed since the detecting of the non-voice information without detecting the further non-voice information, determining the interruption has ended.

### Claim 8 (depends on 1)

The method of claim 1, wherein:
generating the predicted second query further comprises:
retrieving, from a database, a user profile associated with the user; and
generating the predicted second query based at least in part on the retrieved user profile.

### Claim 9 (depends on 1)

The method of claim 1, wherein:
the device that receives the first query is a voice assistant device;
the interruption for the user is detected based on a first communication received by the voice assistant device from a different device; and
the end of the interruption is detected based on a second communication received by the voice assistant device from the different device.

### Claim 10 (depends on 1)

The method of claim 1, wherein:
the particular entity is a first entity;
the first query comprises a first plurality of words, the first plurality of words comprising the indication of the first entity;
the plurality of words of the predicted second query is a second plurality of words, the second plurality of words comprising the indication of the first entity and referencing the first entity in the second context;
the particular historical query comprises a third plurality of words comprising an indication of a second entity that is different from the first entity and that is of a same entity type as the first entity; and
generating the predicted second query further comprises causing the predicted second query to correspond to a modified version of the particular historical query, wherein the modified version of the particular historical query is obtained by replacing the second entity indicated in the third plurality of words of the particular historical query with the first entity of the first plurality of words included in the first query and by maintaining the other words included in the third plurality of words of the particular historical query in the modified version of the particular historical query.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
determining the first entity is a first physical location or landmark;
determining that the second entity is a second physical location or landmark different from the first physical location or landmark; and
based on determining that each of the first entity and the second entity correspond to a physical location or landmark, determining that the first entity and the second entity are of the same entity type.

### Claim 12

A system comprising:
input/output (I/O) circuitry configured to:
receive, by a device executing an assistant application, a first query from a user;
control circuitry configured to:
determine, using a machine learning model, that the first query is related to a particular entity in a first context, wherein the first query comprises a first request to be provided with an indication of a first characteristic of the particular entity and related to the first context;
cause the device to generate for output a first reply to the first query wherein the first reply provides the indication of the first characteristic to the user;
detect an interruption for the user based on determining that the interruption persists for at least a threshold period of time;
detect an end of the interruption; and
in response to detecting the interruption and subsequently detecting the end of the interruption:
generate a predicted second query comprising a plurality of words, wherein the predicted second query comprises a second request to be provided with a second characteristic of the particular entity of the first query in a second context and the predicted second query is a prediction of input that would have been received from the user after the first reply had the interruption not occurred, wherein the second context is different from the first context and the first characteristic is different than the second characteristic, and wherein the control circuitry is configured to generate the predicted second query by:
accessing, at a database storing a plurality of historical queries, historical queries previously received from a plurality of other users different from the user;
determining, using the machine learning model, a particular historical query of the accessed historical queries having one or more attributes that relate to an attribute of the particular entity of the first query; and
generating the predicted second query by identifying the plurality of words to be included in the predicted second query based on a plurality of words included in the particular historical query accessed via the database and having been received from one of the plurality of other users different from the user; and
cause the device to generate for output a prompt to provide a second reply to the predicted second query, the second reply comprising an indication of the second characteristic.

### Claim 13 (depends on 12)

The system of claim 12, wherein:
the input/output (I/O) circuitry is further configured to:
receive from the user a voice confirmation to provide the second reply; and
the control circuitry is further configured to:
generate for output the second reply.

### Claim 14 (depends on 12)

The system of claim 12, wherein the control circuitry is further configured to:
refrain from generating for output replies and prompts during the interruption.

### Claim 15 (depends on 12)

The system of claim 12, wherein the first query is a voice query, the first reply is a voice reply, the prompt is a voice prompt, and the second reply is a voice reply.

### Claim 16 (depends on 12)

The system of claim 12, wherein:
the control circuitry is configured to detect the interruption for the user by:
detecting a voice other than a voice of the user; and
the control circuitry is configured to detect the end of the interruption by:
determining that a predetermined period of time has passed since the detecting of the voice other than the voice of the user.

### Claim 17 (depends on 12)

The system of claim 12, wherein:
the input/output (I/O) circuitry configured to receive the first query is included in a voice assistant device that corresponds to the device; and
the control circuitry is configured to detect the interruption for the user by:
detecting that the user has moved to a location that is outside a predetermined distance from the voice assistant device; and
the control circuitry is configured to detect the end of the interruption by:
detecting that the user has moved back to a location that is within the predetermined distance from the voice assistant device.

### Claim 18 (depends on 12)

The system of claim 12, wherein:
the control circuitry is configured to detect the interruption for the user by:
listening for non-voice information, and detecting the non-voice information; and
determining that a sound level of the non-voice information exceeds a predetermined threshold;
the control circuitry is configured to detect the end of the interruption by:
in response to detecting the non-voice information, continuing to listen for further non-voice information for a predetermined period of time; and
in response to determining, based on the continued listening for the further non-voice information, that the predetermined period of time has passed without detecting the further non-voice information, determining the interruption has ended.

### Claim 19 (depends on 12)

The system of claim 12, wherein:
the control circuitry is configured to generate the predicted second query by:
retrieving, from a database, a user profile associated with the user; and
generating the predicted second query based at least in part on the retrieved user profile.

### Claim 20 (depends on 12)

The system of claim 12, wherein:
the input/output (I/O) circuitry configured to receive the first query is included in a voice assistant device that corresponds to the device; and
the control circuitry is configured to:
detect the interruption for the user based on a first communication received by the voice assistant device from a different device; and
detect the end of the interruption based on a second communication received by the voice assistant device from the different device.