# US 12,190,873 B2 · Determining whether speech input is intended for a digital assistant

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,190,873 B2
- Document ID: US12190873B2
- Status: Granted
- Kind: B2
- Application number: 17/952,005
- Filed: 2022-09-23
- Published: 2023-11-16
- Granted: 2025-01-07
- Earliest claimed: 2022-05-13
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Ahmed S. Hussen Abdelaziz; Saurabh Adya; Alexander W. Churchill; Pranay Dighe; Sachin S. Kajarekar; Chaitanya Mannemala; Erik Marchi; Seyedmahdad Mirsamadi; Ognjen Rudovic; Ahmed H. Tewfik; Barry-John Theobald; Srikanth Vishnubhotla
- Web page: https://patentgrep.com/patent/US12190873B2

## CPC

- G10L 15/197: Probabilistic grammars, e.g. word n-grams
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 40/161: {Detection; Localisation; Normalisation}
- G10L 15/16: using artificial neural networks
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10)
- G06T 2207/30201: Face
- G10L 2015/088: {Word spotting}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L 2015/228: {of application context}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An example process includes: receiving a speech input representing a user utterance; determining, based on a textual representation of the speech input, a first score corresponding to a type of the user utterance; determining, based on the textual representation of the speech input, a second score representing a correspondence between the user utterance and a domain recognized by a digital assistant; determining, based on the first score and the second score, whether the speech input is intended for the digital assistant; in accordance with a determination that the speech input is intended for the digital assistant: initiating, by the digital assistant, a task based on the speech input; and providing an output indicative of the initiated task.

## Claims

### Claim 1

A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device, cause the electronic device to:
receive a first speech input representing a first user utterance;
initiate, by a digital assistant operating on the electronic device, a first task based on the first speech input;
provide a first output indicative of the initiated first task; and
after providing the first output:
receive a second speech input following the first speech input, the second speech input representing a second user utterance;
determine, based on a textual representation of the second speech input, a first score representing a correspondence between the second user utterance and a domain recognized by the digital assistant;
determine, based on the textual representation of the second speech input, a second score representing contextual continuity between the first user utterance and the second user utterance;
determine, based on the first score and the second score, whether the second speech input is intended for the digital assistant; and
in accordance with a determination that the second speech input is intended for the digital assistant:
initiate, by the digital assistant, a second task based on the second speech input; and
provide a second output indicative of the initiated second task.

### Claim 2 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:
in accordance with a determination that the second speech input is not intended for the digital assistant:
forgo initiating the second task.

### Claim 3 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the first speech input and the second speech input are received within a same digital assistant session.

### Claim 4 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:
determine, based on detecting a spoken trigger for initiating a digital assistant session, that the first speech input is intended for the digital assistant, wherein:
initiating the first task is performed in accordance with a determination that the first speech input is intended for the digital assistant; and
determining whether the second speech input is intended for the digital assistant is performed without detecting the spoken trigger.

### Claim 5 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein determining whether the second speech input is intended for the digital assistant is performed without detecting a selection of a displayed affordance and without detecting a selection of a button of the electronic device.

### Claim 6 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein determining the first score includes determining whether the second user utterance corresponds to a plurality of utterances recognized by the digital assistant.

### Claim 7 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein determining the first score includes determining whether the second user utterance corresponds to a vocabulary associated with the domain recognized by the digital assistant.

### Claim 8 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein determining the first score includes determining the first score using a binary classification neural network.

### Claim 9 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein determining the second score includes determining the second score based on a textual representation of the first user utterance.

### Claim 10 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein determining the second score includes determining the second score using a second binary classification neural network.

### Claim 11 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:
determine, based on the textual representation of the second speech input, a third score corresponding to a type of the second user utterance, wherein determining whether the second speech input is intended for the digital assistant is further based on the third score.

### Claim 12 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein determining the third score includes:
determining respective probabilities that the second user utterance corresponds to each of a plurality of user utterance types;
selecting a subset of the respective probabilities, wherein each probability of the subset of the respective probabilities corresponds to a respective predetermined user utterance type of the plurality of user utterance types; and
determining the third score based on the subset of the respective probabilities.

### Claim 13 (depends on 12)

The non-transitory computer-readable storage medium of claim 12, wherein the plurality of user utterance types include:
a first type of command;
a second type of command;
a first type of question;
a second type of question;
a first type of answer;
a second type of answer;
an opinion; and
a statement.

### Claim 14 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, wherein the respective predetermined user utterance types include:
the first type of command; and
the first type of question.

### Claim 15 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein determining whether the second speech input is intended for the digital assistant includes:
weighting the first score, the second score, and the third score to obtain a final score indicating whether the second speech input is intended for the digital assistant;
comparing the final score to a threshold;
in accordance with a determination that the final score is above the threshold:
determining that the second speech input is intended for the digital assistant; and
in accordance with a determination that the final score is below the threshold:
determining that the second speech input is not intended for the digital assistant.

### Claim 16

An electronic device, comprising:
one or more processors;
a memory; and
one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
receiving a first speech input representing a first user utterance;
initiating, by a digital assistant operating on the electronic device, a first task based on the first speech input;
providing a first output indicative of the initiated first task; and
after providing the first output:
receiving a second speech input following the first speech input, the second speech input representing a second user utterance;
determining, based on a textual representation of the second speech input, a first score representing a correspondence between the second user utterance and a domain recognized by the digital assistant;
determining, based on the textual representation of the second speech input, a second score representing contextual continuity between the first user utterance and the second user utterance;
determining, based on the first score and the second score, whether the second speech input is intended for the digital assistant; and
in accordance with a determination that the second speech input is intended for the digital assistant:
initiating, by the digital assistant, a second task based on the second speech input; and
providing a second output indicative of the initiated second task.

### Claim 17 (depends on 16)

The electronic device of claim 16, the one or more programs further including instructions for:
in accordance with a determination that the second speech input is not intended for the digital assistant:
forgoing initiating the second task.

### Claim 18 (depends on 16)

The electronic device of claim 16, wherein the first speech input and the second speech input are received within a same digital assistant session.

### Claim 19 (depends on 16)

The electronic device of claim 16, the one or more programs further including instructions for:
determining, based on detecting a spoken trigger for initiating a digital assistant session, that the first speech input is intended for the digital assistant, wherein:
initiating the first task is performed in accordance with a determination that the first speech input is intended for the digital assistant; and
determining whether the second speech input is intended for the digital assistant is performed without detecting the spoken trigger.

### Claim 20 (depends on 16)

The electronic device of claim 16, wherein determining whether the second speech input is intended for the digital assistant is performed without detecting a selection of a displayed affordance and without detecting a selection of a button of the electronic device.

### Claim 21 (depends on 16)

The electronic device of claim 16, wherein determining the first score includes determining whether the second user utterance corresponds to a plurality of utterances recognized by the digital assistant.

### Claim 22 (depends on 16)

The electronic device of claim 16, wherein determining the first score includes determining whether the second user utterance corresponds to a vocabulary associated with the domain recognized by the digital assistant.

### Claim 23 (depends on 16)

The electronic device of claim 16, wherein determining the first score includes determining the first score using a binary classification neural network.

### Claim 24 (depends on 16)

The electronic device of claim 16, wherein determining the second score includes determining the second score based on a textual representation of the first user utterance.

### Claim 25 (depends on 16)

The electronic device of claim 16, wherein determining the second score includes determining the second score using a second binary classification neural network.

### Claim 26 (depends on 16)

The electronic device of claim 16, the one or more programs further including instructions for:
determining, based on the textual representation of the second speech input, a third score corresponding to a type of the second user utterance, wherein determining whether the second speech input is intended for the digital assistant is further based on the third score.

### Claim 27 (depends on 26)

The electronic device of claim 26, wherein determining the third score includes:
determining respective probabilities that the second user utterance corresponds to each of a plurality of user utterance types;
selecting a subset of the respective probabilities, wherein each probability of the subset of the respective probabilities corresponds to a respective predetermined user utterance type of the plurality of user utterance types; and
determining the third score based on the subset of the respective probabilities.

### Claim 28 (depends on 27)

The electronic device of claim 27, wherein the plurality of user utterance types include:
a first type of command;
a second type of command;
a first type of question;
a second type of question;
a first type of answer;
a second type of answer;
an opinion; and
a statement.

### Claim 29 (depends on 28)

The electronic device of claim 28, wherein the respective predetermined user utterance types include:
the first type of command; and
the first type of question.

### Claim 30 (depends on 26)

The electronic device of claim 26, wherein determining whether the second speech input is intended for the digital assistant includes:
weighting the first score, the second score, and the third score to obtain a final score indicating whether the second speech input is intended for the digital assistant;
comparing the final score to a threshold;
in accordance with a determination that the final score is above the threshold:
determining that the second speech input is intended for the digital assistant; and
in accordance with a determination that the final score is below the threshold:
determining that the second speech input is not intended for the digital assistant.

### Claim 31

A method, comprising:
at an electronic device with one or more processors and memory:
receiving a first speech input representing a first user utterance;
initiating, by a digital assistant operating on the electronic device, a first task based on the first speech input;
providing a first output indicative of the initiated first task; and
after providing the first output:
receiving a second speech input following the first speech input, the second speech input representing a second user utterance;
determining, based on a textual representation of the second speech input, a first score representing a correspondence between the second user utterance and a domain recognized by the digital assistant;
determining, based on the textual representation of the second speech input, a second score representing contextual continuity between the first user utterance and the second user utterance;
determining, based on the first score and the second score, whether the second speech input is intended for the digital assistant; and
in accordance with a determination that the second speech input is intended for the digital assistant:
initiating, by the digital assistant, a second task based on the second speech input, and
providing a second output indicative of the initiated second task.

### Claim 32 (depends on 31)

The method of claim 31, further comprising:
in accordance with a determination that the second speech input is not intended for the digital assistant:
forgoing initiating the second task.

### Claim 33 (depends on 31)

The method of claim 31, wherein the first speech input and the second speech input are received within a same digital assistant session.

### Claim 34 (depends on 31)

The method of claim 31, further comprising:
determining, based on detecting a spoken trigger for initiating a digital assistant session, that the first speech input is intended for the digital assistant, wherein:
initiating the first task is performed in accordance with a determination that the first speech input is intended for the digital assistant; and
determining whether the second speech input is intended for the digital assistant is performed without detecting the spoken trigger.

### Claim 35 (depends on 31)

The method of claim 31, wherein determining whether the second speech input is intended for the digital assistant is performed without detecting a selection of a displayed affordance and without detecting a selection of a button of the electronic device.

### Claim 36 (depends on 31)

The method of claim 31, wherein determining the first score includes determining whether the second user utterance corresponds to a plurality of utterances recognized by the digital assistant.

### Claim 37 (depends on 31)

The method of claim 31, wherein determining the first score includes determining whether the second user utterance corresponds to a vocabulary associated with the domain recognized by the digital assistant.

### Claim 38 (depends on 31)

The method of claim 31, wherein determining the first score includes determining the first score using a binary classification neural network.

### Claim 39 (depends on 31)

The method of claim 31, wherein determining the second score includes determining the second score based on a textual representation of the first user utterance.

### Claim 40 (depends on 31)

The method of claim 31, wherein determining the second score includes determining the second score using a second binary classification neural network.

### Claim 41 (depends on 31)

The method of claim 31, further comprising:
determining, based on the textual representation of the second speech input, a third score corresponding to a type of the second user utterance, wherein determining whether the second speech input is intended for the digital assistant is further based on the third score.

### Claim 42 (depends on 41)

The method of claim 41, wherein determining the third score includes:
determining respective probabilities that the second user utterance corresponds to each of a plurality of user utterance types;
selecting a subset of the respective probabilities, wherein each probability of the subset of the respective probabilities corresponds to a respective predetermined user utterance type of the plurality of user utterance types; and
determining the third score based on the subset of the respective probabilities.

### Claim 43 (depends on 42)

The method of claim 42, wherein the plurality of user utterance types include:
a first type of command;
a second type of command;
a first type of question;
a second type of question;
a first type of answer;
a second type of answer;
an opinion; and
a statement.

### Claim 44 (depends on 43)

The method of claim 43, wherein the respective predetermined user utterance types include:
the first type of command; and
the first type of question.

### Claim 45 (depends on 41)

The method of claim 41, wherein determining whether the second speech input is intended for the digital assistant includes:
weighting the first score, the second score, and the third score to obtain a final score indicating whether the second speech input is intended for the digital assistant;
comparing the final score to a threshold;
in accordance with a determination that the final score is above the threshold:
determining that the second speech input is intended for the digital assistant; and
in accordance with a determination that the final score is below the threshold:
determining that the second speech input is not intended for the digital assistant.