# US 12,118,999 B2 · Reducing the need for manual start/end-pointing and trigger phrases

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,999 B2
- Document ID: US12118999B2
- Status: Granted
- Kind: B2
- Application number: 18/231,135
- Filed: 2023-08-07
- Published: 2023-11-30
- Granted: 2024-10-15
- Earliest claimed: 2014-05-30
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Philippe P. Piernot; Justin G. Binder
- Web page: https://patentgrep.com/patent/US12118999B2

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- H04W 4/025: {using location based information parameters}
- G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 17/00: Speaker identification or verification techniques
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L 2015/227: {of the speaker; Human-factor methodology}
- G10L 2015/228: {of application context}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

Systems and processes for selectively processing and responding to a spoken user input are provided. In one example, audio input containing a spoken user input can be received at a user device. The spoken user input can be identified from the audio input by identifying start and end-points of the spoken user input. It can be determined whether or not the spoken user input was intended for a virtual assistant based on contextual information. The determination can be made using a rule-based system or a probabilistic system. If it is determined that the spoken user input was intended for the virtual assistant, the spoken user input can be processed and an appropriate response can be generated. If it is instead determined that the spoken user input was not intended for the virtual assistant, the spoken user input can be ignored and/or no response can be generated.

## Claims

### Claim 1

A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device, cause the electronic device to:
receive a first spoken input, wherein the first spoken input requests performance of a first task;
activate a virtual assistant operating on the electronic device;
perform, by the activated virtual assistant, the first task based on the first spoken input;
provide, at a first time, a first response indicating the performance of the first task, wherein providing the first response includes providing at least one of audio output and displayed output;
after the first time, monitor received audio input to identify a second spoken input in the audio input, wherein the second spoken input does not comprise a spoken trigger for activating the virtual assistant;
in accordance with identifying the second spoken input, determine whether to respond to the second spoken input based on contextual information that indicates whether a user is facing the electronic device when the second spoken input was received;
in accordance with a determination to respond to the second spoken input, perform a second task based on the second spoken input; and
provide a second response indicating the performance of the second task.

### Claim 2 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:
while monitoring the received audio input, output a visual indicator, the visual indicator indicating that the electronic device is capable of responding to received spoken input that does not include the spoken trigger.

### Claim 3 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the second spoken input is identified without identifying a physical or virtual button input.

### Claim 4 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein determining whether to respond to the second spoken input based on the contextual information includes:
in accordance with a determination that the contextual information indicates that the user is facing the electronic device when the second spoken input was received, determining to respond to the second spoken input; and
in accordance with a determination that the contextual information indicates that the user is not facing the electronic device when the second spoken input was received, determining not to respond to the second spoken input.

### Claim 5 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the electronic device includes a camera, and wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:
capture data from the camera; and
determine, based on analyzing the captured data, the contextual information that indicates whether the user is facing the electronic device when the second spoken input was received.

### Claim 6 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein determining whether to respond to the second spoken input based on the contextual information includes:
calculating a likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information, including:
in accordance with a determination that the user is facing the electronic device when the second spoken input was received, increasing the likelihood score; and
in accordance with a determination that the user is not facing the electronic device when the second spoken input was received, decreasing the likelihood score.

### Claim 7 (depends on 6)

The non-transitory computer-readable storage medium of claim 6, wherein the contextual information further indicates a type of motion of the electronic device, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that the type of motion is a first type of motion, increasing the likelihood score; and
in accordance with a determination that the type of motion is a second type of motion different from the first type of motion, decreasing the likelihood score.

### Claim 8 (depends on 7)

The non-transitory computer-readable storage medium of claim 7, wherein:
the first type of motion corresponds to movement of the electronic device towards the user before the second spoken input is received; and
the second type of motion does not correspond to movement of the electronic device towards the user before the second spoken input is received.

### Claim 9 (depends on 6)

The non-transitory computer-readable storage medium of claim 6, wherein the contextual information further indicates a gaze direction of the user when the second spoken user input was received, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that the gaze direction is pointed at the electronic device when the second spoken user input was received, increasing the likelihood score; and
in accordance with a determination that the gaze direction is not pointed at the electronic device when the second spoken user input was received, decreasing the likelihood score.

### Claim 10 (depends on 6)

The non-transitory computer-readable storage medium of claim 6, wherein the contextual information further indicates an elapsed time between receiving the second spoken input and the first spoken input, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that the elapsed time is greater than a threshold duration, decreasing the likelihood score; and
in accordance with a determination that the elapsed time is less than the threshold duration, increasing the likelihood score.

### Claim 11 (depends on 6)

The non-transitory computer-readable storage medium of claim 6, wherein the contextual information further indicates an orientation of the electronic device when the second spoken input was received, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that the orientation of the electronic device is face down and/or upside down, decreasing the likelihood score; and
in accordance with a determination that the orientation of the electronic device is face up and/or upright, increasing the likelihood score.

### Claim 12 (depends on 6)

The non-transitory computer-readable storage medium of claim 6, wherein the contextual information further indicates whether the second spoken input was recognized by an automatic speech recognizer, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that the second spoken input was recognized by the automatic speech recognizer, increasing the likelihood score; and
in accordance with a determination that the second spoken input was not recognized by the automatic speech recognizer, decreasing the likelihood score.

### Claim 13 (depends on 6)

The non-transitory computer-readable storage medium of claim 6, wherein the contextual information further indicates a semantic relationship between the second spoken input and the first spoken input, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that a value of the semantic relationship is greater than a spoken user input semantic threshold value, increasing the likelihood score; and
in accordance with a determination that the value of the semantic relationship is less than the spoken user input semantic threshold value, decreasing the likelihood score.

### Claim 14 (depends on 6)

The non-transitory computer-readable storage medium of claim 6, wherein the contextual information further indicates an identification of a speaker of the second spoken input, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that the identified speaker is included in a list of authorized speakers, increasing the likelihood score; and
in accordance with a determination that the identified speaker is not included in the list of speakers, decreasing the likelihood score.

### Claim 15 (depends on 6)

The non-transitory computer-readable storage medium of claim 6, wherein the contextual information further indicates whether the electronic device is being held when the second spoken input was received, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that the electronic device was being held when the second spoken input was received, increasing the likelihood score; and
in accordance with a determination that the electronic device was not being held when the second spoken input was received, decreasing the likelihood score.

### Claim 16 (depends on 6)

The non-transitory computer-readable storage medium of claim 6, wherein the contextual information further indicates a semantic relationship between the second spoken input and content being displayed by the electronic device when the second spoken input was received, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that a value of the semantic relationship is greater than a content semantic threshold value, increasing the likelihood score; and
in accordance with a determination that the value of the semantic relationship is less than the content semantic threshold value, decreasing the likelihood score.

### Claim 17 (depends on 6)

The non-transitory computer-readable storage medium of claim 6, wherein the contextual information further indicates a semantic relationship between the second spoken input and the first response, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that a value of the semantic relationship is greater than a semantic threshold value, increasing the likelihood score; and
in accordance with a determination that the value of the semantic relationship is less than the semantic threshold value, decreasing the likelihood score.

### Claim 18

A method, comprising:
at an electronic device with one or more processors and memory:
receiving a first spoken input, wherein the first spoken input requests performance of a first task;
activating a virtual assistant operating on the electronic device;
performing, by the activated virtual assistant, the first task based on the first spoken input;
providing, at a first time, a first response indicating the performance of the first task, wherein providing the first response includes providing at least one of audio output and displayed output;
after the first time, monitoring received audio input to identify a second spoken input in the audio input, wherein the second spoken input does not comprise a spoken trigger for activating the virtual assistant;
in accordance with identifying the second spoken input, determining whether to respond to the second spoken input based on contextual information that indicates whether a user is facing the electronic device when the second spoken input was received;
in accordance with a determination to respond to the second spoken input, performing a second task based on the second spoken input; and
providing a second response indicating the performance of the second task.

### Claim 19 (depends on 18)

The method of claim 18, further comprising:
while monitoring the received audio input, outputting a visual indicator, the visual indicator indicating that the electronic device is capable of responding to received spoken input that does not include the spoken trigger.

### Claim 20 (depends on 18)

The method of claim 18, wherein the second spoken input is identified without identifying a physical or virtual button input.

### Claim 21 (depends on 18)

The method of claim 18, wherein determining whether to respond to the second spoken input based on the contextual information includes:
in accordance with a determination that the contextual information indicates that the user is facing the electronic device when the second spoken input was received, determining to respond to the second spoken input; and
in accordance with a determination that the contextual information indicates that the user is not facing the electronic device when the second spoken input was received, determining not to respond to the second spoken input.

### Claim 22 (depends on 18)

The method of claim 18, wherein the electronic device includes a camera, and wherein the method further comprises:
capturing data from the camera; and
determining, based on analyzing the captured data, the contextual information that indicates whether the user is facing the electronic device when the second spoken input was received.

### Claim 23 (depends on 18)

The method of claim 18, wherein determining whether to respond to the second spoken input based on the contextual information includes:
calculating a likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information, including:
in accordance with a determination that the user is facing the electronic device when the second spoken input was received, increasing the likelihood score; and
in accordance with a determination that the user is not facing the electronic device when the second spoken input was received, decreasing the likelihood score.

### Claim 24 (depends on 23)

The method of claim 23, wherein the contextual information further indicates a type of motion of the electronic device, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that the type of motion is a first type of motion, increasing the likelihood score; and
in accordance with a determination that the type of motion is a second type of motion different from the first type of motion, decreasing the likelihood score.

### Claim 25 (depends on 24)

The method of claim 24, wherein:
the first type of motion corresponds to movement of the electronic device towards the user before the second spoken input is received; and
the second type of motion does not correspond to movement of the electronic device towards the user before the second spoken input is received.

### Claim 26 (depends on 23)

The method of claim 23, wherein the contextual information further indicates a gaze direction of the user when the second spoken user input was received, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that the gaze direction is pointed at the electronic device when the second spoken user input was received, increasing the likelihood score; and
in accordance with a determination that the gaze direction is not pointed at the electronic device when the second spoken user input was received, decreasing the likelihood score.

### Claim 27 (depends on 23)

The method of claim 23, wherein the contextual information further indicates an elapsed time between receiving the second spoken input and the first spoken input, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that the elapsed time is greater than a threshold duration, decreasing the likelihood score; and
in accordance with a determination that the elapsed time is less than the threshold duration, increasing the likelihood score.

### Claim 28 (depends on 23)

The method of claim 23, wherein the contextual information further indicates an orientation of the electronic device when the second spoken input was received, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that the orientation of the electronic device is face down and/or upside down, decreasing the likelihood score; and
in accordance with a determination that the orientation of the electronic device is face up and/or upright, increasing the likelihood score.

### Claim 29 (depends on 23)

The method of claim 23, wherein the contextual information further indicates whether the second spoken input was recognized by an automatic speech recognizer, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that the second spoken input was recognized by the automatic speech recognizer, increasing the likelihood score; and
in accordance with a determination that the second spoken input was not recognized by the automatic speech recognizer, decreasing the likelihood score.

### Claim 30 (depends on 23)

The method of claim 23, wherein the contextual information further indicates a semantic relationship between the second spoken input and the first spoken input, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that a value of the semantic relationship is greater than a spoken user input semantic threshold value, increasing the likelihood score; and
in accordance with a determination that the value of the semantic relationship is less than the spoken user input semantic threshold value, decreasing the likelihood score.

### Claim 31 (depends on 23)

The method of claim 23, wherein the contextual information further indicates an identification of a speaker of the second spoken input, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that the identified speaker is included in a list of authorized speakers, increasing the likelihood score; and
in accordance with a determination that the identified speaker is not included in the list of speakers, decreasing the likelihood score.

### Claim 32 (depends on 23)

The method of claim 23, wherein the contextual information further indicates whether the electronic device is being held when the second spoken input was received, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that the electronic device was being held when the second spoken input was received, increasing the likelihood score; and
in accordance with a determination that the electronic device was not being held when the second spoken input was received, decreasing the likelihood score.

### Claim 33 (depends on 23)

The method of claim 23, wherein the contextual information further indicates a semantic relationship between the second spoken input and content being displayed by the electronic device when the second spoken input was received, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that a value of the semantic relationship is greater than a content semantic threshold value, increasing the likelihood score; and
in accordance with a determination that the value of the semantic relationship is less than the content semantic threshold value, decreasing the likelihood score.

### Claim 34 (depends on 23)

The method of claim 23, wherein the contextual information further indicates a semantic relationship between the second spoken input and the first response, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that a value of the semantic relationship is greater than a semantic threshold value, increasing the likelihood score; and
in accordance with a determination that the value of the semantic relationship is less than the semantic threshold value, decreasing the likelihood score.

### Claim 35

An electronic device, comprising:
one or more processors;
memory; and
one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
receiving a first spoken input, wherein the first spoken input requests performance of a first task;
activating a virtual assistant operating on the electronic device;
performing, by the activated virtual assistant, the first task based on the first spoken input;
providing, at a first time, a first response indicating the performance of the first task, wherein providing the first response includes providing at least one of audio output and displayed output;
after the first time, monitoring received audio input to identify a second spoken input in the audio input, wherein the second spoken input does not comprise a spoken trigger for activating the virtual assistant;
in accordance with identifying the second spoken input, determining whether to respond to the second spoken input based on contextual information that indicates whether a user is facing the electronic device when the second spoken input was received;
in accordance with a determination to respond to the second spoken input, performing a second task based on the second spoken input; and
providing a second response indicating the performance of the second task.

### Claim 36 (depends on 35)

The electronic device of claim 35, wherein the one or more programs further include instructions for:
while monitoring the received audio input, outputting a visual indicator, the visual indicator indicating that the electronic device is capable of responding to received spoken input that does not include the spoken trigger.

### Claim 37 (depends on 35)

The electronic device of claim 35, wherein the second spoken input is identified without identifying a physical or virtual button input.

### Claim 38 (depends on 35)

The electronic device of claim 35, wherein determining whether to respond to the second spoken input based on the contextual information includes:
in accordance with a determination that the contextual information indicates that the user is facing the electronic device when the second spoken input was received, determining to respond to the second spoken input; and
in accordance with a determination that the contextual information indicates that the user is not facing the electronic device when the second spoken input was received, determining not to respond to the second spoken input.

### Claim 39 (depends on 35)

The electronic device of claim 35, wherein the electronic device includes a camera, and wherein the one or more programs further include instructions for:
capturing data from the camera; and
determining, based on analyzing the captured data, the contextual information that indicates whether the user is facing the electronic device when the second spoken input was received.

### Claim 40 (depends on 35)

The electronic device of claim 35, wherein determining whether to respond to the second spoken input based on the contextual information includes:
calculating a likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information, including:
in accordance with a determination that the user is facing the electronic device when the second spoken input was received, increasing the likelihood score; and
in accordance with a determination that the user is not facing the electronic device when the second spoken input was received, decreasing the likelihood score.

### Claim 41 (depends on 40)

The electronic device of claim 40, wherein the contextual information further indicates a type of motion of the electronic device, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that the type of motion is a first type of motion, increasing the likelihood score; and
in accordance with a determination that the type of motion is a second type of motion different from the first type of motion, decreasing the likelihood score.

### Claim 42 (depends on 41)

The electronic device of claim 41, wherein:
the first type of motion corresponds to movement of the electronic device towards the user before the second spoken input is received; and
the second type of motion does not correspond to movement of the electronic device towards the user before the second spoken input is received.

### Claim 43 (depends on 40)

The electronic device of claim 40, wherein the contextual information further indicates a gaze direction of the user when the second spoken user input was received, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that the gaze direction is pointed at the electronic device when the second spoken user input was received, increasing the likelihood score; and
in accordance with a determination that the gaze direction is not pointed at the electronic device when the second spoken user input was received, decreasing the likelihood score.

### Claim 44 (depends on 40)

The electronic device of claim 40, wherein the contextual information further indicates an elapsed time between receiving the second spoken input and the first spoken input, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that the elapsed time is greater than a threshold duration, decreasing the likelihood score; and
in accordance with a determination that the elapsed time is less than the threshold duration, increasing the likelihood score.

### Claim 45 (depends on 40)

The electronic device of claim 40, wherein the contextual information further indicates an orientation of the electronic device when the second spoken input was received, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that the orientation of the electronic device is face down and/or upside down, decreasing the likelihood score; and
in accordance with a determination that the orientation of the electronic device is face up and/or upright, increasing the likelihood score.

### Claim 46 (depends on 40)

The electronic device of claim 40, wherein the contextual information further indicates whether the second spoken input was recognized by an automatic speech recognizer, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that the second spoken input was recognized by the automatic speech recognizer, increasing the likelihood score; and
in accordance with a determination that the second spoken input was not recognized by the automatic speech recognizer, decreasing the likelihood score.

### Claim 47 (depends on 40)

The electronic device of claim 40, wherein the contextual information further indicates a semantic relationship between the second spoken input and the first spoken input, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that a value of the semantic relationship is greater than a spoken user input semantic threshold value, increasing the likelihood score; and
in accordance with a determination that the value of the semantic relationship is less than the spoken user input semantic threshold value, decreasing the likelihood score.

### Claim 48 (depends on 40)

The electronic device of claim 40, wherein the contextual information further indicates an identification of a speaker of the second spoken input, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that the identified speaker is included in a list of authorized speakers, increasing the likelihood score; and
in accordance with a determination that the identified speaker is not included in the list of speakers, decreasing the likelihood score.

### Claim 49 (depends on 40)

The electronic device of claim 40, wherein the contextual information further indicates whether the electronic device is being held when the second spoken input was received, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that the electronic device was being held when the second spoken input was received, increasing the likelihood score; and
in accordance with a determination that the electronic device was not being held when the second spoken input was received, decreasing the likelihood score.

### Claim 50 (depends on 40)

The electronic device of claim 40, wherein the contextual information further indicates a semantic relationship between the second spoken input and content being displayed by the electronic device when the second spoken input was received, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that a value of the semantic relationship is greater than a content semantic threshold value, increasing the likelihood score; and
in accordance with a determination that the value of the semantic relationship is less than the content semantic threshold value, decreasing the likelihood score.

### Claim 51 (depends on 40)

The electronic device of claim 40, wherein the contextual information further indicates a semantic relationship between the second spoken input and the first response, and wherein calculating the likelihood score that the virtual assistant should provide the second response to the second spoken input based on the contextual information includes:
in accordance with a determination that a value of the semantic relationship is greater than a semantic threshold value, increasing the likelihood score; and
in accordance with a determination that the value of the semantic relationship is less than the semantic threshold value, decreasing the likelihood score.