# US 2025/0166626 A1 · Natural assistant interaction

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0166626 A1
- Document ID: US20250166626A1
- Status: Published application
- Kind: A1
- Application number: 19/028,941
- Filed: 2025-01-17
- Published: 2025-05-22
- Earliest claimed: 0001-01-01
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Juan Carlos GARCIA; Paul S. MCCARTHY; Kurt PIERSOL
- Web page: https://patentgrep.com/patent/US20250166626A1

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 40/30: Semantic analysis
- G10L 15/08: Speech classification or search
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10)
- G10L 25/87: Detection of discrete points within a voice signal
- G10L 2015/088: {Word spotting}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L 2025/783: {based on threshold decision}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems and processes for operating a virtual assistant to provide natural assistant interaction are provided. In accordance with one or more examples, a method includes, at an electronic device with one or more processors and memory: receiving a first audio stream including one or more utterances; determining whether the first audio stream includes a lexical trigger; generating one or more candidate text representations of the one or more utterances; determining whether at least one candidate text representation of the one or more candidate text representations is to be disregarded by the virtual assistant. If at least one candidate text representation is to be disregarded, one or more candidate intents are generated based on candidate text representations of the one or more candidate text representations other than the to be disregarded at least one candidate text representation.

## Claims

### Claim 1

An electronic device, comprising:
one or more processors;
memory; and
one or more programs stored in memory, the one or more programs including instructions for:
receiving a user utterance;
determining, based on the user utterance, one or more candidate text representations;
determining whether a first candidate text representation of the one or more candidate text representations includes a lexical trigger;
in accordance with a determination that the first candidate text representation does not include the lexical trigger:
obtaining user gaze data from a sensor communicatively coupled to the electronic device;
determining, based on the user gaze data, a likelihood that the user utterance is directed to a virtual assistant;
determining whether the likelihood that the user utterance is directed to the virtual assistant exceeds a threshold; and
in accordance with a determination that the likelihood that the user utterance is directed to the virtual assistant exceeds a threshold, determining one or more candidate intents based on the first candidate text representation.

### Claim 2 (depends on 1)

The electronic device of claim 1, the one or more programs including further instructions for:
in accordance with a determination that the first candidate text representation does not include the lexical trigger:
obtaining context information; and
determining based on the user gaze data and the context information, a likelihood that the utterance is directed to a virtual assistant.

### Claim 3 (depends on 2)

The electronic device of claim 2, wherein the context information includes a usage pattern of the virtual assistant.

### Claim 4 (depends on 2)

The electronic device of claim 2, wherein the context information includes a time associated with the electronic device.

### Claim 5 (depends on 2)

The electronic device of claim 2, wherein the context information includes a location associated with the electronic device.

### Claim 6 (depends on 2)

The electronic device of claim 2, wherein the sensor is a first sensor and the context information is obtained from a second sensor communicatively coupled to the electronic device.

### Claim 7 (depends on 1)

The electronic device of claim 1, the one or more programs including further instructions for:
in accordance with a determination that the first candidate text representation includes the lexical trigger, determining one or more candidate intents based on the first candidate text representation.

### Claim 8 (depends on 1)

The electronic device of claim 1, the one or more programs including further instructions for:
determining whether a task associated with the one or more candidate intents can be performed.

### Claim 9 (depends on 8)

The electronic device of claim 8, wherein determining whether the task associated with the one or more candidate intents can be performed further comprises:
obtaining context information; and
determining whether the task associated with the one or more candidate intents can be performed using the context information.

### Claim 10 (depends on 8)

The electronic device of claim 8, the one or more programs including further instructions for:
in accordance with a determination that the task associated with the one or more candidate intents can be performed:
performing the task; and
providing an output indicative of the task.

### Claim 11 (depends on 1)

The electronic device of claim 1, the one or more programs including further instructions for:
in accordance with a determination that the likelihood that the user utterance is directed to the virtual assistant is below a threshold, disregarding the user utterance.

### Claim 12 (depends on 1)

The electronic device of claim 1, the one or more programs including further instructions for:
determining whether a second candidate text representation of the one or more candidate text representations includes a lexical trigger;
in accordance with a determination that the second candidate text representation does not include the lexical trigger:
obtaining user gaze data from a sensor communicatively coupled to the electronic device;
determining, based on the user gaze data, a likelihood that the user utterance is directed to a virtual assistant;
determining whether the likelihood that the user utterance is directed to the virtual assistant exceeds a threshold; and
in accordance with a determination that the likelihood that the user utterance is directed to the virtual assistant exceeds a threshold, determining one or more candidate intents based on the second candidate text representation.

### Claim 13

A method comprising:
at an electronic device with one or more processors and memory:
receiving a user utterance;
determining, based on the user utterance, one or more candidate text representations;
determining whether a first candidate text representation of the one or more candidate text representations includes a lexical trigger;
in accordance with a determination that the first candidate text representation does not include the lexical trigger:
obtaining user gaze data from a sensor communicatively coupled to the electronic device;
determining, based on the user gaze data, a likelihood that the user utterance is directed to a virtual assistant;
determining whether the likelihood that the user utterance is directed to the virtual assistant exceeds a threshold; and
in accordance with a determination that the likelihood that the user utterance is directed to the virtual assistant exceeds a threshold, determining one or more candidate intents based on the first candidate text representation.

### Claim 14

A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device, cause the electronic device to:
receive a user utterance;
determine, based on the user utterance, one or more candidate text representations;
determine whether a first candidate text representation of the one or more candidate text representations includes a lexical trigger;
in accordance with a determination that the first candidate text representation does not include the lexical trigger:
obtain user gaze data from a sensor communicatively coupled to the electronic device;
determine, based on the user gaze data, a likelihood that the user utterance is directed to a virtual assistant;
determine whether the likelihood that the user utterance is directed to the virtual assistant exceeds a threshold; and
in accordance with a determination that the likelihood that the user utterance is directed to the virtual assistant exceeds a threshold, determine one or more candidate intents based on the first candidate text representation.
