# US 2023/0410798 A1 · Multimodality in digital assistant systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410798 A1
- Document ID: US20230410798A1
- Status: Published application
- Kind: A1
- Application number: 18/240,986
- Filed: 2023-08-31
- Published: 2023-12-21
- Earliest claimed: 2019-03-18
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Pierre P. GREBORIO; Didier Rene GUZZONI; Philippe P. PIERNOT
- Web page: https://patentgrep.com/patent/US20230410798A1

## CPC

- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G06F 40/30: Semantic analysis
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G06F 40/205: Parsing
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems and processes for operating an intelligent automated assistant are provided. An example process for determining user intent includes receiving a natural language input and detecting an event. The process further includes, determining, at a first time, based on the natural language input, a first value for a first node of a parsing structure; and determining, at a second time, based on the detected data event, a second value for a second node of the parsing structure. The process further includes in accordance with a determination that the first time and the second time are within the predetermined time: determining, using the parsing structure, the first value, and the second value, a user intent associated with the natural language input; initiating a task based on the determined intent; and providing an output indicative of the task.

## Claims

### Claim 1

A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device, cause the electronic device to:
detect a context of the electronic device;
in accordance with a determination that the context of the electronic device corresponds to a predetermined type of event:
receive a natural language input;
determine, based on the natural language input, a first value for a first node of a plurality of nodes of a parsing structure, the first node corresponding to a natural language data stream;
determine, based on the detected context, a second value for a second node of the plurality of nodes of the parsing structure; and
perform a task associated with the natural language input using the parsing structure, the first value, and the second value.

### Claim 2 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the first value is determined at a first time, the second value is determined at a second time, and the user intent is determined in accordance with a determination that the first time and the second time are within a predetermined time.

### Claim 3 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the second value is determined without processing the natural language input.

### Claim 4 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein:
the first node and the second node depend from a group node of the plurality of nodes; and
determining the user intent associated with the natural language input comprises determining, using the group node, one or more values for a property of the user intent based on at least one of the first value and the second value.

### Claim 5 (depends on 4)

The non-transitory computer-readable storage medium of claim 4, wherein the group node determines the one or more values for the property in accordance with a rule associated with the group node being satisfied.

### Claim 6 (depends on 5)

The non-transitory computer-readable storage medium of claim 5, wherein the rule associated with the group node is satisfied when the first value and the second value are each determined within a predetermined time.

### Claim 7 (depends on 5)

The non-transitory computer-readable storage medium of claim 5, wherein the rule associated with the group node is satisfied when the first value is determined before the second value is determined.

### Claim 8 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein determining the user intent comprises:
determining a candidate intent based on the first value and the second value;
determining a confidence score associated with the candidate intent; and
selecting, using a second group node of the plurality of nodes, the candidate intent as the user intent based on the confidence score.

### Claim 9 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by one or more processors of the electronic device, cause the electronic device to:
in accordance with the determination that the context corresponds to the predetermined type of event:
initiate a session of a digital assistant; and
wherein the natural language input is received in accordance with initiating the session of the digital assistant.

### Claim 10 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by one or more processors of the electronic device, cause the electronic device to:
in accordance with a determination that the first value is not determined within a second predetermined time after determining the second value, provide a request for user input, wherein the natural language input is received responsive to the request for user input.

### Claim 11 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the context of the electronic device includes a location of the electronic device, whether a particular setting of the electronic device is activated, and an attribute of the electronic device.

### Claim 12 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the predetermined type of event includes detecting data from a predetermined sensor.

### Claim 13 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the predetermined type of event includes detecting data indicating a user context.

### Claim 14 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, wherein the user context includes an activity of the user.

### Claim 15 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, wherein the user context includes a preference of the user.

### Claim 16

A method for processing natural language requests, the method comprising:
detecting a context of the electronic device;
in accordance with a determination that the context of the electronic device corresponds to a predetermined type of event:
receiving a natural language input;
determining, based on the natural language input, a first value for a first node of a plurality of nodes of a parsing structure, the first node corresponding to a natural language data stream;
determining, based on the detected context, a second value for a second node of the plurality of nodes of the parsing structure; and
performing a task associated with the natural language input using the parsing structure, the first value, and the second value.

### Claim 17

An electronic device, comprising:
one or more processors;
memory; and
one or more programs stored in the memory, the one or more programs including instructions for:
detecting a context of the electronic device;
in accordance with a determination that the context of the electronic device corresponds to a predetermined type of event:
receiving a natural language input;
determining, based on the natural language input, a first value for a first node of a plurality of nodes of a parsing structure, the first node corresponding to a natural language data stream;
determining, based on the detected context, a second value for a second node of the plurality of nodes of the parsing structure; and
performing a task associated with the natural language input using the parsing structure, the first value, and the second value.
