# US 11,907,436 B2 · Raise to speak

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,907,436 B2
- Document ID: US11907436B2
- Status: Granted
- Kind: B2
- Application number: 17/946,959
- Filed: 2022-09-16
- Published: 2023-02-09
- Granted: 2024-02-20
- Earliest claimed: 2018-05-07
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Stephen O. Lemay; Michael R. Bastian; Roman Holenstein; Minwoo Jeong; Charles Maalouf; Brandon J. Newendorp; Heriberto Nieto; Timothy Paek; Joanna Peterson; Shawn Scully; Srikrishna Sridhar; Brandt M. Westing; Shiwen Zhao
- Web page: https://patentgrep.com/patent/US11907436B2

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 1/1684: {Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675}
- G06F 1/1694: {the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer}
- G06F 1/3206: Monitoring of events, devices or parameters that trigger a change in power modality
- G06F 1/3231: Monitoring the presence, absence or movement of users
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G10L 15/07: to the speaker
- G06Q 10/02: Reservations, e.g. for tickets, services or events
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 25/51: for comparison or discrimination
- G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10)
- Y02D 10/00: Energy efficient computing, e.g. low power processors, power management or thermal management
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- Y02D: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE

## Abstract

Systems and processes for operating an intelligent automated assistant are provided. An example process includes detecting input representing motion of an electronic device and sampling an audio input with a microphone of the electronic device. The example process further includes determining, based on the audio input and the input representing motion of the electronic device, whether to initiate a virtual assistant session. In accordance with a determination to initiate the virtual assistant session, the example process includes initiating the virtual assistant session. In accordance with a determination not to initiate the virtual assistant session, the example process includes forgoing initiating the virtual assistant session.

## Claims

### Claim 1

A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device with a microphone and a display, cause the electronic device to:
detect input representing motion of the electronic device;
while detecting at least a portion of the input representing motion of the electronic device, sample, with the microphone, an audio input;
determine, based on the audio input and the input representing motion of the electronic device, whether to initiate a virtual assistant session, wherein determining whether to initiate the virtual assistant session includes:
determining, at a first time, whether to initiate the virtual assistant session based on the audio input;
determining, at a second time, whether to initiate the virtual assistant session based on the input representing motion of the electronic device;
determining whether the first time and the second time are within a predetermined duration; and
determining to initiate the virtual assistant session in accordance with a determination that the first time and the second time are within the predetermined duration;
in accordance with a determination to initiate the virtual assistant session, initiate the virtual assistant session; and
in accordance with a determination not to initiate the virtual assistant session, forgo initiating the virtual assistant session.

### Claim 2 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:
in accordance with initiating the virtual assistant session:
determine a user intent based on the audio input;
perform a task based on the user intent; and
provide an output associated with the task.

### Claim 3 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:
in accordance with initiating the virtual assistant session:
in accordance with not determining a user intent based on the audio input, forgo providing an output responsive to the audio input.

### Claim 4 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the audio input does not include a spoken trigger.

### Claim 5 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:
power on the display based on a first portion of the input representing motion of the electronic device; and
wherein the audio input is sampled in accordance with powering on the display.

### Claim 6 (depends on 5)

The non-transitory computer-readable storage medium of claim 5, wherein the first portion of the input representing motion of the electronic device represents a wrist raise gesture or a wrist rotation gesture.

### Claim 7 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the motion of the electronic device includes a transition of the electronic device between two or more poses of a plurality of poses.

### Claim 8 (depends on 7)

The non-transitory computer-readable storage medium of claim 7, wherein the plurality of poses includes a dropped pose, a raising pose, a raised pose, and a dropping pose.

### Claim 9 (depends on 8)

The non-transitory computer-readable storage medium of claim 8, wherein determining whether to initiate the virtual assistant session includes:
determining, based on the input representing motion of the electronic device, that the motion of the electronic device includes a transition from the raising pose to the raised pose.

### Claim 10 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein determining whether to initiate the virtual assistant session includes:
determining a duration of the transition from the raising pose to the raised pose.

### Claim 11 (depends on 8)

The non-transitory computer-readable storage medium of claim 8, wherein determining whether to initiate the virtual assistant session includes:
determining, based on the input representing motion of the electronic device, whether a duration of the raised pose exceeds a first predetermined duration.

### Claim 12 (depends on 8)

The non-transitory computer-readable storage medium of claim 8, wherein determining whether to initiate the virtual assistant session includes:
determining, based on the input representing motion of the electronic device, whether the motion of the electronic device includes a transition from the raised pose to the dropping pose.

### Claim 13 (depends on 8)

The non-transitory computer-readable storage medium of claim 8, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:
in accordance with initiating the virtual assistant session:
determine, based on the input representing motion of the electronic device, whether the motion of the electronic device includes a transition from the raised pose to the dropping pose; and
in accordance with a determination that the motion of the electronic device includes the second transition from the raised pose to the dropping pose, deactivate the virtual assistant session.

### Claim 14 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:
determine a time associated with the transition from the raised pose to the dropping pose;
determine a time associated with an end of speech in the audio input; and
wherein deactivating the virtual assistant session is further performed in accordance with a determination that the time associated with the second transition from the raised pose to the dropping pose is prior to the time associated with the end of speech in the audio input.

### Claim 15 (depends on 8)

The non-transitory computer-readable storage medium of claim 8, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:
determine a time associated with a transition from the raised pose to the dropping pose; and
determine an endpoint of the audio input based on the time associated with the transition from the raised pose to the dropping pose.

### Claim 16 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein a motion model determines, at the second time, whether to initiate the virtual assistant session based on the input representing motion of the electronic device.

### Claim 17 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein the motion model is specific to a user of the electronic device.

### Claim 18 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein detecting the input representing motion of the electronic device includes sampling the input representing motion of the electronic device using a first processor.

### Claim 19 (depends on 18)

The non-transitory computer-readable storage medium of claim 18, wherein sampling the audio input includes sampling the audio input using a second processor.

### Claim 20 (depends on 19)

The non-transitory computer-readable storage medium of claim 19, wherein the second processor consumes more power while operating than the first processor.

### Claim 21 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein determining whether to initiate the virtual assistant session includes determining whether the audio input includes human speech.

### Claim 22 (depends on 21)

The non-transitory computer-readable storage medium of claim 21, wherein determining whether the audio input includes human speech includes determining whether the audio input includes human speech spoken by an authorized user of the electronic device.

### Claim 23 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein determining whether to initiate the virtual assistant session includes determining a direction associated with the audio input.

### Claim 24 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein determining whether to initiate the virtual assistant session includes determining a proximity of the audio input from the electronic device.

### Claim 25 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein an audio model determines, at the first time, whether to initiate the virtual assistant session based on the audio input.

### Claim 26 (depends on 25)

The non-transitory computer-readable storage medium of claim 25, wherein the audio model is specific to a second user of the electronic device.

### Claim 27 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:
in accordance with initiating the virtual assistant session:
determine, based on a linguistic content of the audio input, whether to deactivate the virtual assistant session;
in accordance with a determination to deactivate the virtual assistant session, deactivate the virtual assistant session; and
in accordance with a determination not to deactivate the virtual assistant session, forgo deactivating the virtual assistant session.

### Claim 28 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:
in accordance with initiating the virtual assistant session:
determine whether a duration of the audio input is less than a threshold duration; and
in accordance with a determination that the duration of the audio input is less than the threshold duration, deactivate the virtual assistant session.

### Claim 29

An electronic device, comprising:
a microphone;
a display;
one or more processors;
a memory; and
one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
detecting input representing motion of the electronic device;
while detecting at least a portion of the input representing motion of the electronic device, sampling, with the microphone, an audio input;
determining, based on the audio input and the input representing motion of the electronic device, whether to initiate a virtual assistant session, wherein determining whether to initiate the virtual assistant session includes:
determining, at a first time, whether to initiate the virtual assistant session based on the audio input;
determining, at a second time, whether to initiate the virtual assistant session based on the input representing motion of the electronic device;
determining whether the first time and the second time are within a predetermined duration; and
determining to initiate the virtual assistant session in accordance with a determination that the first time and the second time are within the predetermined duration;
in accordance with a determination to initiate the virtual assistant session, initiating the virtual assistant session; and
in accordance with a determination not to initiate the virtual assistant session, forgoing initiating the virtual assistant session.

### Claim 30 (depends on 29)

The electronic device of claim 29, wherein the one or more programs further include instructions for:
in accordance with initiating the virtual assistant session:
determining a user intent based on the audio input;
performing a task based on the user intent; and
providing an output associated with the task.

### Claim 31 (depends on 29)

The electronic device of claim 29, wherein the one or more programs further include instructions for:
in accordance with initiating the virtual assistant session:
in accordance with not determining a user intent based on the audio input, forgoing providing an output responsive to the audio input.

### Claim 32 (depends on 29)

The electronic device of claim 29, wherein the audio input does not include a spoken trigger.

### Claim 33 (depends on 29)

The electronic device of claim 29, wherein the one or more programs further include instructions for:
powering on the display based on a first portion of the input representing motion of the electronic device; and
wherein the audio input is sampled in accordance with powering on the display.

### Claim 34 (depends on 33)

The electronic device of claim 33, wherein the first portion of the input representing motion of the electronic device represents a wrist raise gesture or a wrist rotation gesture.

### Claim 35 (depends on 29)

The electronic device of claim 29, wherein the motion of the electronic device includes a transition of the electronic device between two or more poses of a plurality of poses.

### Claim 36 (depends on 35)

The electronic device of claim 35, wherein the plurality of poses includes a dropped pose, a raising pose, a raised pose, and a dropping pose.

### Claim 37 (depends on 36)

The electronic device of claim 36, wherein determining whether to initiate the virtual assistant session includes:
determining, based on the input representing motion of the electronic device, that the motion of the electronic device includes a transition from the raising pose to the raised pose.

### Claim 38 (depends on 37)

The electronic device of claim 37, wherein determining whether to initiate the virtual assistant session includes:
determining a duration of the transition from the raising pose to the raised pose.

### Claim 39 (depends on 36)

The electronic device of claim 36, wherein determining whether to initiate the virtual assistant session includes:
determining, based on the input representing motion of the electronic device, whether a duration of the raised pose exceeds a first predetermined duration.

### Claim 40 (depends on 36)

The electronic device of claim 36, wherein determining whether to initiate the virtual assistant session includes:
determining, based on the input representing motion of the electronic device, whether the motion of the electronic device includes a transition from the raised pose to the dropping pose.

### Claim 41 (depends on 36)

The electronic device of claim 36, wherein the one or more programs further include instructions for:
in accordance with initiating the virtual assistant session:
determining, based on the input representing motion of the electronic device, whether the motion of the electronic device includes a transition from the raised pose to the dropping pose; and
in accordance with a determination that the motion of the electronic device includes the second transition from the raised pose to the dropping pose, deactivating the virtual assistant session.

### Claim 42 (depends on 29)

The electronic device of claim 29, wherein determining whether to initiate the virtual assistant session includes determining whether the audio input includes human speech.

### Claim 43 (depends on 42)

The electronic device of claim 42, wherein determining whether the audio input includes human speech includes determining whether the audio input includes human speech spoken by an authorized user of the electronic device.

### Claim 44 (depends on 29)

The electronic device of claim 29, wherein determining whether to initiate the virtual assistant session includes determining a direction associated with the audio input.

### Claim 45 (depends on 29)

The electronic device of claim 29, wherein determining whether to initiate the virtual assistant session includes determining a proximity of the audio input from the electronic device.

### Claim 46 (depends on 29)

The electronic device of claim 29, wherein an audio model determines, at the first time, whether to initiate the virtual assistant session based on the audio input.

### Claim 47 (depends on 46)

The electronic device of claim 46, wherein the audio model is specific to a second user of the electronic device.

### Claim 48 (depends on 29)

The electronic device of claim 29, wherein the one or more programs further include instructions for:
in accordance with initiating the virtual assistant session:
determining, based on a linguistic content of the audio input, whether to deactivate the virtual assistant session;
in accordance with a determination to deactivate the virtual assistant session, deactivating the virtual assistant session; and
in accordance with a determination not to deactivate the virtual assistant session, forgoing deactivating the virtual assistant session.

### Claim 49 (depends on 29)

The electronic device of claim 29, wherein the one or more programs further include instructions for:
in accordance with initiating the virtual assistant session:
determining whether a duration of the audio input is less than a threshold duration; and
in accordance with a determination that the duration of the audio input is less than the threshold duration, deactivating the virtual assistant session.

### Claim 50

A method, comprising:
at an electronic device with one or processors, memory, a microphone and a display:
detecting input representing motion of the electronic device;
while detecting at least a portion of the input representing motion of the electronic device, sampling, with the microphone, an audio input;
determining, based on the audio input and the input representing motion of the electronic device, whether to initiate a virtual assistant session, wherein determining whether to initiate the virtual assistant session includes:
determining, at a first time, whether to initiate the virtual assistant session based on the audio input;
determining, at a second time, whether to initiate the virtual assistant session based on the input representing motion of the electronic device;
determining whether the first time and the second time are within a predetermined duration; and
determining to initiate the virtual assistant session in accordance with a determination that the first time and the second time are within the predetermined duration;
in accordance with a determination to initiate the virtual assistant session, initiating the virtual assistant session; and
in accordance with a determination not to initiate the virtual assistant session, forgoing initiating the virtual assistant session.

### Claim 51 (depends on 50)

The method of claim 50, further comprising:
in accordance with initiating the virtual assistant session:
determining a user intent based on the audio input;
performing a task based on the user intent; and
providing an output associated with the task.

### Claim 52 (depends on 50)

The method of claim 50, further comprising:
in accordance with initiating the virtual assistant session:
in accordance with not determining a user intent based on the audio input, forgoing providing an output responsive to the audio input.

### Claim 53 (depends on 50)

The method of claim 50, wherein the audio input does not include a spoken trigger.

### Claim 54 (depends on 50)

The method of claim 50, further comprising:
powering on the display based on a first portion of the input representing motion of the electronic device; and
wherein the audio input is sampled in accordance with powering on the display.

### Claim 55 (depends on 54)

The method of claim 54, wherein the first portion of the input representing motion of the electronic device represents a wrist raise gesture or a wrist rotation gesture.

### Claim 56 (depends on 50)

The method of claim 50, wherein the motion of the electronic device includes a transition of the electronic device between two or more poses of a plurality of poses.

### Claim 57 (depends on 56)

The method of claim 56, wherein the plurality of poses includes a dropped pose, a raising pose, a raised pose, and a dropping pose.

### Claim 58 (depends on 57)

The method of claim 57, wherein determining whether to initiate the virtual assistant session includes:
determining, based on the input representing motion of the electronic device, that the motion of the electronic device includes a transition from the raising pose to the raised pose.

### Claim 59 (depends on 58)

The method of claim 58, wherein determining whether to initiate the virtual assistant session includes:
determining a duration of the transition from the raising pose to the raised pose.

### Claim 60 (depends on 57)

The method of claim 57, wherein determining whether to initiate the virtual assistant session includes:
determining, based on the input representing motion of the electronic device, whether a duration of the raised pose exceeds a first predetermined duration.

### Claim 61 (depends on 57)

The method of claim 57, wherein determining whether to initiate the virtual assistant session includes:
determining, based on the input representing motion of the electronic device, whether the motion of the electronic device includes a transition from the raised pose to the dropping pose.

### Claim 62 (depends on 57)

The method of claim 57, further comprising:
in accordance with initiating the virtual assistant session:
determining, based on the input representing motion of the electronic device, whether the motion of the electronic device includes a transition from the raised pose to the dropping pose; and
in accordance with a determination that the motion of the electronic device includes the second transition from the raised pose to the dropping pose, deactivating the virtual assistant session.

### Claim 63 (depends on 50)

The method of claim 50, wherein determining whether to initiate the virtual assistant session includes determining whether the audio input includes human speech.

### Claim 64 (depends on 63)

The method of claim 63, wherein determining whether the audio input includes human speech includes determining whether the audio input includes human speech spoken by an authorized user of the electronic device.

### Claim 65 (depends on 50)

The method of claim 50, wherein determining whether to initiate the virtual assistant session includes determining a direction associated with the audio input.

### Claim 66 (depends on 50)

The method of claim 50, wherein determining whether to initiate the virtual assistant session includes determining a proximity of the audio input from the electronic device.

### Claim 67 (depends on 50)

The method of claim 50, wherein an audio model determines, at the first time, whether to initiate the virtual assistant session based on the audio input.

### Claim 68 (depends on 67)

The method of claim 67, wherein the audio model is specific to a second user of the electronic device.

### Claim 69 (depends on 50)

The method of claim 50, further comprising:
in accordance with initiating the virtual assistant session:
determining, based on a linguistic content of the audio input, whether to deactivate the virtual assistant session;
in accordance with a determination to deactivate the virtual assistant session, deactivating the virtual assistant session; and
in accordance with a determination not to deactivate the virtual assistant session, forgoing deactivating the virtual assistant session.

### Claim 70 (depends on 50)

The method of claim 50, further comprising:
in accordance with initiating the virtual assistant session:
determining whether a duration of the audio input is less than a threshold duration; and
in accordance with a determination that the duration of the audio input is less than the threshold duration, deactivating the virtual assistant session.