US 11,538,469 B2Granted

Low-latency intelligent automated assistant

AssigneeApple Inc.

InventorsAlejandro Acero, Hepeng Zhang

Life of this patent

Estimating the term…

Loading patent…

Abstract

Systems and processes for operating a digital assistant are provided. In an example process, low-latency operation of a digital assistant is provided. In this example, natural language processing, task flow processing, dialogue flow processing, speech synthesis, or any combination thereof can be at least partially performed while awaiting detection of a speech end-point condition. Upon detection of a speech end-point condition, results obtained from performing the operations can be presented to the user. In another example, robust operation of a digital assistant is provided. In this example, task flow processing by the digital assistant can include selecting a candidate task flow from a plurality of candidate task flows based on determined task flow scores. The task flow scores can be based on speech recognition confidence scores, intent confidence scores, flow parameter scores, or any combination thereof. The selected candidate task flow is executed and corresponding results presented to the user.

Claims57

Dependent claims

A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for:

receiving a stream of audio, comprising:
receiving, from a first time to a second time, a first portion of the stream of audio containing at least a portion of a user utterance;
and
receiving, from the second time to a third time, a second portion of the stream of audio;
determining whether the first portion of the stream of audio satisfies a predetermined condition;
in response to determining that the first portion of the stream of audio satisfies the predetermined condition, performing, at least partially between the second time and the third time, operations comprising:
causing generation of a text dialogue that is responsive to the at least a portion of the user utterance;
determining whether a memory of the device stores an audio file having a spoken representation of the text dialogue;
and
in response to determining that the memory of the device does not store an audio file having a spoken representation of the text dialogue:
generating an audio file having a spoken representation of the text dialogue;
and
storing the audio file in the memory;
determining whether a speech end-point condition is detected between the second time and the third time;
and
in response to determining that the speech end-point condition is detected between the second time and the third time, outputting, to a user of the device, the spoken representation of the text dialogue by playing the stored audio file.

13 more elements

An electronic device, comprising:

one or more processors;
and
memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
receiving a stream of audio, comprising:
receiving, from a first time to a second time, a first portion of the stream of audio containing at least a portion of a user utterance;
and
receiving, from the second time to a third time, a second portion of the stream of audio;
determining whether the first portion of the stream of audio satisfies a predetermined condition;
in response to determining that the first portion of the stream of audio satisfies the predetermined condition, performing, at least partially between the second time and the third time, operations comprising:
causing generation of a text dialogue that is responsive to the at least a portion of the user utterance;
determining whether the memory of the device stores an audio file having a spoken representation of the text dialogue;
and
in response to determining that the memory of the device does not store an audio file having a spoken representation of the text dialogue:
generating an audio file having a spoken representation of the text dialogue;
and
storing the audio file in the memory;
determining whether a speech end-point condition is detected between the second time and the third time;
and
in response to determining that the speech end-point condition is detected between the second time and the third time, outputting, to a user of the device, the spoken representation of the text dialogue by playing the stored audio file.

16 more elements

A method for operating a digital assistant, the method comprising:

at an electronic device having one or more processors and memory:
receiving a stream of audio, comprising:
receiving, from a first time to a second time, a first portion of the stream of audio containing at least a portion of a user utterance;
and
receiving, from the second time to a third time, a second portion of the stream of audio;
determining whether the first portion of the stream of audio satisfies a predetermined condition;
in response to determining that the first portion of the stream of audio satisfies the predetermined condition, performing, at least partially between the second time and the third time, operations comprising:
causing generation of a text dialogue that is responsive to the at least a portion of the user utterance;
determining whether the memory of the device stores an audio file having a spoken representation of the text dialogue;
and
in response to determining that the memory of the device does not store an audio file having a spoken representation of the text dialogue:
generating an audio file having a spoken representation of the text dialogue;
and
storing the audio file in the memory;
determining whether a speech end-point condition is detected between the second time and the third time;
and
in response to determining that the speech end-point condition is detected between the second time and the third time, outputting, to a user of the device, the spoken representation of the text dialogue by playing the stored audio file.

14 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/731,122Filed 27 Apr 2022 · continuation
Priority: 12 May 2017
Published: US 2022/0254339 A111 Aug 2022
Granted: 27 Dec 2022
Term ends: Not estimatedThe source records don’t support an estimate.

Classification11 codes

G10L 15/1822
Using natural language modelling: parsing for meaning understanding
G06F 3/16
Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for…: sound…
G10L 13/04
Methods for producing synthetic speech; Speech synthesisers: details of speech synthesis systems, e.g. synthesiser structure or memory management
G10L 15/183
Using natural language modelling: using context dependencies, e.g. language models

Family

Application62/505,546Us provisional application · filed 12 May 2017
Application15/679,595Parent of this continuation · filed 17 Aug 2017
Application16/998,786Parent of this continuation · filed 20 Aug 2020
US 10,789,945 B2Parent of this continuation · filed 29 Sep 2020
US 2022/0254339 A1Pre-grant publication · published 11 Aug 2022
US 11,380,310 B2Parent of this continuation
US 2018/0330723 A1Published application
US 2020/0380966 A1Published application
US 11,538,469 B2This patent

From the US filing history. Foreign family members aren’t listed.