# US 2020/0286467 A1 · Adaptive interface in a voice-based networked system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0286467 A1
- Document ID: US20200286467A1
- Status: Published application
- Kind: A1
- Application number: 16/880,647
- Filed: 2020-05-21
- Published: 2020-09-10
- Earliest claimed: 2018-04-17
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Pu-sen CHAO; Diego Melendo CASADO; Ignacio Lopez MORENO
- Web page: https://patentgrep.com/patent/US20200286467A1

## CPC

- G10L 15/14: using statistical models, e.g. Hidden Markov Models [HMMs] (G10L15/18 takes precedence)
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G10L 15/005: {Language recognition}
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 15/183: using context dependencies, e.g. language models
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G10L 15/32: Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 2015/088: {Word spotting}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L 2015/228: {of application context}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The present disclosure relates generally to determining a language for speech recognition of a spoken utterance, received via an automated assistant interface, for interacting with an automated assistant. The system can enable multilingual interaction with the automated assistant, without necessitating a user explicitly designate a language to be utilized for each interaction. Selection of a speech recognition model for a particular language can based on one or more interaction characteristics exhibited during a dialog session between a user and an automated assistant. Such interaction characteristics can include anticipated user input types, anticipated user input durations, a duration for monitoring for a user response, and/or an actual duration of a provided user response.

## Claims

### Claim at position 1

-23. (canceled)

### Claim 24

A system to generate voice-based interfaces in networked systems, comprising:
a data processing system having one or more processors coupled with memory to:
receive first audio data corresponding to a first spoken utterance acquired via a sensor of a client device, subsequent to processing second audio data corresponding to a second spoken utterance,
identify, from a plurality of speech recognition models, a first speech recognition model for a first language based on the first audio data, the first speech recognition identified for the first language different from a second speech recognition for a second language to process the second audio data;
process the first audio data using the first speech recognition model for the first language to identify a first request;
select, from a plurality of digital components, a digital component based on the first request identified from the first audio data; and
provide, to the client device for presentation, the digital component in response to the first audio data.

### Claim 25 (depends on 24)

The system of claim 24, comprising the data processing system to:
identify an account profile for an account associated with the first spoken utterance based on the first audio data, the account profile identifying at least one of the first language or the second language; and
identify the first speech recognition model from the plurality of speech recognition models based on the account profile.

### Claim 26 (depends on 24)

The system of claim 24, comprising the data processing system to:
identify a prior usage of languages for an account associated with the first spoken utterance based on the first audio data, the prior usage of languages identifying at least the first language or the second language for the account; and
identify the first speech recognition from the plurality of speech recognition models based on the prior usage of languages.

### Claim 27 (depends on 24)

The system of claim 24, comprising the data processing system to:
identify an elapsed time between receipt of the first audio data and processing of the second audio data;
determine that the elapsed time is greater than a threshold time; and
identify from the plurality of speech recognition models in response to the determination that the elapsed time is greater than the threshold time.

### Claim 28 (depends on 24)

The system of claim 24, comprising the data processing system to:
determine that the first spoken utterance corresponds to a first user differing from a second user corresponding to the second spoken utterance based on the first audio data and the second audio data; and
identify the first speech recognition model from the plurality of speech recognition model based on the determination that the first spoken utterance corresponds to the first user differing from the second user.

### Claim 29 (depends on 24)

The system of claim 24, comprising the data processing system to:
identify an interaction characteristic associated with the first audio data corresponding to the first spoken utterance, the interaction characteristic including at least one of a monitoring duration and a response duration; and
identify the first speech recognition model from the plurality of speech recognition model based on a comparison of the interaction characteristic to a reference interaction characteristic.

### Claim 30 (depends on 24)

The system of claim 24, comprising the data processing system to:
process the first audio data using the first speech recognition model for the first language to identify a keyword in the first request, the first keyword in the first language; and
select, from the plurality of digital components, the digital component based on the keyword identified in the first request.

### Claim 31 (depends on 24)

The system of claim 24, comprising the data processing system to:
determine, using the first speech recognition model, a first probability metric that the first spoken utterance is in the first language;
determine, using the second speech recognition model, a second probability metric that the first spoken utterance is in the second language; and
identify the first speech recognition model from the plurality of speech recognition models based on the first probability metric and the second probability metric.

### Claim 32 (depends on 24)

The system of claim 24, comprising the data processing system to select the digital component in the first language corresponding to the first spoken utterance, the digital component different from a second digital component selected in response to the second audio data, the second digital component in the second language corresponding to the second spoken utterance.

### Claim 33 (depends on 24)

The system of claim 24, comprising the data processing system to:
provide, to the client device for presentation, a prompt to receive additional input in response to the second audio data; and
receive, from the client device, the first audio data generated in response to presentation of the prompt on the client device.

### Claim 34

A method of generating voice-based interfaces in networked systems, comprising:
receiving, by a data processing system, first audio data corresponding to a first spoken utterance acquired via a sensor of a client device, subsequent to processing second audio data corresponding to a second spoken utterance,
identifying, by the data processing system, from a plurality of speech recognition models, a first speech recognition model for a first language based on the first audio data, the first speech recognition identified for the first language different from a second speech recognition for a second language to process the second audio data;
processing, by the data processing system, the first audio data using the first speech recognition model for the first language to identify a first request;
selecting, by the data processing system, from a plurality of digital components, a digital component based on the first request identified from the first audio data; and
providing, by the data processing system, to the client device for presentation, the digital component in response to the first audio data.

### Claim 35 (depends on 34)

The method of claim 34, comprising:
identifying, by the data processing system, an account profile for an account associated with the first spoken utterance based on the first audio data, the account profile identifying at least one of the first language or the second language; and
identifying, by the data processing system, the first speech recognition model from the plurality of speech recognition models based on the account profile.

### Claim 36 (depends on 34)

The method of claim 34, comprising:
identifying, by the data processing system, a prior usage of languages for an account associated with the first spoken utterance based on the first audio data, the prior usage of languages identifying at least the first language or the second language for the account; and
identifying, by the data processing system, the first speech recognition from the plurality of speech recognition models based on the prior usage of languages.

### Claim 37 (depends on 34)

The method of claim 34, comprising:
identifying, by the data processing system, an elapsed time between receipt of the first audio data and processing of the second audio data;
determining, by the data processing system, that the elapsed time is greater than a threshold time; and
identifying, by the data processing system, from the plurality of speech recognition models in response to the determination that the elapsed time is greater than the threshold time.

### Claim 38 (depends on 34)

The method of claim 34, comprising:
determining, by the data processing system, that the first spoken utterance corresponds to a first user differing from a second user corresponding to the second spoken utterance based on the first audio data and the second audio data; and
identifying, by the data processing system, the first speech recognition model from the plurality of speech recognition model based on the determination that the first spoken utterance corresponds to the first user differing from the second user.

### Claim 39 (depends on 34)

The method of claim 34, comprising:
identifying, by the data processing system, an interaction characteristic associated with the first audio data corresponding to the first spoken utterance, the interaction characteristic including at least one of a monitoring duration and a response duration; and
identifying, by the data processing system, the first speech recognition model from the plurality of speech recognition model based on a comparison of the interaction characteristic to a reference interaction characteristic

### Claim 40 (depends on 34)

The method of claim 34, comprising:
processing, by the data processing system, the first audio data using the first speech recognition model for the first language to identify a keyword in the first request, the first keyword in the first language; and
selecting, by the data processing system, from the plurality of digital components, the digital component based on the keyword identified in the first request.

### Claim 41 (depends on 34)

The method of claim 34, comprising
determining, by the data processing system, using the first speech recognition model, a first probability metric that the first spoken utterance is in the first language;
determining, by the data processing system, using the second speech recognition model, a second probability metric that the first spoken utterance is in the second language; and
identifying, by the data processing system, the first speech recognition model from the plurality of speech recognition models based on the first probability metric and the second probability metric.

### Claim 42 (depends on 34)

The method of claim 34, comprising selecting, by the data processing system, the digital component in the first language corresponding to the first spoken utterance, the digital component different from a second digital component selected in response to the second audio data, the second digital component in the second language corresponding to the second spoken utterance.

### Claim 43 (depends on 34)

The method of claim 34, comprising:
providing, by the data processing system, to the client device for presentation, a prompt to receive additional input in response to the second audio data; and
receiving, by the data processing system, from the client device, the first audio data generated in response to presentation of the prompt on the client device.
