# US 2023/0419958 A1 · Personalized multi-modal spoken language identification

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0419958 A1
- Document ID: US20230419958A1
- Status: Published application
- Kind: A1
- Application number: 17/937,692
- Filed: 2022-10-03
- Published: 2023-12-28
- Earliest claimed: 2022-06-27
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Divya Neelagiri; Cindy Sushen Tseng; Vijendra Raj Apsingekar
- Web page: https://patentgrep.com/patent/US20230419958A1

## CPC

- G10L 15/197: Probabilistic grammars, e.g. word n-grams
- G10L 15/005: {Language recognition}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A method includes obtaining an audio input of a person speaking, where the audio input is captured by an electronic device. The method also includes, for each of multiple language types, (i) determining a first probability that the person is speaking in the language type by applying a trained spoken language identification model to the audio input, (ii) determining at least one second probability that the person is speaking in the language type based on at least one characteristic of the person or the electronic device, and (iii) determining a score for the language type based on a weighted sum of the first and second probabilities. The method further includes identifying the language type associated with a highest score as a spoken language of the person in the audio input.

## Claims

### Claim 1

A method comprising:
obtaining an audio input of a person speaking, the audio input captured by an electronic device;
for each of multiple language types:
determining a first probability that the person is speaking in the language type by applying a trained spoken language identification model to the audio input;
determining at least one second probability that the person is speaking in the language type based on at least one characteristic of the person or the electronic device; and
determining a score for the language type based on a weighted sum of the first and second probabilities; and
identifying the language type associated with a highest score as a spoken language of the person in the audio input.

### Claim 2 (depends on 1)

The method of claim 1, wherein, for each of the language types, the at least one second probability comprises at least one of:
a probability of the person speaking in the language type based on a location associated with the person or the electronic device;
a probability of the person speaking in the language type based on a history of spoken language types associated with the person or the electronic device; and
a probability of the person speaking in the language type based on a keyboard language setting or a browser language setting associated with the electronic device.

### Claim 3 (depends on 1)

The method of claim 1, wherein, for each of the language types, the at least one second probability comprises:
a probability of the person speaking in the language type based on a location associated with the person or the electronic device;
a probability of the person speaking in the language type based on a history of spoken language types associated with the person or the electronic device; and
a probability of the person speaking in the language type based on a keyboard language setting or a browser language setting associated with the electronic device.

### Claim 4 (depends on 3)

The method of claim 3, wherein, for each of the language types, determining the score for the language type comprises using a formula of:
y*=argmax log P(y|x)+WlocPloc(y)+WspeechPspeech(Y)+WtextPtext(y).
where:
y* represents the score;
P(y|x) represents the first probability;
Ploc(y) represents the probability based on the location associated with the person;
Pspeech(Y) represents the probability based on the history of spoken language types associated with the person;
Ptext(y) represents the probability based on the keyboard language setting or the browser language setting associated with the electronic device; and
Wloc, Wspeech, and Wtext represent weights.

### Claim 5 (depends on 3)

The method of claim 3, wherein the history of spoken language types associated with the person includes a model trained using at least one of:
Mel frequency cepstral coefficients;
linear prediction cepstral coefficients representing vocal cord movements; and
wavelet octave coefficients of residues representing pitch, harmonic structure, and spectral energy distribution.

### Claim 6 (depends on 1)

The method of claim 1, wherein obtaining the audio input comprises obtaining the audio input from the electronic device at a second electronic device, the second electronic device identifying the language type associated with the highest score.

### Claim 7 (depends on 6)

The method of claim 6, wherein the second electronic device provides a personalized spoken language identification service for multiple people who speak different language types using the trained spoken language identification model.

### Claim 8

An apparatus comprising:
at least one processing device configured to:
obtain an audio input of a person speaking captured by an electronic device;
for each of multiple language types:
determine a first probability that the person is speaking in the language type by applying a trained spoken language identification model to the audio input;
determine at least one second probability that the person is speaking in the language type based on at least one characteristic of the person or the electronic device; and
determine a score for the language type based on a weighted sum of the first and second probabilities; and
identify the language type associated with a highest score as a spoken language of the person in the audio input.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein, for each of the language types, the at least one second probability comprises at least one of:
a probability of the person speaking in the language type based on a location associated with the person or the electronic device;
a probability of the person speaking in the language type based on a history of spoken language types associated with the person or the electronic device; and
a probability of the person speaking in the language type based on a keyboard language setting or a browser language setting associated with the electronic device.

### Claim 10 (depends on 8)

The apparatus of claim 8, wherein, for each of the language types, the at least one second probability comprises:
a probability of the person speaking in the language type based on a location associated with the person or the electronic device;
a probability of the person speaking in the language type based on a history of spoken language types associated with the person or the electronic device; and
a probability of the person speaking in the language type based on a keyboard language setting or a browser language setting associated with the electronic device.

### Claim 11 (depends on 10)

The apparatus of claim 10, wherein, for each of the language types, the score for the language type is based on a formula of:
y*=argmax log P(y|x)+WlocPloc(y)+WspeechPspeech(y)+Wtext Ptext(y).
where:
y* represents the score;
P(y|x) represents the first probability;
Ploc(y) represents the probability based on the location associated with the person;
Pspeech(Y) represents the probability based on the history of spoken language types associated with the person;
Ptext(y) represents the probability based on the keyboard language setting or the browser language setting associated with the electronic device; and
Wloc, Wspeech, and Wtext represent weights.

### Claim 12 (depends on 10)

The apparatus of claim 10, wherein the history of spoken language types associated with the person includes a model trained using at least one of:
Mel frequency cepstral coefficients;
linear prediction cepstral coefficients representing vocal cord movements; and
wavelet octave coefficients of residues representing pitch, harmonic structure, and spectral energy distribution.

### Claim 13 (depends on 8)

The apparatus of claim 8, wherein the at least one processing device is configured to obtain the audio input from the electronic device.

### Claim 14 (depends on 8)

The apparatus of claim 8, wherein the at least one processing device is configured to provide a personalized spoken language identification service for multiple people who speak different language types using the trained spoken language identification model.

### Claim 15

A non-transitory computer readable medium containing instructions that when executed cause at least one processor to:
obtain an audio input of a person speaking captured by an electronic device;
for each of multiple language types;
determine a first probability that the person is speaking in the language type by applying a trained spoken language identification model to the audio input;
determine at least one second probability that the person is speaking in the language type based on at least one characteristic of the person or the electronic device; and
determine a score for the language type based on a weighted sum of the first and second probabilities; and
identify the language type associated with a highest score as a spoken language of the person in the audio input.

### Claim 16 (depends on 15)

The non-transitory computer readable mediums of claim 15, wherein, for each of the language types, the at least one second probability comprises at least one of:
a probability of the person speaking in the language type based on a location associated with the person or the electronic device;
a probability of the person speaking in the language type based on a history of spoken language types associated with the person or the electronic device; and
a probability of the person speaking in the language type based on a keyboard language setting or a browser language setting associated with the electronic device.

### Claim 17 (depends on 15)

The non-transitory computer readable mediums of claim 15, wherein, for each of the language types, the at least one second probability comprises:
a probability of the person speaking in the language type based on a location associated with the person or the electronic device;
a probability of the person speaking in the language type based on a history of spoken language types associated with the person or the electronic device; and
a probability of the person speaking in the language type based on a keyboard language setting or a browser language setting associated with the electronic device.

### Claim 18 (depends on 17)

The non-transitory computer readable mediums of claim 17, wherein, for each of the language types, the score for the language type is based on a formula of:
y*=argmax log P(y|x)+WlocPloc(y)+WspeechPspeech(y)+Wtext Ptext(y).
where:
y* represents the score;
P(y|x) represents the first probability;
Ploc(y) represents the probability based on the location associated with the person;
Pspeech(y) represents the probability based on the history of spoken language types associated with the person;
Ptext(y) represents the probability based on the keyboard language setting or the browser language setting associated with the electronic device; and
Wloc, Wspeech, and Wtext represent weights.

### Claim 19 (depends on 17)

The non-transitory computer readable mediums of claim 17, wherein the history of spoken language types associated with the person includes a model trained using at least one of:
Mel frequency cepstral coefficients;
linear prediction cepstral coefficients representing vocal cord movements; and
wavelet octave coefficients of residues representing pitch, harmonic structure, and spectral energy distribution.

### Claim 20 (depends on 15)

The non-transitory computer readable mediums of claim 15, further containing instructions that when executed cause the at least one processor to provide a personalized spoken language identification service for multiple people who speak different language types using the trained spoken language identification model.
