# US 2019/0172444 A1 · Spoken dialog device, spoken dialog method, and recording medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0172444 A1
- Document ID: US20190172444A1
- Status: Published application
- Kind: A1
- Application number: 16/320,810
- Filed: 2017-07-24
- Published: 2019-06-06
- Earliest claimed: 2016-07-28
- Assignees: National Institute of Information and Communications Technology
- Applicants: National Institute of Information and Communications Technology
- Inventors: Atsuo HIROE; Takuma OKAMOTO
- Web page: https://patentgrep.com/patent/US20190172444A1

## CPC

- G10L 15/005: {Language recognition}
- G06F 3/16: Sound input; Sound output (speech processing G10L)
- G06F 40/268: Morphological analysis
- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 13/086: {Detection of language}
- G10L 15/00: Speech recognition (G10L17/00 takes precedence)
- G10L 15/04: Segmentation; Word boundary detection
- G10L 15/10: using distance or distortion measures between unknown speech and reference templates
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G10L 2015/225: {Feedback of the input speech}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

[Problem] With conventional technology, it is impossible to appropriately support spoken dialog that is carried out in multiple languages. [Solution] A spoken dialog device includes: a receiving unit that detects a voice section from a start point to an end point of an input speech that is spoken in any of two or more different languages, and acquires speech data corresponding to the voice section; a language identifier acquisition unit that acquires a language identifier that identifies a language in which the input speech was spoken; a speech recognition unit that generates a text resulting from speech recognition, based on the input speech and the language identifier; a dialog control unit to which a text resulting from speech recognition and a language identifier are input, and that generates a different output sentence depending on a language identifier, while maintaining dialog history even when the language identifier is different from the previous language identifier; a speech synthesizing unit that generates a speech waveform based on the output sentence and the language identifier; and a speech output unit that outputs a speech that is based on a speech waveform generated by the speech synthesizing unit. With such a spoken dialog device, it is possible to appropriately support spoken dialog that is carried out in multiple languages.

## Claims

### Claim 1

A spoken dialog device comprising:
a receiving unit that detects a voice section from a start point to an end point of an input speech that is spoken in any of two or more different languages, and acquires speech data corresponding to the voice section;
a language identifier acquisition unit that acquires a language identifier that identifies a language in which the input speech was spoken;
a speech recognition unit that generates a text resulting from speech recognition, based on the input speech and the language identifier;
a dialog control unit to which a text resulting from speech recognition and a language identifier are input, and that generates a different output sentence depending on a language identifier, while maintaining dialog history even when the language identifier is different from the previous language identifier;
a speech synthesizing unit that generates a speech waveform based on an output sentence and a language identifier; and
a speech output unit that outputs a speech that is based on a speech waveform acquired by the speech synthesizing unit.

### Claim 2 (depends on 1)

The spoken dialog device according to claim 1, further comprising:
a dialog structure information storage unit configured to store dialog structure information that includes one or more pieces of dialog link information indicating links between dialogs and each including a first dialog action tag that abstracts a speech input by a first speaker, and a second dialog action tag that abstracts an output speech output by a second speaker; and
an output sentence information storage unit configured to store output speech information that includes a second dialog action tag and one or more sentences,
wherein the dialog control unit includes:
a first dialog action tag acquisition subunit that uses a character string acquired by the speech recognition subunit, to acquire, from the dialog structure information storage unit, a first dialog action tag corresponding to the character string;
a second dialog action tag acquisition subunit that acquires one second dialog action tag out of one or more second dialog action tags corresponding to the first dialog action tag acquired by the first dialog action tag acquisition subunit;
a sentence acquisition subunit that acquires, from the output sentence information storage unit, a sentence corresponding to the second dialog action tag acquired by the second dialog action tag acquisition subunit; and
a sentence changing unit that changes the sentence acquired by the sentence acquisition unit according to the language identifier acquired by the language identifier acquisition unit, and acquires the sentence thus changed.

### Claim 3 (depends on 1)

The spoken dialog device according to claim 1, further comprising:
a dialog structure information storage unit configured to store dialog structure information that includes one or more pieces of dialog link information indicating links between dialogs and each including a first dialog action tag that abstracts a speech input by a first speaker, and a second dialog action tag that abstracts an output speech output by a second speaker, the dialog structure information including two or more second dialog action tags that correspond to a language identifier and that are linked to a first dialog action tag; and
an output sentence information storage unit configured to store output speech information that includes a second dialog action tag and one or more sentences,
wherein the dialog control unit includes:
a first dialog action tag acquisition subunit that uses a character string acquired by the speech recognition subunit, to acquire, from the dialog structure information storage unit, a first dialog action tag corresponding to the character string;
a second dialog action tag acquisition subunit that acquires one second dialog action tag that corresponds to the language identifier acquired by the language identifier acquisition unit, out of one or more second dialog action tags corresponding to the first dialog action tag acquired by the first dialog action tag acquisition subunit; and
a sentence acquisition subunit that acquires, from the output sentence information storage unit, a sentence corresponding to the second dialog action tag acquired by the second dialog action tag acquisition subunit.

### Claim 4

A spoken dialog method realized using a receiving unit, a language identifier acquisition unit, a speech recognition unit, a dialog control unit, a speech synthesizing unit, and a speech output unit, the spoken dialog method comprising:
a receiving step in which the receiving unit receives an input speech that is spoken in any of two or more different languages;
a language identifier acquisition step in which the language identifier acquisition unit acquires a language identifier that identifies a language in which the input speech was spoken;
a speech recognition step in which a speech recognition unit generates a text resulting from speech recognition, based on the input speech and the language identifier;
a dialog control step in which the dialog control unit generates an output sentence based on the result of speech recognition and the language identifier;
a speech synthesizing step of generating a speech waveform based on the output sentence and the language identifier; and
a speech output step in which the speech output unit outputs a speech based on the speech waveform acquired in the speech acquisition step.

### Claim 5

A recording medium on which a program is recorded, the program causing a computer to function as:
a receiving unit that detects a voice section from a start point to an end point of an input speech that is spoken in any of two or more different languages, and acquires speech data corresponding to the voice section;
a language identifier acquisition unit that acquires a language identifier that identifies a language in which the input speech was spoken;
a speech recognition unit that generates a text resulting from speech recognition based on the input speech and the language identifier;
a dialog control unit to which a text resulting from speech recognition and a language identifier are input, and that generates a different output sentence depending on a language identifier, while maintaining dialog history even when the language identifier is different from the previous language identifier;
a speech synthesizing unit that generates a speech waveform based on the output sentence and the language identifier; and
a speech output unit that outputs a speech that is based on a speech waveform acquired by the speech synthesizing unit.
