# US 12,002,486 B2 · Tag estimation device, tag estimation method, and program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,486 B2
- Document ID: US12002486B2
- Status: Granted
- Kind: B2
- Application number: 17/279,009
- Filed: 2019-09-13
- Published: 2022-02-03
- Granted: 2024-06-04
- Earliest claimed: 2018-09-26
- Assignees: Nippon Telegraph and Telephone Corporation
- Applicants: Nippon Telegraph and Telephone Corporation
- Inventors: Ryo Masumura; Tomohiro Tanaka
- Web page: https://patentgrep.com/patent/US12002486B2

## CPC

- G10L 25/30: using neural networks
- G10L 25/51: for comparison or discrimination
- G06F 16/00: Information retrieval; Database structures therefor; File system structures therefor
- G06F 16/65: Clustering; Classification
- G06F 16/90: Details of database functions independent of the retrieved data types
- G06F 16/906: Clustering; Classification
- G06F 40/274: Converting codes to words; Guess-ahead of partial word inputs
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/10: using distance or distortion measures between unknown speech and reference templates
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 25/48: specially adapted for particular use
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A tag estimation device capable of estimating, for an utterance made among several persons, a tag representing a result of analyzing the utterance is provided. The tag estimation device includes an utterance sequence information vector generation unit that adds a t-th utterance word feature vector and a t-th speaker vector to a (t−1)-th utterance sequence information vector ut-1 that includes an utterance word feature vector that precedes the t-th utterance word feature vector and a speaker vector that precedes the t-th speaker vector to generate a t-th utterance sequence information vector ut, where t is a natural number, and a tagging unit that determines a tag lt that represents a result of analyzing a t-th utterance from a model parameter set in advance and the t-th utterance sequence information vector ut.

## Claims

### Claim 1

A tag estimation device comprising:
a hardware processor that:
retrieves a t-th utterance of a sequence of utterances in a dialogue from memory, wherein the sequence of utterances was stored in the memory following receipt of the sequence of utterances from a microphone or from a transmission of data over a communication line, wherein the t-th utterance is spoken by a speaker of a plurality of speakers in the dialogue, the t-th utterance includes a word, and the t is a natural number;
generates, based on the t-th utterance, a t-th speaker vector and a t-th utterance word feature vector;
retrieves from the memory a (t−1)-th utterance sequence information vector ut-1 that includes an utterance word feature vector that precedes the t-th utterance word feature vector and a speaker vector that precedes the t-th speaker vector,
generates, by a recurrent neural network (RNN), based on the t-th utterance word feature vector, the t-th speaker vector, and the (t−1)-th utterance sequence information vector ut-1, a t-th utterance sequence information vector ut,
wherein the t-th utterance sequence information vector ut, is based on recursively combining pieces of a series of respective uttered word feature vectors and speaker vectors from a first utterance to the t-th utterance of the sequence of utterances in the dialogue, each speaker vector of an utterance in the sequence of utterances represents a speaker of the plurality of speakers speaking the utterance,
wherein the t-th utterance word feature vector is associated with at least the word in the t-th utterance spoken by the speaker, the t-th speaker vector is associated with the speaker, and the t-th utterance sequence information vector ut represents a feature associated with the t-th utterance in a sequence of utterances in a dialogue, the generating the t-th utterance sequence information vector ut further comprises operating according to the formula:
ct=[wtT,rtT]T, and
ut=RNN(ct, ut-1), the RNN represents a function having capabilities of a recurrent neural network, the T represents a transposition of a vector, the wt represents an utterance word feature vector of the t-th utterance, and the rt represents a speaker vector of the t-th utterance;
determines a tag lt associated with the t-th utterance, wherein the tag lt represents a result of analyzing the t-th utterance from a predetermined model parameter and the t-th utterance sequence information vector ut, wherein the tag lt specifies a scene in the dialogue;
storing in the memory the t-th speaker vector, the t-th utterance word feature vector, and the i-th utterance sequence information vector ut for performing tag estimation of a subsequent utterance; and
transmitting the tag lt associated with the t-th utterance to program instructions configured to output the tag lt.

### Claim 2 (depends on 1)

The tag estimation device according to claim 1, comprising:
the hardware processor that:
transforms a word sequence of the t-th utterance into the utterance word feature vector of the t-th utterance; and
transforms a speaker label of the t-th utterance into the speaker vector of the t-th utterance.

### Claim 3 (depends on 1)

The tag estimation device according to claim 1, comprising:
the hardware processor that:
transforms a word sequence of the t-th utterance into the utterance word feature vector of the t-th utterance; and
transforms voice of the t-th utterance into the speaker vector of the t-th utterance.

### Claim 4 (depends on 1)

The tag estimation device according to claim 1, comprising:
the hardware processor that:
transforms voice of the t-th utterance into the utterance word feature vector of the t-th utterance; and
transforms a speaker label of the t-th utterance into the speaker vector of the t-th utterance.

### Claim 5 (depends on 1)

The tag estimation device according to claim 1, comprising:
the hardware processor that:
transforms voice of the t-th utterance into the utterance word feature vector of the t-th utterance; and
transforms the voice of the t-th utterance into the speaker vector of the t-th utterance.

### Claim 6 (depends on 1)

The tag estimation device according to claim 1, wherein
the tag includes at least any one of utterance scenes, an utterance type, an emotion of an utterance utterer, and utterance paralinguistic information.

### Claim 7 (depends on 1)

The tag estimation device according to claim 1, wherein the hardware processor uses an estimation model that is learned on a basis of teacher data including a set of an utterance sequence and a sequence of labels that are right tags corresponding to the utterance sequence.

### Claim 8 (depends on 1)

A non-transitory computer-readable storage medium storing a program for causing a computer to function as a tag estimation device according to claim 1.

### Claim 9

A tag estimation method to be executed by a tag estimation device, comprising:
retrieving a t-th utterance of a sequence of utterances in a dialogue from at least one of microphone input, from a memory, or transmission of data over a communication line, wherein the t-th utterance is spoken by a speaker of a plurality of speakers in the dialogue, the t-th utterance includes a word, and the t is a natural number;
generating, based on the t-th utterance, a t-th speaker vector and a t-th utterance word feature vector;
retrieving from the memory a (t−1)-th utterance sequence information vector ut-1 that includes an utterance word feature vector that precedes the t-th utterance word feature vector and a speaker vector that precedes the t-th speaker vector;
generating, by a recurrent neural network, based on the t-th utterance word feature vector and the t-th speaker vector to the (t−1)-th utterance sequence information ut-1, a t-th utterance sequence information vector ut,
wherein the t-th utterance sequence information vector ut is based on recursively combining pieces of a series of respective uttered word feature vectors and speaker vectors from a first utterance to the t-th utterance of the sequence of utterances in the dialogue, each speaker vector of an utterance in the sequence of utterances represents a speaker of the plurality of speakers speaking the utterance,
wherein the t-th utterance word feature vector is associated with at least the word in the t-th utterance spoken by the speaker, the t-th speaker vector is associated with the speaker, and the t-th utterance sequence information vector ut represents a feature associated with the t-th utterance in a sequence of utterances in a dialogue, and the generating the t-th utterance sequence information vector ut further comprises operating according to the formula:
ct=[wtT,rtT]T, and
ut=RNN(ct, ut-1), the RNN represents a function having capabilities of a recurrent neural network, the T represents a transposition of a vector, the wt represents an utterance word feature vector of the t-th utterance, and the rt represents a speaker vector of the t-th utterance;
determining a tag lt associated with the t-th utterance, wherein the tag lt represents a result of analyzing the t-th utterance from a predetermined model parameter and the t-th utterance sequence information vector ut, wherein the tag lt specifies a scene of the dialogue;
storing in the memory the t-th speaker vector and the t-th utterance word feature vector, and the i-th utterance sequence information vector ut for performing tag estimation of a subsequent utterance; and
transmitting the tag lt associated with the t-th utterance to program instructions configured to output the tag lt.