# US 2026/0088039 A1 · Apparatus and method using joint of discrete emotional representation and dimensional emotional representation for speech emotion recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0088039 A1
- Document ID: US20260088039A1
- Status: Published application
- Kind: A1
- Application number: 19/308,548
- Filed: 2025-08-25
- Published: 2026-03-26
- Earliest claimed: 2024-09-20
- Assignees: Electronics and Telecommunications Research Institute
- Applicants: Electronics and Telecommunications Research Institute
- Inventors: John Lorenzo BAUTISTA; HyunSoon SHIN; Yun Kyung LEE
- Web page: https://patentgrep.com/patent/US20260088039A1

## CPC

- G10L 25/63: for estimating an emotional state
- G10L 25/18: the extracted parameters being spectral information of each sub-band
- G10L 25/24: the extracted parameters being the cepstrum
- G10L 25/30: using neural networks
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Disclosed is a speech emotion recognition apparatus. The speech emotion recognition apparatus includes a processor. The processor generates result data derived from a speech signal using an artificial neural network model. The artificial neural network model includes an encoder layer, an attention layer, and an output layer. The encoder layer outputs a plurality of latent features based on the speech signal and a pre-processed signal obtained by preprocessing the speech signal. The attention layer outputs attention data by applying a self-attention mechanism and a co-attention mechanism to the plurality of latent features. The output layer outputs the result data including probability values associated with discrete emotional representations and numerical values associated with dimensional emotional representations based on the attention data.

## Claims

### Claim 1

A speech emotion recognition apparatus comprising:
a processor configured to receive a speech signal and to generate result data derived from the speech signal using an artificial neural network model, and
wherein the artificial neural network model includes:
an encoder layer configured to output a plurality of latent features based on the speech signal and a pre-processed signal obtained by preprocessing the speech signal;
an attention layer configured to output attention data by applying a self-attention mechanism and a co-attention mechanism to the plurality of latent features; and
an output layer configured to output the result data including probability values associated with discrete emotional representations and numerical values associated with dimensional emotional representations based on the attention data.

### Claim 2 (depends on 1)

The speech emotion recognition apparatus of claim 1, wherein the encoder layer includes a plurality of feature encoders,
wherein some of the feature encoders are configured to generate some of the plurality of latent features based on the speech signal, and
wherein others of the feature encoders are configured to generate other some of the plurality of latent features based on the speech signal or the pre-processed signal.

### Claim 3 (depends on 2)

The speech emotion recognition apparatus of claim 2, wherein the pre-processed signal includes a Mel-spectrogram, an STF (short-term feature), and MFCC (Mel-Frequency Cepstral Coefficients), which are related to the speech signal.

### Claim 4 (depends on 3)

The speech emotion recognition apparatus of claim 3, wherein the plurality of latent features include a first latent feature, a second latent feature, a third latent feature, and a fourth latent feature, and
wherein the plurality of feature encoders include:
a first feature encoder configured to generate the first latent feature based on the speech signal;
a second feature encoder configured to generate the second latent feature based on the Mel-spectrogram;
a third feature encoder configured to generate the third latent feature based on the STF; and
a fourth feature encoder configured to generate the fourth latent feature based on the MFCC.

### Claim 5 (depends on 4)

The speech emotion recognition apparatus of claim 4, wherein the first feature encoder is configured to use a first learning model,
wherein the second feature encoder is configured to use a CNN (Convolutional Neural Network) model, and
wherein each of the third feature encoder and the fourth feature encoder is configured to use a CNN-LSTM (CNN-Long Short-Term Memory) model.

### Claim 6 (depends on 1)

The speech emotion recognition apparatus of claim 1, wherein the attention layer includes:
a plurality of self-attention sub-layers configured to generate a plurality of self-attention result values respectively corresponding to the plurality of latent features; and
a co-attention sub-layer configured to generate a co-attention result value corresponding to all of the plurality of latent features.

### Claim 7 (depends on 6)

The speech emotion recognition apparatus of claim 6, wherein the encoder layer is configured to provide the plurality of latent features to the plurality of self-attention sub-layers, and
wherein the plurality of self-attention sub-layers are configured to provide the plurality of self-attention result values to the co-attention sub-layer.

### Claim 8 (depends on 7)

The speech emotion recognition apparatus of claim 7, wherein the attention layer further includes a matrix concatenator configured to concatenate the plurality of self-attention result values and the co-attention result value to generate the attention data.

### Claim 9 (depends on 1)

The speech emotion recognition apparatus of claim 1, wherein the output layer includes:
a first sub-output layer configured to generate the probability values associated with the discrete emotional representations; and
a second sub-output layer configured to generate the numerical values associated with the dimensional emotional representations.

### Claim 10 (depends on 9)

The speech emotion recognition apparatus of claim 9, wherein the first sub-output layer is configured to calculate the probability values associated with the discrete emotional representations based on a first activation function, and
wherein the second sub-output layer is configured to calculate the numerical values associated with the dimensional emotional representations based on a second activation function.

### Claim 11 (depends on 10)

The speech emotion recognition apparatus of claim 10, wherein the first activation function includes a softmax function, and
wherein the second activation function includes a linear function.

### Claim 12 (depends on 1)

The speech emotion recognition apparatus of claim 1, wherein the processor is configured to:
generate a joint loss function based on a first loss function associated with the discrete emotional representations and a second loss function associated with the dimensional emotional representations; and
train the artificial neural network model based on the joint loss function.

### Claim 13 (depends on 12)

The speech emotion recognition apparatus of claim 12, wherein the processor is configured to, based on a first coefficient, perform a matrix addition of the first loss function and the second loss function to generate the joint loss function.

### Claim 14 (depends on 13)

The speech emotion recognition apparatus of claim 13, wherein the processor updates the first coefficient at each epoch associated with the training of the artificial neural network model based on one of a uniform weighting method, a task-specific weighting method, a dynamic weighting method, and a joint weighting method.

### Claim 15

A speech emotion recognition method using an artificial neural network model, the method comprising:
outputting a plurality of latent features based on a speech signal;
outputting attention data by applying a self-attention mechanism and a co-attention mechanism to the plurality of latent features; and
outputting result data including probability values associated with discrete emotional representations and numerical values associated with dimensional emotional representations based on the attention data.

### Claim 16 (depends on 15)

The method of claim 15, wherein the plurality of latent features include a first latent feature, a second latent feature, a third latent feature, and a fourth latent feature, and
wherein the outputting of the plurality of latent features includes:
generating the first latent feature based on the speech signal;
generating the second latent feature based on a Mel-spectrogram associated with the speech signal;
generating the third latent feature based on an STF associated with the speech signal; and
generating the fourth latent feature based on an MFCC associated with the speech signal.

### Claim 17 (depends on 15)

The method of claim 15, wherein the outputting of the attention data includes:
generating a plurality of self-attention result values respectively corresponding to the plurality of latent features;
generating a co-attention result value corresponding to all of the latent features; and
generating the attention data by concatenating the plurality of self-attention result values and the co-attention result value.

### Claim 18 (depends on 17)

The method of claim 17, wherein the generating of the plurality of self-attention result values includes calculating the plurality of self-attention result values by applying a self-attention mechanism to the latent features, and
wherein the generating of the co-attention result value includes calculating the co-attention result value by applying a co-attention mechanism to the plurality of self-attention result values.

### Claim 19 (depends on 15)

The method of claim 15, wherein the outputting of the result data includes:
generating the probability values associated with the discrete emotional representations based on a first activation function; and
generating the numerical values associated with the dimensional emotional representations based on a second activation function.

### Claim 20 (depends on 15)

The method of claim 15, further comprising:
generating a joint loss function based on a first loss function associated with the discrete emotional representations and a second loss function associated with the dimensional emotional representations; and
training the artificial neural network model based on the joint loss function.
