# US 2024/0404511 A1 · Method and system for fair speech emotion recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0404511 A1
- Document ID: US20240404511A1
- Status: Published application
- Kind: A1
- Application number: 18/380,847
- Filed: 2023-10-17
- Published: 2024-12-05
- Earliest claimed: 2023-06-02
- Assignees: National Tsing Hua University
- Applicants: National Tsing Hua University
- Inventors: WOAN-SHIUAN CHIEN; CHI-CHUN LEE
- Web page: https://patentgrep.com/patent/US20240404511A1

## CPC

- G10L 15/063: {Training}
- G10L 25/30: using neural networks
- G10L 25/63: for estimating an emotional state
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A computer-implemented method for speech emotion recognition is provided. The computer-implemented method includes an emotion prediction corresponding to speech data that is generated based on the speech data and a speech emotion recognition model without bias. The speech emotion recognition model without bias is trained by training a fairness-constrained adversarial network based on a labeled training set with known bias and a loss function. The fairness-constrained adversarial network includes a domain classifier for bias classification and the speech emotion recognition model. The loss function used for training the fairness-constrained adversarial network is positively related to a Wasserstein distance (WD) loss.

## Claims

### Claim 1

A computer-implemented method for training a speech emotion recognition model, comprising:
providing a fairness-constrained adversarial network comprising a domain classifier and a speech emotion recognition model, the domain classifier configured to classify a bias in an input data, and the speech emotion recognition model configured to determine an emotion corresponding to the input data; and
training the fairness-constrained adversarial network based on a labeled training set with known bias and a first loss function to obtain the speech emotion recognition model without bias,
wherein the first loss function is positively related to a Wasserstein distance loss.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the first loss function is negatively related to a first loss associated with the domain classifier and positively related to a second loss associated with the speech emotion recognition model.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the labeled training set with known bias comprises a plurality of training data, each of the plurality of training data corresponds to an overall emotion label and a plurality of perspective emotion labels for a plurality of perspectives.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, further comprising:
generating a fairness embedding set corresponding to the labeled training set based on the speech emotion recognition model without bias; and
training a multi-perspective speech emotion recognition model based on the fairness embedding set and a second loss function,
wherein the multi-perspective speech emotion recognition model is configured to generate a plurality of perspective-specific emotion prediction results corresponding to the plurality of perspectives, and the second loss function is positively related to a metric learning loss.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the metric learning loss comprises a triplet loss, and an anchor, a positive sample, and a negative sample of the triplet loss are set based on the overall emotion label and the plurality of perspective emotion labels.

### Claim 6 (depends on 4)

The computer-implemented method of claim 4, wherein the metric learning loss comprises at least one of a triplet loss, a quadruplet loss, an N-pair loss, a contrastive loss, and a center loss.

### Claim 7 (depends on 4)

The computer-implemented method of claim 4, wherein the second loss function is positively related to a third loss associated with the multi-perspective speech emotion recognition model, and the third loss is associated with the plurality of perspective emotion labels.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the domain classifier is configured to classify a binary bias.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein the binary bias comprises an evaluator gender bias.

### Claim 10

A computer-implemented method for speech emotion recognition, comprising:
receiving speech data; and
generating an emotion prediction result corresponding to the speech data based on the speech data and a speech emotion recognition model without bias, wherein:
the speech emotion recognition model without bias is trained by training a fairness-constrained adversarial network based on a labeled training set with known bias and a first loss function;
the first loss function is positively related to a Wasserstein distance loss; and
the fairness-constrained adversarial network comprises a domain classifier for bias classification and the speech emotion recognition model.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, wherein the first loss function is negatively related to a first loss associated with the domain classifier and positively related to a second loss associated with the speech emotion recognition model.

### Claim 12 (depends on 10)

The computer-implemented method of claim 10, wherein the labeled training set with known bias comprises a plurality of training data, and each of the plurality of training data corresponds to an overall emotion label and a plurality of perspective emotion labels for a plurality of perspectives.

### Claim 13 (depends on 12)

The computer-implemented method of claim 12, further comprising:
generating a fairness embedding corresponding to the speech data based on the speech data and the speech emotion recognition model without bias; and
generating at least one of a plurality of perspective-specific emotion prediction results corresponding to the speech data based on the fairness embedding and a multi-perspective speech emotion recognition model, wherein:
the multi-perspective speech emotion recognition model is trained based on a fairness embedding set corresponding to the labeled training set and a second loss function;
the fairness embedding set is generated based on the speech emotion recognition model without bias; and
the second loss function is positively related to a metric learning loss.

### Claim 14 (depends on 13)

The computer-implemented method of claim 13, wherein the metric learning loss comprises a triplet loss, and an anchor, a positive sample, and a negative sample of the triplet loss are set based on the overall emotion label and the plurality of perspective emotion labels.

### Claim 15 (depends on 13)

The computer-implemented method of claim 13, wherein the metric learning loss comprises at least one of a triplet loss, a quadruplet loss, an N-pair loss, a contrastive loss, or a center loss.

### Claim 16 (depends on 13)

The computer-implemented method of claim 13, wherein the second loss function is positively related to a third loss associated with the multi-perspective speech emotion recognition model, and the third loss is associated with the plurality of perspective emotion labels.

### Claim 17 (depends on 10)

The computer-implemented method of claim 10, wherein the domain classifier is configured to classify a binary bias.

### Claim 18 (depends on 17)

The computer-implemented method of claim 17, wherein the binary bias comprises an evaluator gender bias.

### Claim 19

A non-transitory computer-readable medium, comprising at least one instruction, when executed by a processor of an electronic device, causes the electronic device to:
receive speech data; and
generate an emotion prediction result corresponding to the speech data based on the speech data and a speech emotion recognition model without bias, wherein:
the speech emotion recognition model without bias is trained by training a fairness- constrained adversarial network based on a labeled training set with known bias and a first loss function;
the first loss function is positively related to a Wasserstein distance loss; and
the fairness-constrained adversarial network comprises a domain classifier for bias classification and the speech emotion recognition model.
