# US 12,119,004 B2 · Systems and methods for voice audio data processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,119,004 B2
- Document ID: US12119004B2
- Status: Granted
- Kind: B2
- Application number: 17/447,068
- Filed: 2021-09-08
- Published: 2022-03-17
- Granted: 2024-10-15
- Earliest claimed: 2020-09-17
- Assignees: Zhejiang Tonghuashun Intelligent Technology Co., Ltd.
- Applicants: Zhejiang Tonghuashun Intelligent Technology Co., Ltd.
- Inventors: Yichen Yu; Yunsan Guo
- Web page: https://patentgrep.com/patent/US12119004B2

## CPC

- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G06F 40/109: Font handling; Temporal or kinetic typography
- G06N 20/00: Machine learning
- G10L 25/27: characterised by the analysis technique
- G10L 25/63: for estimating an emotional state
- G10L 25/21: the extracted parameters being power information
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The present disclosure may provide a voice audio data processing system. The voice audio data processing system may obtain voice audio data, which includes one or more voices, each being respectively associated with one of one or more subjects. For one of the one or more voices and the subject associated with the voice, the voice audio processing system may generate a text based on the voice audio data. The text may have one or more sizes, each size corresponding to one of one or more volumes of the voice. The text may have one or more colors, each color corresponding to one of one or more emotion types of the voice.

## Claims

### Claim 1

A system, comprising:
at least one storage device including a set of instructions; and
at least one processor in communication with the at least one storage device, wherein when executing the set of instructions, the at least one processor is configured to cause the system to:
obtain voice audio data, which includes one or more voices, each being respectively associated with one of one or more subjects;
for one of the one or more voices and the subject associated with the voice, generate a text based on the voice audio data, wherein:
the text has one or more sizes, each size corresponding to one of one or more volumes of the voice, and
the text has one or more colors, each color corresponding to one of one or more emotion types of the voice;
determine location information of a voice source corresponding to the voice, wherein the location information includes at least one of a location of the voice source relative to a location of one of a plurality of voice collection modules or a distance between the voice source and one of the plurality of voice collection modules;
instruct a display device to display the text based on the location information of the voice source to indicate the location information of the voice source;
determine a first coordinate of the one of the plurality of voice collection modules in a coordinate system;
determine a second coordinate of the voice source in the coordinate system based on the first coordinate and the location information of the voice source; and
instruct the display device to display the text at the second coordinate, the text indicating the location information of the voice source.

### Claim 2 (depends on 1)

The system of claim 1, wherein the at least one processor is further configured to cause the system to instruct a display device to display the text.

### Claim 3 (depends on 2)

The system of claim 2, wherein to determine one of the one or more emotion types and the color corresponding to the emotion type, the at least one processor is configured to cause the system to:
determine, based on the voice audio data, the emotion type with a trained emotion determination model; and
determine the color corresponding to the emotion type based on the emotion type.

### Claim 4 (depends on 1)

The system of claim 1, wherein to determine the color corresponding to one of the one or more emotion types of the voice, the at least one processor is configured to cause the system to:
determine, based on the voice audio data, one or more preliminary emotion types and one or more confidence values corresponding to the one or more preliminary emotion types with a trained emotion determination model; and
determine the color corresponding to the emotion type of the voice based on the one or more preliminary emotion types and the one or more confidence values.

### Claim 5 (depends on 4)

The system of claim 4, wherein to determine the color corresponding to the emotion type of the voice based on the one or more preliminary emotion types and the one or more confidence values, the at least one processor is configured to cause the system to:
determine one or more weights corresponding to the one or more preliminary emotion types based on the one or more confidence values;
determine one or more gamuts corresponding to the one or more preliminary emotion types; and
determine the color corresponding to the emotion type of the voice based on the one or more weights and the one or more gamuts.

### Claim 6 (depends on 5)

The system of claim 5, wherein to determine the color corresponding to the emotion type of the voice based on the one or more weights and the one or more gamuts, the at least one processor is configured to cause the system to:
determine the color corresponding to the emotion type of the voice by performing a weighted summation of the one or more gamuts based on the one or more weights.

### Claim 7 (depends on 1)

The system of claim 1, wherein to determine the size corresponding to one of the one or more volumes of the voice, the at least one processor is configured to cause the system to:
obtain a reference size corresponding to a reference volume of a reference voice;
determine a ratio of the volume to the reference volume; and
determine the size corresponding to the volume of the voice based on the ratio and the reference size.

### Claim 8 (depends on 1)

The system of claim 1, wherein to determine the size corresponding to one of the one or more volumes of the voice, the at least one processor is configured to cause the system to:
obtain a first size corresponding to a predetermined maximum volume;
obtain a second size corresponding to a predetermined minimum volume; and
determine the size corresponding to the volume of the voice based on the volume of the voice, the first size, the second size, the predetermined maximum volume, and the predetermined minimum volume.

### Claim 9 (depends on 1)

The system of claim 1, wherein the voice audio data is collected by a voice collection device, the voice collection device including a plurality of voice collection modules disposed in different locations of an environment where the one or more subjects are located.

### Claim 10 (depends on 1)

The system of claim 1, wherein to determine location information of a voice source corresponding to the voice, the at least one processor is configured to cause the system to:
determine one or more phases of one or more signals corresponding to the voice collected by one or more of the plurality of voice collection modules, respectively;
determine at least one phase difference between the one or more phases; and
determine the location information of the voice source based on the at least one phase difference.

### Claim 11 (depends on 1)

The system of claim 1, wherein to determine location information of a voice source corresponding to the voice, the at least one processor is configured to cause the system to:
for each of at least two of the plurality of voice collection modules,
determine a distance between the voice source and the voice collection module; and
determine a spherical surface with a location of the voice collection module as a center and the distance as a radius; and
determine the location information of the voice source based on location information of an intersection between at least two spherical surfaces corresponding to the at least two voce collection modules.

### Claim 12 (depends on 1)

The system of claim 1, wherein the at least one processor is configured to cause the system to:
determine a third coordinate of a first voice source of a first subject;
determine a fourth coordinate of a second voice source of a second subject, the third coordinate being different from the fourth coordinate; and
instruct the display device to display a first text from the first subject at the third coordinate and display a second text from the second subject at the fourth coordinate to distinguish different contents from different subjects.

### Claim 13

A method implemented on a computing device having at least one processor, at least one storage medium, and a communication platform connected to a network, the method comprising:
obtaining voice audio data, which includes one or more voices, each being respectively associated with one of one or more subjects;
for one of the one or more voices and the subject associated with the voice, generating a text based on the voice audio data, wherein:
the text has one or more sizes, each size corresponding to one of one or more volumes of the voice, and
the text has one or more colors, each color corresponding to one of one or more emotion types of the voice;
determining location information of a voice source corresponding to the voice, wherein the location information includes at least one of a location of the voice source relative to a location of one of a plurality of voice collection modules or a distance between the voice source and one of the plurality of voice collection modules;
instructing a display device to display the text based on the location information of the voice source to indicate the location information of the voice source;
determining a first coordinate of the one of the plurality of voice collection modules in a coordinate system;
determining a second coordinate of the voice source in the coordinate system based on the first coordinate and the location information of the voice source; and
instructing the display device to display the text at the second coordinate, the text indicating the location information of the voice source.

### Claim 14 (depends on 13)

The method of claim 13, wherein the determining one of the one or more emotion types and the color corresponding to the emotion type includes:
determining, based on the voice audio data, the emotion type with a trained emotion determination model; and
determining the color corresponding to the emotion type based on the emotion type.

### Claim 15 (depends on 13)

The method of claim 13, wherein the determining the color corresponding to one of the one or more emotion types of the voice includes:
determining, based on the voice audio data, one or more preliminary emotion types and one or more confidence values corresponding to the one or more preliminary emotion types with a trained emotion determination model; and
determining the color corresponding to the emotion type of the voice based on the one or more preliminary emotion types and the one or more confidence values.

### Claim 16 (depends on 15)

The method of claim 15, wherein the determining the color corresponding to the emotion type of the voice based on the one or more preliminary emotion types and the one or more confidence values includes:
determining one or more Weights corresponding to the one or more preliminary emotion types based on the one or more confidence values;
determining one or more gamuts corresponding to the one or more preliminary emotion types; and
determining the color corresponding to the emotion type of the voice based on the one or more Weights and the one or more gamuts.

### Claim 17

A non-transitory computer readable medium, comprising executable instructions that, when executed by at least one processor, directs the at least one processor to perform a method, the method comprising:
obtaining voice audio data, which includes one or more voices, each being respectively associated with one of one or more subjects;
for one of the one or more voices and the subject associated with the voice, generating a text based on the voice audio data, wherein:
the text has one or more sizes, each size corresponding to one of one or more volumes of the voice, and
the text has one or more colors, each color corresponding to one of one or more emotion types of the voice;
determining location information of a voice source corresponding to the voice, wherein the location information includes at least one of a location of the voice source relative to a location of one of a plurality of voice collection modules or a distance between the voice source and one of the plurality of voice collection modules;
instructing a display device to display the text based on the location information of the voice source to indicate the location information of the voice source;
determining a first coordinate of the one of the plurality of voice collection modules in a coordinate system;
determining a second coordinate of the voice source in the coordinate system based on the first coordinate and the location information of the voice source; and
instructing the display device to display the text at the second coordinate, the text indicating the location information of the voice source.

### Claim 18 (depends on 17)

The non-transitory computer readable medium of claim 17, the method further comprising:
causing the system to instruct a display device to display the text.