# US 12,119,005 B2 · Audio data processing method for wake-up speech detection, apparatus, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,119,005 B2
- Document ID: US12119005B2
- Status: Granted
- Kind: B2
- Application number: 18/323,496
- Filed: 2023-05-25
- Published: 2023-09-21
- Granted: 2024-10-15
- Earliest claimed: 2018-11-23
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Yi Gao
- Web page: https://patentgrep.com/patent/US12119005B2

## CPC

- G10L 17/20: Pattern transformations or operations aimed at increasing system robustness, e.g. against channel noise or different working conditions
- G10L 21/02: Speech enhancement, e.g. noise reduction or echo cancellation (reducing echo effects in line transmission systems H04B3/20; echo suppression in hands-free telephones H04M9/08)
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G10L 15/08: Speech classification or search
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 17/02: Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction
- G10L 17/06: Decision making techniques; Pattern matching strategies
- G10L 17/22: Interactive procedures; Man-machine interfaces
- G10L 21/0232: Processing in the frequency domain
- G10L 25/18: the extracted parameters being spectral information of each sub-band
- H04R 1/406: {microphones}
- H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)}
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/32: Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
- G10L 21/0208: Noise filtering
- G10L 2015/088: {Word spotting}
- G10L 2021/02082: {the noise being echo, reverberation of the speech}
- G10L 2021/02166: {Microphone arrays; Beamforming}
- H04R 2201/401: 2D or 3D arrays of transducers
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)

## Abstract

An audio data processing method is provided. The method includes: obtaining multi-path audio data in an environmental space, obtaining a speech data set based on the multi-path audio data, and separately generating, in a plurality of enhancement directions, enhanced speech information corresponding to the speech data set; matching a speech hidden feature in the enhanced speech information with a target matching word, and determining an enhancement direction corresponding to the enhanced speech information having a highest degree of matching with the target matching word as a target audio direction; obtaining speech spectrum features in the enhanced speech information, and obtaining, from the speech spectrum features, a speech spectrum feature in the target audio direction; and performing speech authentication on the speech hidden feature and the speech spectrum feature that are in the target audio direction based on the target matching word, to obtain a target authentication result.

## Claims

### Claim 1

An audio data processing method, performed by an audio data processing device, the method comprising:
obtaining multi-path audio data in an environmental space, obtaining a speech data set based on the multi-path audio data, and separately generating, in a plurality of enhancement directions, a plurality of pieces of enhanced speech information corresponding to the speech data set;
matching, by using a first wake-up detection model, a speech hidden feature in each piece of enhanced speech information with a target matching word to obtain a degree of matching with the target matching word, and determining an enhancement direction corresponding to target enhanced speech information having a highest degree of matching with the target matching word among the plurality of pieces of enhancement speech information as a target audio direction;
obtaining, by using a second wake-up detection model, a speech spectrum feature of the target enhanced speech information in the target audio direction; and
determining a probability of existence of the target matching word in the target audio direction for controlling a terminal, comprising:
concatenating the speech hidden feature and the speech spectrum feature of the target enhanced speech information in the target audio direction, to obtain a concatenated vector feature; and
determining, by using the second wake-up detection model, a degree of matching between the concatenated vector feature and a target wake-up feature, to generate the probability of existence of the target matching word.

### Claim 2 (depends on 1)

The method according to claim 1, further comprising:
storing, in a buffer of the second wake-up detection model, the speech hidden feature of the target enhanced speech information output by the first wake-up detection model,
wherein when obtaining the concatenated vector feature, the speech hidden feature of the target enhanced speech information is obtained from the buffer.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the obtaining multi-path audio data in an environmental space comprises:
obtaining a microphone array corresponding to the environmental space in which the terminal is located, the microphone array including a plurality of microphones and an array structure of the plurality of microphones;
acquiring an audio signal in the environmental space based on the array structure of the plurality of the microphones, the audio signal including at least one speech signal; and
separately determining the at least one speech signal acquired by the each of the microphones as one-path audio data corresponding to the each of the microphones, the one-path audio data being the at least one speech signal acquired by one microphone.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the one-path audio data corresponding to the each of the microphones comprises a first speech signal and a second speech signal, the first speech signal being a sound signal that is transmitted by a user and acquired by the microphone array, the second speech signal being a sound signal that is transmitted by the terminal and acquired by the microphone array; and
the obtaining a speech data set based on the multi-path audio data comprises:
obtaining a target microphone from the microphones of the microphone array, and using audio data that comprises the first speech signal and the second speech signal and that corresponds to the target microphone as target audio data;
reducing the second speech signal in the target audio data by using an echo canceler, and determining the target audio data from which the second speech signal is reduced as to-be-enhanced speech data;
obtaining to-be-enhanced speech data separately corresponding to each path of audio data, in response to determining the each of the microphones in the microphone array is the target microphone; and
separately adding each piece of to-be-enhanced speech data to the speech data set.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the separately generating, in a plurality of enhancement directions, a plurality of pieces of enhanced speech information corresponding to the speech data set comprises:
selecting one of the plurality of enhancement directions of a beamformer as a target enhancement direction, enhancing the first speech signal in the speech data set based on the beamformer, and using the enhanced first speech signal as directional enhanced data in the target enhancement direction;
filtering out environmental noise carried in the directional enhanced data based on a noise canceler and a reverb canceler, and determining the directional enhanced data from which the environmental noise is filtered out as the enhanced speech information corresponding to the speech data set; and
in response to determining each of the plurality of enhancement directions is selected as the target enhancement direction, obtaining the enhanced speech information of the speech data set in the enhancement directions.

### Claim 6 (depends on 5)

The method according to claim 5, wherein the first speech signal in the speech data set comprises a sound sub-signal transmitted by a first user and a sound sub-signal transmitted by a second user, the first user being a user in the target enhancement direction, and the second user being a user in one of the plurality of enhancement directions except the target enhancement direction; and
the enhancing the first speech signal in the speech data set based on the beamformer, and using the enhanced first speech signal as directional enhanced data in the target enhancement direction comprises:
enhancing, based on the beamformer, the sound sub-signal transmitted by the first user in the speech data set, and inhibiting, in the target enhancement direction, interference data generated by the sound sub-signal transmitted by the second user, to output the enhanced first speech signal; and
using the enhanced first speech signal as the directional enhanced data in the target enhancement direction.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the matching a speech hidden feature in the enhanced speech information with a target matching word, and determining an enhancement direction corresponding to target enhanced speech information having a highest degree of matching with the target matching word among the plurality of pieces of enhancement speech information as a target audio direction comprises:
obtaining a speech hidden feature in the each piece of enhanced speech information based on the first wake-up detection model, a speech hidden feature being a feature obtained after feature extraction is performed by the first wake-up detection model on a speech spectrum feature of one piece of enhanced speech information;
performing speech recognition on each speech hidden feature based on the target matching word, to obtain a speech recognition result corresponding to the first wake-up detection model, the speech recognition result comprising the degree of matching between the speech hidden feature corresponding to the each enhancement direction and the target matching word; and
determining, based on the speech recognition result, the enhancement direction corresponding to the enhanced speech information having the highest degree of matching with the target matching word as the target audio direction.

### Claim 8 (depends on 7)

The method according to claim 7, wherein the performing speech recognition on each speech hidden feature based on the target matching word, to obtain a speech recognition result corresponding to the first wake-up detection model comprises:
obtaining, based on the first wake-up detection model, a degree of matching between the each speech hidden feature and a plurality of wake-up features in the first wake-up detection model; and
associating the degree of matching obtained by the first wake-up detection model with the target matching word corresponding to the plurality of wake-up features in the first wake-up detection model, to obtain the speech recognition result corresponding to the first wake-up detection model.

### Claim 9 (depends on 1)

The method according to claim 1, further comprising:
generating a target authentication result according to the degree of matching between the concatenated vector feature and the target wake-up feature outputted by the second wake-up detection model; and
waking up the terminal in response to determining the degree of matching in the target authentication result is greater than or equal to a matching threshold corresponding to the target matching word.

### Claim 10 (depends on 9)

The method according to claim 9, further comprising:
in response to determining the degree of matching in the target authentication result is less than the matching threshold corresponding to the target matching word, performing the operation of obtaining multi-path audio data in an environmental space.

### Claim 11

An audio data processing apparatus, comprising: at least one memory storing computer program instructions; and at least one processor coupled to the at least one memory and configured to executing the computer program instructions and perform:
obtaining multi-path audio data in an environmental space;
obtaining a speech data set based on the multi-path audio data;
separately generating, in a plurality of enhancement directions, a plurality of pieces of enhanced speech information corresponding to the speech data set;
matching, by using a first wake-up detection model, a speech hidden feature in each piece of enhanced speech information with a target matching word to obtain a degree of matching with the target matching word, and determining an enhancement direction corresponding to target enhanced speech information having a highest degree of matching with the target matching word among the plurality of pieces of enhancement speech information as a target audio direction;
obtaining, by using a second wake-up detection model, a speech spectrum feature in the target enhanced speech information in the target audio direction; and
determining a probability of existence of the target matching word in the target audio direction for controlling a terminal, comprising:
concatenating the speech hidden feature and the speech spectrum feature of the target enhanced speech information in the target audio direction, to obtain a concatenated vector feature; and
determining, by using the second wake-up detection model, a degree of matching between the concatenated vector feature and a target wake-up feature, to generate the probability of existence of the target matching word.

### Claim 12 (depends on 11)

The apparatus according to claim 11, wherein the at least one processor is further configured to perform:
storing, in a buffer of the second wake-up detection model, the speech hidden feature of the target enhanced speech information output by the first wake-up detection model,
wherein when obtaining the concatenated vector feature, the speech hidden feature of the target enhanced speech information is obtained from the buffer.

### Claim 13 (depends on 11)

The apparatus according to claim 11, wherein the at least one processor is further configured to perform:
obtaining a microphone array corresponding to the environmental space in which the terminal is located, the microphone array including a plurality of microphones, and an array structures of the plurality of the microphones;
acquiring an audio signal in the environmental space based on the array structure of the plurality of the microphones, the audio signal including at least one speech signal; and
separately determining the at least one speech signal acquired by the each of the microphones as one-path audio data corresponding to the each of the microphones, the one-path audio data being the at least one speech signal acquired by one microphone.

### Claim 14 (depends on 13)

The apparatus according to claim 13, wherein the one-path audio data corresponding to the each of the microphones comprises a first speech signal and a second speech signal, the first speech signal being a sound signal that is transmitted by a user and acquired by the microphone array, the second speech signal being a sound signal that is transmitted by the terminal and acquired by the microphone array; and
the at least one processor is further configured to perform:
obtaining a target microphone from the microphones of the microphone array, and use audio data that comprises the first speech signal and the second speech signal and that corresponds to the target microphone as target audio data;
reducing the second speech signal in the target audio data by using an echo canceler, and determine the target audio data from which the second speech signal is reduced as to-be-enhanced speech data;
obtaining to-be-enhanced speech data separately corresponding to each path of audio data, in response to determining the each of the microphones in the microphone array is the target microphone; and
separately adding each piece of to-be-enhanced speech data to the speech data set.

### Claim 15 (depends on 14)

The apparatus according to claim 14, wherein the at least one processor is further configured to perform:
selecting one of the plurality of enhancement directions of a beamformer as a target enhancement direction, enhance the first speech signal in the speech data set based on the beamformer, and use the enhanced first speech signal as directional enhanced data in the target enhancement direction;
filtering out environmental noise carried in the directional enhanced data based on a noise canceler and a reverb canceler, and determine the directional enhanced data from which the environmental noise is filtered out as the enhanced speech information corresponding to the speech data set; and
in response to determining each of the plurality of enhancement directions is selected as the target enhancement direction, obtaining the enhanced speech information of the speech data set in the enhancement directions.

### Claim 16 (depends on 15)

The apparatus according to claim 15, wherein the first speech signal in the speech data set comprises a sound sub-signal transmitted by a first user and a sound sub-signal transmitted by a second user, the first user being a user in the target enhancement direction, and the second user being a user in one of the plurality of enhancement directions except the target enhancement direction; and the at least one processor is further configured to perform:
enhancing, based on the beamformer, the sound sub-signal transmitted by the first user in the first speech signal, and inhibit, in the target enhancement direction, interference data generated by the sound sub-signal transmitted by the second user, to output the enhanced first speech signal; and
using the enhanced first speech signal as the directional enhanced data in the target enhancement direction.

### Claim 17 (depends on 11)

The apparatus according to claim 11, wherein the at least one processor is further configured to perform:
obtaining a speech hidden feature in the each piece of enhanced speech information based on the first wake-up detection model, a speech hidden feature being a feature obtained after feature extraction is performed by the first wake-up detection model on a speech spectrum feature of one piece of enhanced speech information;
performing speech recognition on each speech hidden feature based on the target matching word, to obtain a speech recognition result corresponding to the first wake-up detection model, the speech recognition result comprising the degree of matching between the speech hidden feature corresponding to the each enhancement direction and the target matching word; and
determining, based on the speech recognition result, the enhancement direction corresponding to the enhanced speech information having the highest degree of matching with the target matching word as the target audio direction.

### Claim 18 (depends on 17)

The apparatus according to claim 17, wherein the at least one processor is further configured to perform:
obtaining, based on the first wake-up detection model, a degree of matching between the each speech hidden feature and a plurality of wake-up features in the first wake-up detection model; and
associating the degree of matching obtained by the first wake-up detection model with the target matching word corresponding to the plurality of wake-up features in the first wake-up detection model, to obtain the speech recognition result corresponding to the first wake-up detection model.

### Claim 19 (depends on 11)

The apparatus according to claim 11, wherein the at least one processor is further configured to perform:
generating a target authentication result according to the degree of matching between the concatenated vector feature and the target wake-up feature outputted by the second wake-up detection model;
waking up the terminal in response to determining the degree of matching in the target authentication result is greater than or equal to a matching threshold corresponding to the target matching word; and
in response to determining the degree of matching in the target authentication result is less than the matching threshold corresponding to the target matching word, performing the operation of obtaining multi-path audio data in an environmental space.

### Claim 20

A non-transitory electronic-readable storage medium storing computer program instructions executable by at least one processor to perform:
obtaining multi-path audio data in an environmental space;
obtaining a speech data set based on the multi-path audio data;
separately generating, in a plurality of enhancement directions, a plurality of pieces of enhanced speech information corresponding to the speech data set;
matching, by using a first wake-up detection model, a speech hidden feature in each piece of enhanced speech information with a target matching word to obtain a degree of matching with the target matching word, and determining an enhancement direction corresponding to target enhanced speech information having a highest degree of matching with the target matching word among the plurality of pieces of enhancement speech information as a target audio direction;
obtaining, by using a second wake-up detection model, a speech spectrum feature in the target enhanced speech information in the target audio direction; and
determining a probability of existence of the target matching word in the target audio direction for controlling a terminal, comprising:
concatenating the speech hidden feature and the speech spectrum feature of the target enhanced speech information in the target audio direction, to obtain a concatenated vector feature; and
determining, by using the second wake-up detection model, a degree of matching between the concatenated vector feature and a target wake-up feature, to generate the probability of existence of the target matching word.