# US 2021/0174792 A1 · Audio recognition method, method, apparatus for positioning target audio, and device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174792 A1
- Document ID: US20210174792A1
- Status: Published application
- Kind: A1
- Application number: 17/183,209
- Filed: 2021-02-23
- Published: 2021-06-10
- Earliest claimed: 2018-11-30
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (shenzhen) Company Limited
- Inventors: Jimeng ZHENG; Ian Ernan LIU; Yi GAO; Weiwei LI
- Web page: https://patentgrep.com/patent/US20210174792A1

## CPC

- G10L 15/20: Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech (G10L21/02 takes precedence)
- G01S 3/8006: {Multi-channel systems specially adapted for direction-finding, i.e. having a single aerial system capable of giving simultaneous indications of the directions of different signals}
- G01S 3/802: Systems for determining direction or deviation from predetermined direction
- G10L 15/08: Speech classification or search
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 21/0224: Processing in the time domain
- G10L 21/0232: Processing in the frequency domain
- G10L 25/51: for comparison or discrimination
- G10L 2015/088: {Word spotting}
- G10L 2021/02082: {the noise being echo, reverberation of the speech}
- G10L 2021/02166: {Microphone arrays; Beamforming}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES

## Abstract

Embodiments of this application disclose method and apparatus for positioning a target audio signal by an audio interaction device, and an audio interaction device The method includes: obtaining audio signals in a plurality of directions in a space, and performing echo cancellation on the audio signal, the audio signal including a target-audio direct signal; obtaining weights of a plurality of time-frequency points in the audio signals, a weight of each time-frequency point indicating, at the time-frequency point, a relative proportion of the target-audio direct signal in the audio signals; weighting time-frequency components of the audio signal at the plurality of time-frequency points separately for each of the plurality of directions by using the weights of the plurality of time-frequency points, to obtain a weighted audio signal energy distribution; and obtaining a sound source azimuth corresponding to the target-audio direct signal in the audio signals accordingly.

## Claims

### Claim 1

An audio recognition method, comprising:
obtaining audio signals collected in a plurality of directions in a space, the audio signals comprising a target-audio direct signal;
performing echo cancellation on the audio signals;
obtaining weights of a plurality of time-frequency points in the echo-canceled audio signals, a weight of each time-frequency point indicating a relative proportion of the target-audio direct signal in the echo-canceled audio signals at the time-frequency point;
weighting time-frequency components of the audio signals at the plurality of time-frequency points separately for each of the plurality of directions by using the weights of the plurality of time-frequency points, to obtain a weighted audio signal energy distribution of the audio signals in the plurality of directions;
obtaining a sound source azimuth corresponding to the target-audio direct signal in the audio signals by using the weighted audio signal energy distribution of the audio signals in the plurality of directions; and
performing audio recognition to the audio signals based on the sound source azimuth corresponding to the target-audio direct signal.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the weighting time-frequency components of the audio signals at the plurality of time-frequency points separately for each of the plurality of directions by using the weights of the plurality of time-frequency points, to obtain a weighted audio signal energy distribution of the audio signals in the plurality of directions comprises:
obtaining a weighted covariance matrix of each of the plurality of time-frequency points based on the weights and time-frequency domain expressions of the echo-canceled audio signals, the weighted covariance matrix being used for describing the relative proportion of the target-audio direct signal in the audio signals; and
performing weighted calculation on a spatial spectrum of the audio signals by using the weighted covariance matrix, to obtain a spatial spectrum of the audio signals weighted at the plurality of time-frequency points.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the obtaining weights of a plurality of time-frequency points in the echo-canceled audio signals comprises:
obtaining time-frequency domain expressions of the audio signals;
estimating an audio feature parameter according to each time-frequency point for the time-frequency domain expressions of the audio signals; and
obtaining weights of the audio signals at the time-frequency point according to the audio feature parameter corresponding to the time-frequency point.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the audio feature parameter comprises a signal-to-noise ratio (SNR) estimate of the time-frequency domain expressions at the time-frequency point, and the estimating an audio feature parameter according to the time-frequency point for the time-frequency domain expressions of the audio signals comprises:
estimating an SNR estimate of each time-frequency point for the time-frequency domain expressions of the audio signals, the SNR estimate being used for obtaining the weights of the audio signals at the time-frequency point through mapping.

### Claim 5 (depends on 3)

The method according to claim 3, wherein there are a plurality of audio feature parameters, and the obtaining weights of the audio signals at the time-frequency point according to the audio feature parameter corresponding to the time-frequency point comprises:
obtaining weights mapped respectively by different audio feature parameters of the time-frequency domain expressions of the audio signals at the time-frequency point; and
performing weighted calculation on the obtained weights, to obtain weights of the audio signals at the time-frequency point.

### Claim 6 (depends on 5)

The method according to claim 5, wherein the audio feature parameter further comprises at least one or any combination of a direct-to-reverb ratio (DRR) and a residual echo signal estimate of the time-frequency domain expression at the time-frequency point and a probability that the time-frequency point corresponds to a target wakeup word.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the audio feature parameter comprises the probability that the time-frequency point corresponds to the target wakeup word, and the estimating an audio feature parameter according to the time-frequency point for the time-frequency domain expressions of the audio signals comprises:
spotting a target wakeup word in the audio signals, and estimating a time-frequency point at which the spotted target wakeup word is located, to obtain a probability that each time-frequency point corresponds to the target wakeup word.

### Claim 8 (depends on 2)

The method according to claim 2, wherein the obtaining a weighted covariance matrix of each of the plurality of time-frequency points comprises:
performing weighted calculation on the time-frequency domain expressions and the weights of the echo-canceled audio signals at the time-frequency point, to obtain weighted element values corresponding to the time-frequency point in the weighted covariance matrix, the weighted element values corresponding to the time-frequency point forming the weighted covariance matrix.

### Claim 9 (depends on 1)

The method according to claim 1, wherein the performing echo cancellation on the audio signals comprises:
obtaining an echo reference signal from an audio interaction device;
performing, by using the echo reference signal, linear echo cancellation on the audio signals; and
performing time-frequency analysis on the echo-canceled audio signals, to obtain time-frequency domain expressions of the echo-canceled audio signals.

### Claim 10

An audio interaction device, comprising:
a processor; and
a memory, the memory storing computer-readable instructions, the computer-readable instructions, when executed by the processor, causing the audio interaction device to perform a plurality of operations including:
obtaining audio signals collected in a plurality of directions in a space, the audio signals comprising a target-audio direct signal;
performing echo cancellation on the audio signals;
obtaining weights of a plurality of time-frequency points in the echo-canceled audio signals, a weight of each time-frequency point indicating a relative proportion of the target-audio direct signal in the echo-canceled audio signals at the time-frequency point;
weighting time-frequency components of the audio signals at the plurality of time-frequency points separately for each of the plurality of directions by using the weights of the plurality of time-frequency points, to obtain a weighted audio signal energy distribution of the audio signals in the plurality of directions;
obtaining a sound source azimuth corresponding to the target-audio direct signal in the audio signals by using the weighted audio signal energy distribution of the audio signals in the plurality of directions; and
performing audio recognition to the audio signals based on the sound source azimuth corresponding to the target-audio direct signal.

### Claim 11 (depends on 10)

The audio interaction device according to claim 10, wherein the weighting time-frequency components of the audio signals at the plurality of time-frequency points separately for each of the plurality of directions by using the weights of the plurality of time-frequency points, to obtain a weighted audio signal energy distribution of the audio signals in the plurality of directions comprises:
obtaining a weighted covariance matrix of each of the plurality of time-frequency points based on the weights and time-frequency domain expressions of the echo-canceled audio signals, the weighted covariance matrix being used for describing the relative proportion of the target-audio direct signal in the audio signals; and
performing weighted calculation on a spatial spectrum of the audio signals by using the weighted covariance matrix, to obtain a spatial spectrum of the audio signals weighted at the plurality of time-frequency points.

### Claim 12 (depends on 10)

The audio interaction device according to claim 10, wherein the obtaining weights of a plurality of time-frequency points in the echo-canceled audio signals comprises:
obtaining time-frequency domain expressions of the audio signals;
estimating an audio feature parameter according to each time-frequency point for the time-frequency domain expressions of the audio signals; and
obtaining weights of the audio signals at the time-frequency point according to the audio feature parameter corresponding to the time-frequency point.

### Claim 13 (depends on 12)

The audio interaction device according to claim 12, wherein the audio feature parameter comprises a signal-to-noise ratio (SNR) estimate of the time-frequency domain expressions at the time-frequency point, and the estimating an audio feature parameter according to the time-frequency point for the time-frequency domain expressions of the audio signals comprises:
estimating an SNR estimate of each time-frequency point for the time-frequency domain expressions of the audio signals, the SNR estimate being used for obtaining the weights of the audio signals at the time-frequency point through mapping.

### Claim 14 (depends on 12)

The audio interaction device according to claim 12, wherein there are a plurality of audio feature parameters, and the obtaining weights of the audio signals at the time-frequency point according to the audio feature parameter corresponding to the time-frequency point comprises:
obtaining weights mapped respectively by different audio feature parameters of the time-frequency domain expressions of the audio signals at the time-frequency point; and
performing weighted calculation on the obtained weights, to obtain weights of the audio signals at the time-frequency point.

### Claim 15 (depends on 14)

The audio interaction device according to claim 14, wherein the audio feature parameter further comprises at least one or any combination of a direct-to-reverb ratio (DRR) and a residual echo signal estimate of the time-frequency domain expression at the time-frequency point and a probability that the time-frequency point corresponds to a target wakeup word.

### Claim 16 (depends on 15)

The audio interaction device according to claim 15, wherein the audio feature parameter comprises the probability that the time-frequency point corresponds to the target wakeup word, and the estimating an audio feature parameter according to the time-frequency point for the time-frequency domain expressions of the audio signals comprises:
spotting a target wakeup word in the audio signals, and estimating a time-frequency point at which the spotted target wakeup word is located, to obtain a probability that each time-frequency point corresponds to the target wakeup word.

### Claim 17 (depends on 11)

The audio interaction device according to claim 11, wherein the obtaining a weighted covariance matrix of each of the plurality of time-frequency points comprises:
performing weighted calculation on the time-frequency domain expressions and the weights of the echo-canceled audio signals at the time-frequency point, to obtain weighted element values corresponding to the time-frequency point in the weighted covariance matrix, the weighted element values corresponding to the time-frequency point forming the weighted covariance matrix.

### Claim 18 (depends on 10)

The audio interaction device according to claim 10, wherein the performing echo cancellation on the audio signals comprises:
obtaining an echo reference signal from an audio interaction device;
performing, by using the echo reference signal, linear echo cancellation on the audio signals; and
performing time-frequency analysis on the echo-canceled audio signals, to obtain time-frequency domain expressions of the echo-canceled audio signals.

### Claim 19

A non-transitory computer readable medium, storing computer-readable instructions, the computer readable instructions being executable by one or more processors of an audio interaction device to perform a plurality of operations including:
obtaining audio signals collected in a plurality of directions in a space, the audio signals comprising a target-audio direct signal;
performing echo cancellation on the audio signals;
obtaining weights of a plurality of time-frequency points in the echo-canceled audio signals, a weight of each time-frequency point indicating a relative proportion of the target-audio direct signal in the echo-canceled audio signals at the time-frequency point;
weighting time-frequency components of the audio signals at the plurality of time-frequency points separately for each of the plurality of directions by using the weights of the plurality of time-frequency points, to obtain a weighted audio signal energy distribution of the audio signals in the plurality of directions;
obtaining a sound source azimuth corresponding to the target-audio direct signal in the audio signals by using the weighted audio signal energy distribution of the audio signals in the plurality of directions; and
performing audio recognition to the audio signals based on the sound source azimuth corresponding to the target-audio direct signal.

### Claim 20 (depends on 19)

The non-transitory computer readable medium according to claim 19, wherein the weighting time-frequency components of the audio signals at the plurality of time-frequency points separately for each of the plurality of directions by using the weights of the plurality of time-frequency points, to obtain a weighted audio signal energy distribution of the audio signals in the plurality of directions comprises:
obtaining a weighted covariance matrix of each of the plurality of time-frequency points based on the weights and time-frequency domain expressions of the echo-canceled audio signals, the weighted covariance matrix being used for describing the relative proportion of the target-audio direct signal in the audio signals; and
performing weighted calculation on a spatial spectrum of the audio signals by using the weighted covariance matrix, to obtain a spatial spectrum of the audio signals weighted at the plurality of time-frequency points.