# US 12,118,989 B2 · Speech processing method and method for generating speech processing model > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,989 B2 - Document ID: US12118989B2 - Status: Granted - Kind: B2 - Application number: 17/507,437 - Filed: 2021-10-21 - Published: 2022-02-10 - Granted: 2024-10-15 - Earliest claimed: 2020-10-27 - Assignees: Beijing Baidu Netcom Science Technology Co., Ltd. - Applicants: Beijing Baidu Netcom Science and Technology Co., Ltd. - Inventors: Xu Chen; Jinfeng Bai; Runqiang Han; Lei Jia - Web page: https://patentgrep.com/patent/US12118989B2 ## CPC - G10L 15/20: Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech (G10L21/02 takes precedence) - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G10L 15/063: {Training} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 21/0232: Processing in the frequency domain - G10L 21/038: using band spreading techniques - G10L 25/30: using neural networks - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/045: Combinations of networks - G10L 2021/02082: {the noise being echo, reverberation of the speech} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract The present disclosure provides a speech processing method, and a method for generating a speech processing model, related to a field of signal processing technologies. The speech processing method includes: obtaining M speech signals to be processed and N reference signals; performing sub-band decomposition on each of the M speech signals and each of the N reference signals to obtain frequency-band components in each speech signal and each reference signal; processing the frequency-band components in each speech signal and each reference signal by using an echo cancellation model, to obtain an ideal ratio mask corresponding to the N reference signals in each frequency band of each speech signal; and performing echo cancellation on each frequency-band component of each speech signal based on the ideal ratio mask corresponding to the N reference signals in each frequency band of each speech signal, to obtain M echo-cancelled speech signals. ## Claims ### Claim 1 A speech processing method, comprising: obtaining M speech signals to be processed and N reference signals, wherein M and N are positive integers equal to or greater than 1; performing sub-band decomposition on each of the M speech signals and each of the N reference signals to obtain frequency-band components in each speech signal and frequency-band components in each reference signal; processing the frequency-band components in each speech signal and the frequency-band components in each reference signal by using an echo cancellation model, to obtain an ideal ratio mask (IRM) corresponding to the N reference signals in each frequency band of each speech signal; and performing echo cancellation on each frequency-band component of each speech signal based on the ideal ratio mask corresponding to the N reference signals in each frequency band of each speech signal, to obtain M echo-cancelled speech signals; wherein processing the frequency-band components in each speech signal and the frequency-band components in each reference signal by using the echo cancellation model to obtain the IRM corresponding to the N reference signals in each frequency band of each speech signal comprises: performing multi-granularity feature extraction on each frequency-band component in the M speech signals and the N reference signals to obtain features of each frequency-band component in each of the M speech signals and the N reference signals; and fusing the features of each frequency-band components in each of the M speech signals and the N reference signals, to obtain the IRM corresponding to the N reference signals in each frequency band of each speech signal. ### Claim 2 (depends on 1) The speech processing method according to claim 1, wherein performing sub-band decomposition on each of the M speech signals and each of the N reference signals comprises: determining a sub-band decomposition mode based on a type of a speech processing device; and performing sub-band decomposition on each of the M speech signals and each of the N reference signals according to the sub-band decomposition mode. ### Claim 3 (depends on 1) The speech processing method according to claim 1, wherein performing sub-band decomposition on each of the M speech signals and each of the N reference signals comprises: determining a sub-band decomposition mode based on environment in which a speech processing device is located; and performing sub-band decomposition on each of the M speech signals and each of the N reference signals according to the sub-band decomposition mode. ### Claim 4 (depends on 1) The speech processing method according to claim 1, before processing the frequency-band components in each speech signal and the frequency-band components in each reference signal by using the echo cancellation model, further comprising: normalizing the frequency-band components in each speech signal and the frequency-band components in each reference signal. ### Claim 5 (depends on 1) The speech processing method according to claim 1, after obtaining the M echo-cancelled speech signals, further comprising: inputting the M echo-cancelled speech signals into a speech recognition model to obtain a recognition result corresponding to the M echo-cancelled speech signals. ### Claim 6 A method for generating a speech processing model, comprising: obtaining a training data set, wherein each group of data in the training data set comprises a reference signal, a speech signal carrying echo, and a labelled ideal ration mask (IRM); performing sub-band decomposition on the reference signal and the speech signal carrying echo in each group to obtain frequency-band components of the reference signal and the speech signal carrying echo in each group; processing the frequency-band components of the reference signal and the speech signal carrying echo in each group by using an initial neural network model, to obtain a predicted IRM corresponding to the reference signal in each group in each frequency-band component of the speech signal carrying echo; and performing back gradient propagation correction on the initial neural network model based on a difference between the predicted IRM of each group and the corresponding labelled IRM, to obtain a trained speech processing model; wherein processing the frequency-band components of the reference signal and the speech signal carrying echo in each group by using an initial neural network model, to obtain a predicted IRM corresponding to the reference signal in each group in each frequency-band component of the speech signal carrying echo comprises: performing multi-granularity feature extraction on each frequency-band component of the reference signal and the speech signal carrying echo in each group, to obtain features of each frequency-band component in each of the reference signal and the speech signal; and fusing the features of each frequency-band component in each of the reference signal and the speech signal, to obtain the predicted IRM corresponding to the reference signal in each group in each frequency-band component of the speech signal carrying echo. ### Claim 7 (depends on 6) The method according to claim 6, wherein obtaining the training data set comprises: obtaining a plurality of data groups, wherein each data group comprises a standard speech signal, a reference signal and an echo signal corresponding to the reference signal; fusing the echo signal and the standard speech signal in each data group to generate a speech signal carrying echo for each group; performing sub-band decomposition on the standard speech signal and the speech signal carrying echo in each data group to obtain frequency-band components in the standard speech signal and the speech signal carrying echo in each data group; and determining a labelled IRM corresponding to the speech signal carrying echo in each data group based on the frequency-band components in the standard speech signal and the speech signal carrying echo in a corresponding data group. ### Claim 8 (depends on 7) The method according to claim 7, wherein fusing the echo signal and the standard speech signal in each data group to obtain the speech signal carrying echo for each data group comprises: fusing the echo signal and the standard speech signal in each data group based on different signal-to-noise rations to generate a plurality of groups of speech signals carrying echo. ### Claim 9 (depends on 7) The method according to claim 7, wherein determining the labelled IRM corresponding to the speech signal carrying echo in each data group based on the frequency-band components in the standard speech signal and the speech signal carrying echo in the corresponding data group comprises: performing amplitude calculation on each frequency-band component of the standard speech signal and the speech signal carrying echo in each data group to obtain an amplitude of each frequency-band component; determining the labelled IRM corresponding to the speech signal carrying echo in each data group based on a ratio of an amplitude of each frequency-band component in the standard speech signal to an amplitude of each frequency-band component in the speech signal carrying echo in the corresponding data group. ### Claim 10 (depends on 6) The method according to claim 6, before processing the frequency-band components of the reference signal and the speech signal carrying echo in each group by using the initial neural network model, further comprising: normalizing the frequency-band components of the standard speech signal and the speech signal carrying echo in each data group. ### Claim 11 An electronic device, comprising: at least one processor; and a memory communicatively coupled with the at least one processor; wherein, the memory stores instructions executable by the at least one processor that, when executed by the at least one processor, cause the at least one processor to perform a speech processing method, the method comprising: obtaining M speech signals to be processed and N reference signals, wherein M and N are positive integers equal to or greater than 1; performing sub-band decomposition on each of the M speech signals and each of the N reference signals to obtain frequency-band components in each speech signal and frequency-band components in each reference signal; processing the frequency-band components in each speech signal and the frequency-band components in each reference signal by using an echo cancellation model, to obtain an ideal ratio mask (IRM) corresponding to the N reference signals in each frequency band of each speech signal; and performing echo cancellation on each frequency-band component of each speech signal based on the ideal ratio mask corresponding to the N reference signals in each frequency band of each speech signal, to obtain M echo-cancelled speech signals; wherein processing the frequency-band components in each speech signal and the frequency-band components in each reference signal by using the echo cancellation model to obtain the IRM corresponding to the N reference signals in each frequency band of each speech signal comprises: performing multi-granularity feature extraction on each frequency-band component in the M speech signals and the N reference signals to obtain features of each frequency-band component in each of the M speech signals and the N reference signals; and fusing the features of each frequency-band components in each of the M speech signals and the N reference signals, to obtain the IRM corresponding to the N reference signals in each frequency band of each speech signal. ### Claim 12 (depends on 11) The electronic device according to claim 11, wherein performing sub-band decomposition on each of the M speech signals and each of the N reference signals comprises: determining a sub-band decomposition mode based on a type of a speech processing device; and performing sub-band decomposition on each of the M speech signals and each of the N reference signals according to the sub-band decomposition mode. ### Claim 13 (depends on 11) The electronic device according to claim 11, wherein performing sub-band decomposition on each of the M speech signals and each of the N reference signals comprises: determining a sub-band decomposition mode based on environment in which a speech processing device is located; and performing sub-band decomposition on each of the M speech signals and each of the N reference signals according to the sub-band decomposition mode. ### Claim 14 (depends on 11) The electronic device according to claim 11, wherein, before processing the frequency-band components in each speech signal and the frequency-band components in each reference signal by using the echo cancellation model, the method further comprises: normalizing the frequency-band components in each speech signal and the frequency-band components in each reference signal. ### Claim 15 (depends on 11) The electronic device according to claim 11, wherein, after obtaining the M echo-cancelled speech signals, the method further comprises: inputting the M echo-cancelled speech signals into a speech recognition model to obtain a recognition result corresponding to the M echo-cancelled speech signals. ### Claim 16 (depends on 1) The method of claim 1, wherein performing multi-granularity feature extraction on each frequency-band component in the M speech signals and the N reference signals to obtain features of each frequency-band component in each of the M speech signals and the N reference signals comprises: performing causal convolution processing on each frequency-band component to achieve feature extraction of each frequency-band component using the echo cancellation model. ### Claim 17 (depends on 1) The method of claim 1, wherein fusing the features of each frequency-band components in each of the M speech signals and the N reference signals, to obtain the IRM corresponding to the N reference signals in each frequency band of each speech signal comprises: sending the features into a stacked LSTM network to model dynamic characteristics of a speech over time, wherein an output of the LSTM network constitutes learned high-level features; sending the high-level features to a mask layer to obtain the IRM corresponding to the N reference signals in each frequency band of each speech signal.