# US 2021/0174791 A1 · Systems and methods for processing meeting information obtained from multiple sources

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174791 A1
- Document ID: US20210174791A1
- Status: Published application
- Kind: A1
- Application number: 17/177,186
- Filed: 2021-02-16
- Published: 2021-06-10
- Earliest claimed: 2018-05-02
- Assignees: Melo Inc.
- Applicants: Melo Inc.
- Inventors: Guobin Shen; Zheng Han
- Web page: https://patentgrep.com/patent/US20210174791A1

## CPC

- G10L 15/20: Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech (G10L21/02 takes precedence)
- H04L 12/1827: {Network arrangements for conference optimisation or adaptation}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G10L 15/08: Speech classification or search
- G10L 15/16: using artificial neural networks
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 21/0216: characterised by the method used for estimating noise
- G10L 25/06: the extracted parameters being correlation coefficients
- G10L 25/51: for comparison or discrimination
- G10L 25/84: for discriminating voice from noise
- G06V 40/161: {Detection; Localisation; Normalisation}
- G10L 17/00: Speaker identification or verification techniques
- G10L 25/30: using neural networks
- G10L 2015/088: {Word spotting}
- G10L 2021/02166: {Microphone arrays; Beamforming}
- H04L 12/1831: {Tracking arrangements for later retrieval, e.g. recording contents, participants activities or behavior, network status}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods are provided for processing audio information. An exemplary system may include a communication interface configured to receive a plurality of audio streams obtained by multiple terminal devices. The system may also include a memory and a processor. The processor may execute instructions stored on the memory to perform operations. The operations may include beamforming the plurality of audio streams based on a spectral mask indicating signal and noise presence probabilities. The operations may also include synchronizing the beamformed audio streams and determining signal-to-noise-ratio (SNR) indicators associated with the synchronized audio streams. The operations may further include selecting a candidate audio stream based on the SNR indicators and generating a synthesis audio stream including at least a portion of the candidate audio stream.

## Claims

### Claim 1

A system for processing audio information, the system comprising:
a communication interface configured to receive a plurality of audio streams obtained by multiple terminal devices, wherein each of the multiple terminal devices obtains one or more of the plurality of audio streams;
a memory storing computer-executable instructions; and
a processor in communication with the communication interface and the memory, the processor being configured to execute the computer-executable instructions to perform operations, wherein the operations comprise:
beamforming the plurality of audio streams based on a spectral mask indicating signal and noise presence probabilities;
synchronizing the beamformed audio streams;
determining signal-to-noise-ratio (SNR) indicators associated with the synchronized audio streams;
selecting, from the synchronized audio streams, a candidate audio stream based on the SNR indicators, wherein the SNR indicator associated with the candidate audio stream indicates that the candidate audio stream has a higher average SNR than that of a predetermined number of other audio streams; and
generating a synthesis audio stream including at least a portion of the candidate audio stream.

### Claim 2 (depends on 1)

The system of claim 1, wherein the operations comprise:
applying a neural network to the plurality of audio streams received by the communication interface to generate the spectral mask.

### Claim 3 (depends on 2)

The system of claim 2, wherein the neural network is trained using data generated by the multiple terminal devices under a predetermined condition.

### Claim 4 (depends on 2)

The system of claim 2, wherein the operations comprise:
detecting, from the plurality of audio streams received by the communication interface, an audio sample having an SNR higher than a predetermined threshold; and
extracting the audio sample for retraining the neural network.

### Claim 5 (depends on 4)

The system of claim 4, wherein the neural network is retrained by minimizing a difference between the audio sample and a corresponding audio segment in the synthesis audio stream.

### Claim 6 (depends on 1)

The system of claim 1, wherein beamforming the plurality of audio streams based on the spectral mask comprises:
computing spatial covariance matrices for signal and noise; and
determining beamforming coefficients based on the spatial covariance matrices.

### Claim 7 (depends on 1)

The system of claim 1, wherein:
the plurality of audio streams received by the communication interface comprise time stamps; and
synchronizing the beamformed audio streams comprises synchronizing the beamformed audio streams based on the time stamps.

### Claim 8 (depends on 1)

The system of claim 1, wherein synchronizing the beamformed audio streams comprises:
determining cross-correlation values among the beamformed audio streams; and
synchronizing the beamformed audio streams based on a peak of the cross-correlation values.

### Claim 9 (depends on 1)

The system of claim 1, wherein the operations comprise:
performing wakeup word detection based on the synthesis audio stream.

### Claim 10

A method for processing audio information, comprising:
receiving, by a communication interface, a plurality of audio streams obtained by multiple terminal devices, wherein each of the multiple terminal devices obtains one or more of the plurality of audio streams;
beamforming the plurality of audio streams based on a spectral mask indicating signal and noise presence probabilities;
synchronizing the beamformed audio streams;
determining signal-to-noise-ratio (SNR) indicators associated with the synchronized audio streams;
selecting, from the synchronized audio streams, a candidate audio stream based on the SNR indicators, wherein the SNR indicator associated with the candidate audio stream indicates that the candidate audio stream has a higher average SNR than that of a predetermined number of other audio streams; and
generating a synthesis audio stream including at least a portion of the candidate audio stream.

### Claim 11 (depends on 10)

The method of claim 10, comprising:
applying a neural network to the plurality of audio streams received by the communication interface to generate the spectral mask.

### Claim 12 (depends on 11)

The method of claim 11, wherein the neural network is trained using data generated by the multiple terminal devices under a predetermined condition.

### Claim 13 (depends on 11)

The method of claim 11, comprising:
detecting, from the plurality of audio streams received by the communication interface, an audio sample having an SNR higher than a predetermined threshold; and
extracting the audio sample for retraining the neural network.

### Claim 14 (depends on 13)

The method of claim 13, wherein the neural network is retrained by minimizing a difference between the audio sample and a corresponding audio segment in the synthesis audio stream.

### Claim 15 (depends on 10)

The method of claim 10, wherein beamforming the plurality of audio streams based on the spectral mask comprises:
computing spatial covariance matrices for signal and noise; and
determining beamforming coefficients based on the spatial covariance matrices.

### Claim 16 (depends on 10)

The method of claim 10, wherein:
the plurality of audio streams received by the communication interface comprise time stamps; and
synchronizing the beamformed audio streams comprises synchronizing the beamformed audio streams based on the time stamps.

### Claim 17 (depends on 10)

The method of claim 10, wherein synchronizing the beamformed audio streams comprises:
determining cross-correlation values among the beamformed audio streams; and
synchronizing the beamformed audio streams based on a peak of the cross-correlation values.

### Claim 18 (depends on 10)

The method of claim 10, comprising:
performing wakeup word detection based on the synthesis audio stream.

### Claim 19

A non-transitory computer-readable medium storing instructions that are executable by at least one processor to cause performance of a method for processing audio information, the method comprising:
receiving, by a communication interface, a plurality of audio streams obtained by multiple terminal devices, wherein each of the multiple terminal devices obtains one or more of the plurality of audio streams;
beamforming the plurality of audio streams based on a spectral mask indicating signal and noise presence probabilities;
synchronizing the beamformed audio streams;
determining signal-to-noise-ratio (SNR) indicators associated with the synchronized audio streams;
selecting, from the synchronized audio streams, a candidate audio stream based on the SNR indicators, wherein the SNR indicator associated with the candidate audio stream indicates that the candidate audio stream has a higher average SNR than that of a predetermined number of other audio streams; and
generating a synthesis audio stream including at least a portion of the candidate audio stream.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein beamforming the plurality of audio streams based on the spectral mask comprises:
computing spatial covariance matrices for signal and noise; and
determining beamforming coefficients based on the spatial covariance matrices.
