# US 11,308,971 B2 · Intelligent noise cancellation system for video conference calls in telepresence rooms > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,971 B2 - Document ID: US11308971B2 - Status: Granted - Kind: B2 - Application number: 16/929,318 - Filed: 2020-07-15 - Published: 2022-01-20 - Granted: 2022-04-19 - Earliest claimed: 2020-07-15 - Assignees: Bank of America Corporation - Applicants: Bank of America Corporation - Inventors: Sandhya Vanapalli; Santosh Govindram Bhardwaj; Jitu Nayak; Satish Radheshyam Pandey - Web page: https://patentgrep.com/patent/US11308971B2 ## CPC - G10L 21/0208: Noise filtering - H04N 7/147: {Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q)} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G10K 11/17823: {Reference signals, e.g. ambient acoustic environment} - G10L 21/0272: Voice signal separating - G10L 21/14: by displaying frequency domain information - H04N 7/15: Conference systems - H04R 1/1083: {Reduction of ambient noise (active noise reduction per se G10K11/175; protective devices for the ear, e.g. providing acoustic protection A61F11/06)} - G06N 3/08: Learning methods - G10K 2210/1081: Earphones, e.g. for telephones, ear protectors or headsets - G10L 15/16: using artificial neural networks - G10L 21/0216: characterised by the method used for estimating noise - G10L 25/30: using neural networks - G10L 25/84: for discriminating voice from noise - G10L 2015/088: {Word spotting} - H04R 2460/01: Hearing devices using active noise cancellation - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G10K: SOUND-PRODUCING DEVICES; METHODS OR DEVICES FOR PROTECTING AGAINST, OR FOR DAMPING, NOISE OR OTHER ACOUSTIC WAVES IN GENERAL; ACOUSTICS NOT OTHERWISE PROVIDED FOR - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) ## Abstract An intelligent noise cancellation process for audio or video conference calls. Different levels of deep learning model classifiers are leveraged to determine, in real-time, the presence of noise data and voice data in audio input signals being received at numerous audio input devices. In response, appropriate action is taken to prevent the noise data from being included in the subsequent audio communication. Specifically, a lightweight neural network-based model classifier is initially used to identify noise data and/or the presence of predetermined trigger words or phrases in audio input signals. In the event that the lightweight model is unable to identify the presence of the triggering words/phrases, a heavyweight neural network-based model classifier is called upon, whereby the audio signals are attempted to be converted to a human-understandable language format (i.e., a text format) as a means of positively identifying voice data in audio input signals. ## Claims ### Claim 1 An apparatus for intelligent noise cancellation for audio and video conference calls, the system comprising: a computing platform including a memory and at least one processing device stored in the memory; a plurality of audio input devices, each audio input device configured to receive an audio input signal; a first neural network model classifier stored in the memory, executable by the processing device and configured to analyze information related to one or more of the audio input signals to identify noise data and a presence of one or more predetermined words or phrases in the one or more of the audio input signals; a second neural network model classifier stored in the memory, executable by the processing device and configured to, in response to the first neural network model classifier failing to identify the presence of one or more of the predetermined words and phrases in at least one of the one or more audio input signals, attempt to convert the at least one of the audio input signals to human-comprehendible text format; an output controller stored in the memory, executable by the processor and configured to determine whether to pass or block each audio input signal based on results from the first and second neural network model classifiers; and an output device in communication with the output controller and configured to output the input audio signals that are passed by the output controller. ### Claim 2 (depends on 1) The apparatus of claim 1, wherein the output controller is further configured to (i) pass audio input signals that comprise only voice data or have a higher ratio voice data than noise data, and (ii) block audio signals that comprise only noise data or have a higher ratio noise data than voice data. ### Claim 3 (depends on 1) The apparatus of claim 1, wherein the output controller is further configured to (i) check for a mute state of each audio input device having an audio input signal that is determined to pass, and (ii) in response to determining that one or more of the audio input devices are in a muted state, generate and communicate an alert signal configured to notify a user that the audio input device requires unmuting. ### Claim 4 (depends on 3) The apparatus of claim 3, wherein the alert signal further comprises a visual signal that is displayed on a display proximate to the audio input device. ### Claim 5 (depends on 3) The apparatus of claim 3, wherein the alert signal further comprises an audio signal that is wireless communicated to headphones or earphones worn by the user. ### Claim 6 (depends on 1) The apparatus of claim 1, wherein the output controller is further configured to generate and communicate (i) an increase volume signal to each audio input device having an audio input signal comprising only voice data, and (ii) a decrease volume signal to each audio input device having an audio input signal comprising only noise data. ### Claim 7 (depends on 1) The apparatus of claim 1, further comprising an activation controller stored in the memory, executable by the processing device, in communication with each of the audio input device, and configured to (i) receive an audio input signal in analog format from each of the audio input devices, (ii) convert the audio input signal to digital format, (ii) calculate the decibel value for each of the audio input signals, and (iv) block audio input signals from processing by the first and second neural network model classifier in response to determining that the decibel value for a corresponding audio input signal is less than a predetermined decibel threshold value. ### Claim 8 (depends on 7) The apparatus of claim 7, further comprising a spectrogram converter stored in the memory, executable by the processing device, in communication with the activation controller and the first neural network model classifier and configured to (i) receive, from the activation controller, audio input signals having a decibel value greater than the predetermined decibel threshold value, and (ii) convert the audio input signal from the digital format to a spectrogram that comprises a visual representation of a spectrum of frequencies of the audio input signal as the audio input signal varies over time. ### Claim 9 A computer-implemented method for intelligent noise cancellation for audio and video conference calls, the method executed by one or more processing devices and comprising: receiving a plurality of audio input signals, each audio input signal being received form a corresponding one of a plurality of audio input devices; analyzing, by a first neural network model classifier, information related to one or more of the audio input signals to identify noise data and a presence of one or more predetermined words or phrases in the one or more of the audio input signals; in response to the first neural network model classifier failing to identify the presence of one or more of the predetermined words and phrases in at least one of the one or more of the audio input signals, attempting to convert, by a second neural network model classifier, the at least one of the audio input signals to human-comprehendible text format; determining whether to pass or block each audio input signal based on results from the first and second neural network model classifiers; and outputting the input audio signals that are determined to be passed. ### Claim 10 (depends on 9) The computer-implemented method of claim 9, wherein determining further comprises determining to (i) pass audio input signals that comprise only voice data or have a higher ratio voice data than noise data, and (ii) block audio signals that comprise only noise data or have a higher ratio noise data than voice data. ### Claim 11 (depends on 9) The computer-implemented method of claim 9, further comprising: checking for a mute state of each audio input device having an audio input signal that is determined to pass; and in response to determining that one or more of the audio input devices are in a muted state, generating and communicating an alert signal configured to notify a user that the audio input device requires unmuting. ### Claim 12 (depends on 9) The computer-implemented method of claim 9, further comprising: generating and communicating (i) an increase volume signal to each audio input device having an audio input signal comprising only voice data, and (ii) a decrease volume signal to each audio input device having an audio input signal comprising only noise data. ### Claim 13 (depends on 9) The computer-implemented method of claim 9, further comprising: receiving an audio input signal in analog format from each of the audio input devices; converting the audio input signal from analog format to digital format; calculating the decibel value for each of the audio input signals; and blocking audio input signals from processing by the first and second neural network model classifier in response to determining that the decibel value for a corresponding audio input signal is less than a predetermined decibel threshold value. ### Claim 14 (depends on 13) The computer-implemented method of claim 13, further comprising: receiving audio input signals having a decibel value greater than the predetermined decibel threshold value; and converting the audio input signal from the digital format to a spectrogram that comprises a visual representation of a spectrum of frequencies of the audio input signal as the audio input signal varies over time. ### Claim 15 A computer program product including non-transitory computer-readable medium that comprises: a first set of codes configured to cause a computer processor to receive a plurality of audio input signal, each audio input signal being received form a corresponding one of a plurality of audio input devices; a second set of codes for causing a computer processor to analyze, by a first neural network model classifier, information related to one or more of the audio input signals to identify noise data and a presence of one or more predetermined words or phrases in the one or more of the audio input signals; a third set of codes for causing a computer processor to, in response to the first neural network model classifier failing to identify the presence of one or more of the predetermined words and phrases in at least one of the one or more of the audio input signals, attempt to convert, by a second neural network model classifier, the at least one of the audio input signals to human-comprehendible text format; a fourth set of codes for causing computer processor to determine whether to pass or block each audio input signal based on results from the first and second neural network model classifiers; and a fifth set of codes for causing a computer processing device to output the input audio signals that are determined to be passed. ### Claim 16 (depends on 15) The computer program product of claim 15, wherein the fourth set of codes are further configured to cause the computer processor to determine to (i) pass audio input signals that comprise only voice data or have a higher ratio voice data than noise data, and (ii) block audio signals that comprise only noise data or have a higher ratio noise data than voice data. ### Claim 17 (depends on 15) The computer program product of claim 15, wherein the computer-readable medium further comprises a sixth set of codes for causing a computer processor to (i) check for a mute state of each audio input device having an audio input signal that is determined to pass, and (ii) in response to determining that one or more of the audio input devices are in a muted state, generate and communicate an alert signal configured to notify a user that the audio input device requires unmuting. ### Claim 18 (depends on 15) The computer program product of claim 15, wherein the computer-readable medium further comprises a sixth set of codes for causing a computer processor to generate and communicate (i) an increase volume signal to each audio input device having an audio input signal comprising only voice data, and (ii) a decrease volume signal to each audio input device having an audio input signal comprising only noise data. ### Claim 19 (depends on 18) The computer program product of claim 18, wherein the computer-readable medium further comprises a sixth set of codes for causing a computer processor to (i) receive an audio input signal in analog format from each of the audio input devices, (ii) convert the audio input signal from analog format to digital format, (iii) calculate the decibel value for each of the audio input signals, and (iv) block audio input signals from processing by the first and second neural network model classifier in response to determining that the decibel value for a corresponding audio input signal is less than a predetermined decibel threshold value. ### Claim 20 (depends on 19) The computer program product of claim 19, wherein the computer-readable medium further comprises a seventh set of codes for causing a computer processor to (i) receive audio input signals having a decibel value greater than the predetermined decibel threshold value, and (ii) convert the audio input signal from the digital format to a spectrogram that comprises a visual representation of a spectrum of frequencies of the audio input signal as the audio input signal varies over time.