# US 10,771,894 B2 · Method and apparatus for audio capture using beamforming > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,771,894 B2 - Document ID: US10771894B2 - Status: Granted - Kind: B2 - Application number: 16/473,370 - Filed: 2017-12-28 - Published: 2020-05-07 - Granted: 2020-09-08 - Earliest claimed: 2017-01-03 - Assignees: Koninklijke Philips N.V. - Applicants: Koninklijke Philips N.V. - Inventors: Cornelis Pieter Janse; Brian Brand Antonius Johannes Bloemendal; Patrick Kechichian; Rik Jozef Martinus Janssen - Web page: https://patentgrep.com/patent/US10771894B2 ## CPC - H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)} - H04R 1/406: {microphones} - G10L 2021/02166: {Microphone arrays; Beamforming} - H04R 2430/20: Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic (H04R2203/12 takes precedence) - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract An apparatus for capturing audio comprises a first beamformer (305) coupled to a microphone array (301) and arranged to generate a first beamformed audio output. A plurality of constrained beamformers (309, 311) each generates a constrained beamformed audio output. A first adapter (307) adapts beamform parameters of the first beamformer (305) and a second adapter (313) adapts constrained beamform parameters for the plurality of constrained beamformers (309, 311). A difference processor (317) determines a difference measure for the constrained beamformers (309, 311) where the difference measure is indicative of the difference between beams formed by the first beamformer (305) and the constrained beamformers (309, 311). The second adapter (313) is arranged to adapt constrained beamform parameters with the constraint that beamform parameters are adapted only for constrained beamformers of the plurality of constrained beamformers (309, 311) for which a difference measure has been determined that meets a similarity criterion. ## Claims ### Claim 1 An apparatus for capturing audio, the apparatus comprising: a microphone array; a first beamformer, the beamformer coupled to the microphone array, wherein the beamformer is arranged to generate a first beamformed audio output; a plurality of constrained beamformers, the plurality of constrained beamformers coupled to the microphone array, wherein each of the plurality of constrained beamformers is arranged to generate a constrained beamformed audio output; a first adapter, wherein the first adaptor is arranged to adapt beamform parameters of the first beamformer; a second adapter, wherein the second adaptor is arranged to adapt constrained beamform parameters for the plurality of constrained beamformers; a difference processor circuit, wherein the difference processor circuit is arranged to determine a difference measure for at least one of the plurality of constrained beamformers, wherein the difference measure is indicative of a difference between beams formed by the first beamformer and the at least one of the plurality of constrained beamformers; wherein the second adapter is arranged to adapt constrained beamform parameters with a constraint such that constrained beamform parameters are adapted only for constrained beamformers of the plurality of constrained beamformers for which a difference measure has been determined that meets a similarity criterion, wherein the difference processor circuit is arranged to determine the difference measure for a first constrained beamformer as a difference between a first set of parameters and the constrained set of parameters for the first constrained beamformer. ### Claim 2 (depends on 1) The apparatus of claim 1 further comprising an audio source detector, wherein the audio source detector is arranged to detect point audio sources in the constrained beamformed audio outputs, wherein the second adapter is arranged to adapt constrained beamform parameters only for constrained beamformers for which a presence of a point audio source is detected in the constrained beamformed audio output. ### Claim 3 (depends on 2) The apparatus of claim 2, wherein the audio source detector is arranged to detect point audio sources in the first beamformed audio output, wherein the apparatus further comprises a controller circuit is arranged to set constrained beamform parameters for a first constrained beamformer in response to beamform parameters of the first beamformer if a point audio source is detected in the first beamformed audio output but not in any constrained beamformed audio outputs. ### Claim 4 (depends on 3) The apparatus of claim 3, wherein the controller circuit is arranged to set the constrained beamform parameters for the first constrained beamformer in response to the beamform parameters of the first beamformer, wherein the controller circuit is arranged to set the constrained beamform parameters only if a difference measure for the first constrained beamformer exceeds a threshold. ### Claim 5 (depends on 2) The apparatus of claim 2, wherein the audio source detector is arranged to detect audio sources in the first beamformed audio output, wherein the apparatus further comprises a controller circuit arranged to set constrained beamform parameters for a first constrained beamformer in response to the beamform parameters of the first beamformer, wherein the controller circuit is arranged to set the constrained beamform parameters if a point audio source is detected in the first beamformed audio output and in a second beamformed audio output from the first constrained beamformer and a difference measure has been determined for the first constrained beamformer which exceeds a threshold. ### Claim 6 (depends on 5) The apparatus of claim 5, wherein the plurality of constrained beamformers is an active subset of the constrained beamformers, wherein the active subset of constrained beamformers is selected from a pool of constrained beamformers, wherein the controller circuit is arranged to increase a number of active constrained beamformers to include the first constrained beamformer by initializing a constrained beamformer from the pool of constrained beamformers using the beamform parameters of the first beamformer. ### Claim 7 (depends on 1) The apparatus of claim 1, wherein the second adapter is arranged to only adapt the constrained beamform parameters for a first constrained beamformer if a criterion is met comprising at least one requirement selected from the group of: a requirement that a level of the second beamformed audio output from the first constrained beamformer is higher than for any other second beamformed audio output, a requirement that a level of a point audio source in the second beamformed audio output from the first constrained beamformer is higher than any point audio source in any other second beamformed audio output, a requirement that a signal to noise ratio for the second beamformed audio output from the first constrained beamformer exceeds a threshold, and a requirement that the second beamformed audio output from the first constrained beamformer comprises a speech component. ### Claim 8 (depends on 1) The apparatus of claim 1, wherein an adaptation rate for the first beamformer is higher than for the plurality of constrained beamformers. ### Claim 9 (depends on 1) The apparatus of claim 1 wherein the first beamformer and the plurality of constrained beamformers are filter-and-combine beamformers. ### Claim 10 (depends on 1) The apparatus of claim 1, wherein the first beamformer is a filter-and-combine beamformer comprising a first plurality of beamform filters, wherein each of the first plurality of beamform filters has a first adaptive impulse responses, wherein a second beamformer is a constrained beamformer of the plurality of constrained beamformers, wherein the second beamformer is a filter-and-combine beamformer comprising a second plurality of beamform filters, wherein each of the second plurality of beamform filters has having a second adaptive impulse response, wherein the difference processor circuit is arranged to determine the difference measure between beams of the first beamformer and the second beamformer in response to a comparison of the first adaptive impulse responses to the second adaptive impulse responses. ### Claim 11 (depends on 1) The apparatus of claim 1 further comprising: a noise reference beamformer, wherein the noise reference beamformer is arranged to generate a beamformed audio output signal and at least one noise reference signal, wherein the noise reference beamformer is one of the first beamformer and the plurality of constrained beamformers; a first transformer, wherein the first transform is arranged to generate a first frequency domain signal from a frequency transform of the beamformed audio output signal, wherein the first frequency domain signal is represented by time frequency tile values; a second transformer, wherein the first transform is arranged to generate a second frequency domain signal from a frequency transform of the at least one noise reference signal, wherein the second frequency domain signal is represented by time frequency tile values; the difference processor circuit, the difference processor circuit arranged to generate time frequency tile difference measures, wherein a time frequency tile difference measure for a first frequency is indicative of a difference between a first monotonic function of a norm of a time frequency tile value of the first frequency domain signal for the first frequency and a second monotonic function of a norm of a time frequency tile value of the second frequency domain signal for the first frequency; and a point audio source estimator, wherein the point audio source estimator is arranged to generate a point audio source estimate indicative of whether the beamformed audio output signal comprises a point audio source, wherein the point audio source estimator is arranged to generate the point audio source estimate in response to a combined difference value for time frequency tile difference measures for frequencies above a frequency threshold. ### Claim 12 (depends on 11) The audio capturing apparatus of claim 11, wherein the point audio source estimator is arranged to detect a presence of a point audio source in the beamformed audio output in response to the combined difference value exceeding a threshold. ### Claim 13 A method of capturing audio the method comprising, generating a first beamformed audio output using a first beamformer coupled to a microphone array; generating a constrained beamformed audio output using a plurality of constrained beamformers coupled to the microphone array; adapting beamform parameters of the first beamformer; adapting constrained beamform parameters for the plurality of constrained beamformers; determining a difference measure for at least one of the plurality of constrained beamformers, wherein the difference measure is indicative of a difference between beams formed by the first beamformer and the at least one of the plurality of constrained beamformers, wherein adapting constrained beamform parameters comprises adapting constrained beamform parameters with a constraint such that constrained beamform parameters are adapted only for constrained beamformers of the plurality of constrained beamformers for which a difference measure has been determined that meets a similarity criterion, wherein the difference processor circuit is arranged to determine the difference measure for a first constrained beamformer as a difference between a first set of parameters and the constrained set of parameters for the first constrained beamformer. ### Claim 14 (depends on 13) A computer program product comprising computer program code in a non-transitory media, wherein the computer code is arranged to perform all the steps of claim 13 when the program is run on a computer. ### Claim 15 (depends on 13) The method of claim 13 further comprising: detecting point audio sources in the constrained beamformed audio outputs, adapting constrained beamform parameters only for constrained beamformers for which a presence of a point audio source is detected in the constrained beamformed audio output. ### Claim 16 (depends on 15) The method of claim 15, further comprising: wherein the detecting of point audio sources arranged to detect point audio sources in the first beamformed audio output, setting constrained beamform parameters for a first constrained beamformer in response to beamform parameters of the first beamformer if a point audio source is detected in the first beamformed audio output but not in any constrained beamformed audio outputs. ### Claim 17 (depends on 16) The method of claim 16, further comprising: setting the constrained beamform parameters for the first constrained beamformer in response to the beamform parameters of the first beamformer, setting the constrained beamform parameters only if a difference measure for the first constrained beamformer exceeds a threshold. ### Claim 18 (depends on 15) The method of claim 15, further comprising: detecting audio sources in the first beamformed audio output, setting constrained beamform parameters for a first constrained beamformer in response to the beamform parameters of the first beamformer, setting the constrained beamform parameters if a point audio source is detected in the first beamformed audio output and in a second beamformed audio output from the first constrained beamformer and a difference measure has been determined for the first constrained beamformer which exceeds a threshold. ### Claim 19 (depends on 18) The method of claim 18, wherein the plurality of constrained beamformers is an active subset of the constrained beamformers, wherein the active subset of constrained beamformers is selected from a pool of constrained beamformers, wherein a number of active constrained beamformers to include the first constrained beamformer increases by initializing a constrained beamformer from the pool of constrained beamformers using the beamform parameters of the first beamformer. ### Claim 20 (depends on 13) The method of claim 13, wherein an adaptation rate for the first beamformer is higher than an adaption rate for the plurality of constrained beamformers.