# US 2015/0086038 A1 · Time-frequency directional processing of audio signals

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0086038 A1
- Document ID: US20150086038A1
- Status: Published application
- Kind: A1
- Application number: 14/494,838
- Filed: 2014-09-24
- Published: 2015-03-26
- Earliest claimed: 2013-09-24
- Assignees: Analog Devices, Inc.
- Applicants: Analog Devices, Inc.
- Inventors: Noah Stein; Johannes Traa; David Wingate
- Web page: https://patentgrep.com/patent/US20150086038A1

## CPC

- H04R 1/326: {for microphones (H04R1/34 and H04R1/40 take precedence)}
- G10L 21/0272: Voice signal separating
- G10L 2021/02166: {Microphone arrays; Beamforming}
- H04R 1/406: {microphones}
- H04R 2201/003: Mems transducers or their use
- H04R 2430/21: Direction finding using differential microphone array [DMA]
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

An approach to processing of acoustic signals acquired at a user's device include one or both of acquisition of parallel signals from a set of closely spaced microphones, and use of a multi-tier computing approach in which some processing is performed at the user's device and further processing is performed at one or more server computers in communication with the user's device. The acquired signals are processed using time versus frequency estimates of both energy content as well as direction of arrival. In some examples, a non-negative matrix or tensor factorization approach is used to identify multiple sources each associated with a corresponding direction of arrival of a signal from that source. In some examples, data characterizing direction of arrival information is passed from the user's device to a server computer where direction-based processing is performed.

## Claims

### Claim 1

A method for processing a plurality of signals acquired using a corresponding plurality of acoustic sensors at a user device, said signals having parts from a plurality of spatially distributed acoustic sources, the method comprising:
computing, using a processor at the user device, time-dependent spectral characteristics from at least one signal of the plurality of acquired signals, the spectral characteristics comprising a plurality of components;
computing, using the processor at the user device, direction estimates from at least two signals of the plurality of acquired signals, each computed component of the spectral characteristics having a corresponding one of the direction estimates;
performing a decomposition procedure using the computed spectral characteristics and the computed direction estimates as input to identify a plurality of sources of the plurality of signals, each component of the spectral characteristics having a computed degree of association with at least one of the identified sources and each source having a computed degree of association with at least one direction estimate; and
using a result of the decomposition procedure to selectively process a signal from one of the sources.

### Claim 2 (depends on 1)

The method of claim 1 wherein each component of the plurality of components of the time-dependent spectral characteristics computed from the acquired signals is associated with a time frame of a plurality of successive time frames.

### Claim 3 (depends on 2)

The method of claim 2 wherein each component of the plurality of components of the time-dependent spectral characteristics computed from the acquire signals is associated with a frequency range, whereby the computed components form a time-frequency characterization of the acquired signals.

### Claim 4 (depends on 3)

The method of claim 3 wherein each component represents energy at a corresponding range of time and frequency.

### Claim 5 (depends on 1)

The method of claim 1 computing the direction estimates of component comprises computing data representing a direction of arrival of the component in the acquired signals.

### Claim 6 (depends on 5)

The method of claim 5 wherein computing the data representing the directional of arrival comprises at least one of (a) computing data representing one direction of arrival, and (b) computing data representing an exclusion of at least one direction of arrival.

### Claim 7 (depends on 5)

The method of claim 5 wherein computing the data representing the direction of arrival comprises determining an optimized direction associated with the component using at least one of (a) phases, and (b) times of arrivals of the acquired signals.

### Claim 8 (depends on 7)

The method of claim 7 wherein determining the optimized direction comprises performing at least one of (a) a pseudo-inverse calculation, and (b) a least-squared-error estimation.

### Claim 9 (depends on 5)

The method of claim 5 wherein computing the data representing the direction of arrival comprises computing at least one of (a) an angle representation of the direction of arrival, (b) a direction vector representation of the direction of arrival, and (c) a quantized representation of the direction of arrival.

### Claim 10 (depends on 1)

The method of claim 1 wherein performing the decomposing comprises combining the computed spectral characteristics and the computed direction estimates to form a data structure representing a distribution indexed by time, frequency, and direction.

### Claim 11 (depends on 10)

The method of claim 10 further comprising performing a non-negative tensor factorization using the formed data structure.

### Claim 12 (depends on 10)

The method of claim 10 wherein forming the data structure comprises forming a sparse data structure in which a majority of the entries of the distribution are absent.

### Claim 13 (depends on 10)

The method of claim 10 wherein performing the decomposition comprises determining the result including a degree of association of each component with a corresponding source.

### Claim 14 (depends on 13)

The method of claim 13 wherein the degree of association comprises a binary degree of association.

### Claim 15 (depends on 1)

The method of claim 1 wherein using the result of the decomposition to selectively process the signal from one of the sources comprises forming a time signal as an estimate of a part of the acquired signals corresponding to said source.

### Claim 16 (depends on 15)

The method of claim 15 wherein forming the time signal comprises using the computed degrees of association of the components with the identified sources to form said time signal.

### Claim 17 (depends on 1)

The method of claim 1 wherein using the result of the decomposition to selectively process the signal from one of the sources comprises performing an automatic speech recognition using an estimated part of the acquired signals corresponding to said source.

### Claim 18 (depends on 1)

The method of claim 1 wherein at least part of performing the decomposition process and using the result of the decomposition procedure is performed as a server computing system in data communication with the user device.

### Claim 19 (depends on 18)

The method of claim 18 further comprising communicating from the user device to the server computing system at least one of (a) the direction estimates, (b) a result of the decomposition procedure, and (c) a signal formed using a result of the decomposition as an estimate of a part of the acquired signals.

### Claim 20 (depends on 18)

The method of claim 18 further comprising communicating a result of the using of the result of the decomposition procedure from the server computing system to the user device.

### Claim 21 (depends on 18)

The method of claim 18 further comprising communicating data from the server computing system to the user device for use in performing the decomposition procedure at the user device.

### Claim 22

A signal processing system comprising:
an acoustic sensor, integrated in a user device, having multiple sensor elements; and
a processor integrated in the user device;
wherein the processor is configured to
compute, using the processor at the user device, time-dependent spectral characteristics from at least one signal of the plurality of acquired signals, the spectral characteristics comprising a plurality of components;
compute, using the processor at the user device, direction estimates from at least two signals of the plurality of acquired signals, each computed component of the spectral characteristics having a corresponding one of the direction estimates;
perform a decomposition procedure using the computed spectral characteristics and the computed direction estimates as input to identify a plurality of sources of the plurality of signals, each component of the spectral characteristics having a computed degree of association with at least one of the identified sources and each source having a computed degree of association with at least one direction estimate; and
cause use of a result of the decomposition procedure to selectively process a signal from one of the sources.

### Claim 23 (depends on 22)

The signal processing system of claim 22 wherein causing use of the result comprises using the processor of the user device to selectively process the signal.

### Claim 24 (depends on 22)

The signal processing system of claim 22 further comprising a communication interface for communicating with a server computer, and wherein causing use of the result comprises transmitting the result of the decomposition procedure via the communication interface to the server computer.

### Claim 25

Software comprising instructions embodied on a non-transitory machine readable medium, execution of said instructions on one or more processors of a data processing system causing said system to
compute time-dependent spectral characteristics from at least one signal of the plurality of acquired signals, the spectral characteristics comprising a plurality of components;
compute direction estimates from at least two signals of the plurality of acquired signals, each computed component of the spectral characteristics having a corresponding one of the direction estimates;
perform a decomposition procedure using the computed spectral characteristics and the computed direction estimates as input to identify a plurality of sources of the plurality of signals, each component of the spectral characteristics having a computed degree of association with at least one of the identified sources and each source having a computed degree of association with at least one direction estimate; and
cause use of a result of the decomposition procedure to selectively process a signal from one of the sources.

### Claim 26

A method for source characterization from a plurality of signals acquired using a plurality of sensors comprising:
computing, from the plurality of acquired signals, a directional characteristic for each of at least some of components of the signals for each of a plurality of successive time frames;
computing a directional distribution over components of the directional characteristics for at least some of the successive time frames;
computing a statistical characterization of the directional distributions, the statistical characterization including degrees of association of the directional characteristics; and
identifying a directional characteristic of a least one source in the acquired signals from the statistical characterization of the directional distributions.

### Claim 27

A method for signal separation comprising:
acquiring one or more input signals;
computing an input mask comprising a plurality of values indexed by a time index and a signal component index, the mask values indicative of components of a desired signal represented in the one or more input signals;
performing an iterative computation using the input mask to form a smoothed mask, each iteration of the computation including
processing the smoothed mask using a convolution operation;
selecting a subset of time index and component index pairs;
updating the smoothed mask, including for each selected time index and component index pair, independently of the other selected time index and component index pairs, setting the value of the smoothed mask to a randomly selected quantity, wherein the selection depends on a result of the combination operation of the smoothed mask values in a vicinity of that selected time and component index pair;
using the smoothed mask determined at one or more or the iterations to separate the desired signal from the one or more input signal.
