# US 2024/0194213 A1 · Audio Source Separation using Hyperbolic Embeddings

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0194213 A1
- Document ID: US20240194213A1
- Status: Published application
- Kind: A1
- Application number: 18/191,417
- Filed: 2023-03-28
- Published: 2024-06-13
- Earliest claimed: 2022-12-07
- Assignees: Mitsubishi Electric Research Laboratories, Inc.
- Applicants: Mitsubishi Electric Research Laboratories, Inc.
- Inventors: Gordon Wichern; Jonathan Le Roux; Darius Petermann; Aswin Shanmugam Subramanian
- Web page: https://patentgrep.com/patent/US20240194213A1

## CPC

- G10L 21/0308: characterised by the type of parameter measurement, e.g. correlation techniques, zero crossing techniques or predictive techniques
- G01H 3/08: Analysing frequencies present in complex vibrations, e.g. comparing harmonics present {(acoustic presence detection G01V1/001)}
- G10L 21/0272: Voice signal separating
- G10L 21/06: Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids (G10L15/26 takes precedence)
- G10L 25/18: the extracted parameters being spectral information of each sub-band
- G10L 25/21: the extracted parameters being power information
- G10L 25/30: using neural networks
- G10L 25/51: for comparison or discrimination
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G01H: MEASUREMENT OF MECHANICAL VIBRATIONS OR ULTRASONIC, SONIC OR INFRASONIC WAVES

## Abstract

There is provided an audio processing system and method comprising an input interface that receives an input audio mixture and transforms it into a time-frequency representation defined by values of time-frequency bins, a processor that maps the values of time-frequency bins into a hyperbolic space by executing an embedding neural network trained to associate each time-frequency bin to a high-dimensional embedding and projecting each high-dimensional embedding into the hyperbolic space, and an output interface that accepts a selection of at least a portion of the hyperbolic space and renders selected hyperbolic embeddings falling within the selected portion of the hyperbolic space.

## Claims

### Claim 1

An audio processing system, comprising:
an input interface configured to receive an input audio mixture and transform it into a time-frequency representation defined by values of time-frequency bins;
a processor configured to map the values of time-frequency bins into a hyperbolic space by executing an embedding neural network trained to associate each time-frequency bin to a high-dimensional embedding and projecting each high-dimensional embedding into the hyperbolic space; and
an output interface configured to accept a selection of at least a portion of the hyperbolic space and render selected hyperbolic embeddings falling within the selected portion of the hyperbolic space.

### Claim 2 (depends on 1)

The audio processing system of claim 1, wherein
to render the selected hyperbolic embeddings, the output interface is further configured to transform the selected hyperbolic embeddings into a separated output audio signal and send the separated output audio signal to a memory, and
the memory stores the separated output signal.

### Claim 3 (depends on 2)

The audio processing system of claim 2, wherein the output interface is further configured to send the separated output signal to a loudspeaker.

### Claim 4 (depends on 1)

The audio processing system of claim 1, wherein the output interface is further configured to transform the selected hyperbolic embeddings into a separated output audio signal by creating a time-frequency mask based on the selected hyperbolic embeddings and applying the time-frequency mask to the time-frequency representation of the input audio mixture.

### Claim 5 (depends on 4)

The audio processing system of claim 4, wherein the output interface is further configured to create the time-frequency mask based on a softmax operation.

### Claim 6 (depends on 1)

The audio processing system of claim 1, wherein the hyperbolic space is a Poincaré ball or a Poincaré disk classified according to a hyperbolic geometry that carries a notion of classification hierarchy of audio sources based on locations of the hyperbolic embeddings with respect to an origin of the Poincaré ball or the Poincaré disk.

### Claim 7 (depends on 6)

The audio processing system of claim 6, wherein a distance from the origin of the hyperbolic space to each hyperbolic embedding is used to derive a measure of certainty of the processing, and the creating of the time-frequency mask is based on the measure of certainty of the processing.

### Claim 8 (depends on 6)

The audio processing system of claim 6, wherein the processor is further configured to determine, based on a distance of a hyperbolic embedding from the origin of the Poincare ball or the Poincaré disk, a measure of certainty of the hyperbolic embedding to belong to only a single specific audio class on the classification hierarchy.

### Claim 9 (depends on 6)

The audio processing system of claim 6, wherein the processor is further configured to determine the input audio mixture as an anomalous sound based on a number of the hyperbolic embeddings located within a threshold distance from the origin of the Poincaré ball or the Poincaré disk is greater than or equal to a threshold value.

### Claim 10 (depends on 6)

The audio processing system of claim 6, wherein
the input audio mixture is generated by components of a machine,
the processor is further configured to detect an anomaly in the machine based on a number of the hyperbolic embeddings located within a threshold distance from the origin of the Poincaré ball or the Poincaré disk is greater than or equal to a threshold value.

### Claim 11 (depends on 10)

The audio processing system of claim 10, wherein the processor is further configured to control the machine based on the detected anomaly in the machine.

### Claim 12 (depends on 1)

The audio processing system of claim 1, wherein the hyperbolic space is classified according to a classification hierarchy of audio sources, and wherein the embedding neural network is trained end-to-end with a classifier trained to classify the hyperbolic embeddings according to the classification hierarchy.

### Claim 13 (depends on 1)

The audio processing system of claim 1, wherein the output interface is operatively connected to a display device configured to display a visual representation of the hyperbolic embeddings mapped to different locations of the hyperbolic space to enable the selection of the portion of the hyperbolic space.

### Claim 14 (depends on 1)

The audio processing system of claim 1, wherein
training data set for training the embedding neural network includes an audio mixture of at least two parent classes and at least five child classes,
the at least two parent classes include music and speech, and
the at least five child classes include bass, drum, guitar, speech-male, and speech-female.

### Claim 15 (depends on 1)

The audio processing system of claim 1, wherein
the processor is further configured to receive a user input, and
a size and a shape of the selected portion of the hyperbolic space is based on the received user input.

### Claim 16 (depends on 12)

The audio processing system of claim 12, wherein
the classifier segments the hyperbolic space using hyperbolic hyperplanes according to the classification hierarchy, and
the output interface is further configured to create a T-F mask for each audio class based on the hyperbolic hyperplanes.

### Claim 17 (depends on 16)

The audio processing system of claim 16, wherein the output interface is further configured to generate an output signal for each audio class based on the T-F mask created for a corresponding audio class.

### Claim 18 (depends on 1)

The audio processing system of claim 1, wherein the processor is further configured to:
accept a selection of weight on energy of the selected hyperbolic embeddings; and
render the selected hyperbolic embeddings based on the weight on the energy of the selected hyperbolic embeddings.

### Claim 19

An audio processing method, comprising:
receiving an input audio mixture and transform it into a time-frequency representation defined by values of time-frequency bins;
mapping the values of time-frequency bins into a hyperbolic space by executing an embedding neural network trained to associate each time-frequency bin to a high-dimensional embedding and projecting each high-dimensional embedding into the hyperbolic space; and
accepting a selection of at least a portion of the hyperbolic space and rendering selected hyperbolic embeddings falling within the selected portion of the hyperbolic space.

### Claim 20

A non-transitory computer-readable storage medium embodied thereon a program executable by a processor for performing a method, comprising:
receiving an input audio mixture and transform it into a time-frequency representation defined by values of time-frequency bins;
mapping the values of time-frequency bins into a hyperbolic space by executing an embedding neural network trained to associate each time-frequency bin to a high-dimensional embedding and projecting each high-dimensional embedding into the hyperbolic space; and
accepting a selection of at least a portion of the hyperbolic space and render selected hyperbolic embeddings falling within the selected portion of the hyperbolic space.
