# US 2023/0402055 A1 · System and method for matching a visual source with a sound signal

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0402055 A1
- Document ID: US20230402055A1
- Status: Published application
- Kind: A1
- Application number: 18/163,618
- Filed: 2023-02-02
- Published: 2023-12-14
- Earliest claimed: 2022-06-08
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Sathyanarayanan KULASEKARAN; Sarthak SENGUPTA; Sujoy SAHA
- Web page: https://patentgrep.com/patent/US20230402055A1

## CPC

- G10L 21/0308: characterised by the type of parameter measurement, e.g. correlation techniques, zero crossing techniques or predictive techniques
- G10L 25/51: for comparison or discrimination
- G10L 21/10: Transforming into visible information
- G10L 25/30: using neural networks
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A method for matching a visual source with a respective sound signal is provided. The method includes receiving a real-world sound input including a combination of one or more sound signals originating from a plurality of sound-generating objects, separating one or more sound signals from the real-world sound input, detecting sound generating objects included in the visual source, generating an association between each of the sound generating objects and the one or more separated sound signals, and matching, in real-time, each of the detected sound generating objects with respective sound signals from the one or more separated sound signals based on the association.

## Claims

### Claim 1

A method for matching a visual source with a respective sound signal, the method comprising:
receiving a real-world sound input including a combination of one or more sound signals originating from a plurality of sound-generating objects;
separating the one or more sound signals of the real-world sound input;
detecting one or more sound generating objects included in the visual source;
generating an association between each of the sound generating objects and the one or more separated sound signals; and
matching, in real-time, each of the detected sound generating objects with a respective sound signal from the one or more separated sound signals based on the association.

### Claim 2 (depends on 1)

The method of claim 1,
wherein generating the association between each of the sound generating objects and the one or more separated sound signals is based on contrastive learning, and
wherein the contrastive learning is based on a permutation invariant contrastive learning.

### Claim 3 (depends on 1)

The method of claim 1, wherein the real-world sound input corresponds to the visual source indicative of a camera-preview.

### Claim 4 (depends on 1)

The method of claim 1, wherein detecting the one or more sound generating objects is based on an object detection technique.

### Claim 5 (depends on 1)

The method of claim 1, wherein separating the one or more sound signals comprises:
generating a spectrogram of the real-world sound input;
identifying at least one class for each of the sound signals in the spectrogram, wherein the at least one class is indicative of a type of a sound generating object;
generating a heat map for the at least one class; and
determining the separated sound signals based on the heat map.

### Claim 6 (depends on 5)

The method of claim 5, wherein identifying the at least one class for the sound signals in the spectrogram, further comprises:
splitting the spectrogram into a plurality of horizontal patches based on a plurality of predefined classes, using a neural network trainable for splitting the spectrogram based on the plurality of predefined classes;
extracting a plurality of features from the plurality of horizontal patches;
concatenating the plurality of features;
determining weights from the concatenated plurality of features; and
identifying the separated sound signals corresponding to the predefined classes based on the determined weights.

### Claim 7 (depends on 5)

The method of claim 5, wherein the heat map is generated using a Class Activation Mapping (CAM) technique.

### Claim 8 (depends on 1)

The method of claim 1, further comprising computing a masking loss based on the separated sound signals.

### Claim 9

A method of processing a visual source (106), the method comprising:
receiving a preview of the visual source including a plurality of objects;
detecting one or more sound generating objects from the plurality of objects in the visual source as a source of real-world sound; and
displaying one or more controlling markers in a user-interface for controlling sound signals generated by each of the detected sound generating objects,
wherein each of the identified sound generating objects is mapped to a respective sound signal.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
measuring a magnitude of each of the respective sound signals mapped to each of the sound generating objects in the preview, upon separating each of the respective sound signals from a real-world sound input associated with the preview;
indicating the magnitude of each of the respective sound signals using one or more user interface controls, wherein the one or more user interface controls correspond to the one or more controlling markers; and
receiving a controlling input from a user to vary a position of at least one user interface control of the one or more user interface controls to one of increase or decrease the magnitude of a sound signal of the respective sound generating object associated with the at least one user interface control.

### Claim 11

A system for matching a visual source with a respective sound signal, the system comprising:
a receiving module configured to receive a real-world sound input including a combination of one or more sound signals originating from a plurality of sound-generating objects;
a separating module configured to separate the one or more sound signals of the real-world sound input;
a detecting module configured to detect one or more sound generating objects included in the visual source; and
a generating module configured to:
generate an association between each of the sound generating objects and the one or more separated sound signals, and
match, in real-time, each of the detected sound generating objects with a respective sound signal from the one or more separated sound signals based on the association.

### Claim 12 (depends on 11)

The system of claim 11,
wherein the generating module is further configured to:
generate the association between each of the sound generating objects and the one or more separated sound signals based on contrastive learning, and
wherein the contrastive learning is based on a permutation invariant contrastive learning.

### Claim 13 (depends on 11)

The system of claim 11, wherein the real-world sound input corresponds to the visual source indicative of a camera-preview.

### Claim 14 (depends on 11)

The system of claim 11, wherein the detecting module is further configured to detect the one or more sound generating objects based on an object detection technique.

### Claim 15 (depends on 14)

The system of claim 14, wherein the detecting module is further configured to apply an object detection bounding box technique to mark a box area around each sound generating object in the visual source.

### Claim 16 (depends on 11)

The system of claim 11, wherein the separating module is further configured to separate the one or more sound signals by:
generating a spectrogram of the real-world sound input,
identifying at least one class for each of the sound signals in the spectrogram, wherein the at least one class is indicative of a type of a sound generating object,
generating a heat map for the at least one class, and
determining the separated sound signals based on the generated heat map.

### Claim 17 (depends on 16)

The system of claim 16, wherein the separating module is further configured to identify the at least one class for the sound signals in the spectrogram by:
splitting the spectrogram into a plurality of horizontal patches based on a plurality of predefined classes, wherein a neural network is trainable for splitting the spectrogram based on the plurality of predefined classes;
extracting a plurality of features from the plurality of horizontal patches;
concatenating the plurality of features;
determining weights from the concatenated plurality of features; and
identifying the separated sound signals corresponding to the predefined classes based on the determined weights.

### Claim 18 (depends on 17)

The system of claim 17, wherein the neural network is trained in a supervised learning technique with a labeled training set.

### Claim 19 (depends on 18)

The system of claim 18, wherein the labeled training set is indicative of the predefined classes.
