# US 2024/0265932 A1 · Device and method for automatically removing a background sound source of a video

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0265932 A1
- Document ID: US20240265932A1
- Status: Published application
- Kind: A1
- Application number: 18/641,485
- Filed: 2024-04-22
- Published: 2024-08-08
- Earliest claimed: 2021-10-26
- Assignees: SK Telecom Co., Ltd.
- Applicants: SK Telecom Co., Ltd.
- Inventors: Dong Won KIM; Suk Bong KWON; Yong Hyun PARK; Jong Kil YUN; Jeong Yeon LIM
- Web page: https://patentgrep.com/patent/US20240265932A1

## CPC

- G10L 21/0272: Voice signal separating
- G06N 3/08: Learning methods
- G06N 20/00: Machine learning
- G10L 13/027: Concept to speech synthesisers; Generation of natural phrases from machine-based concepts (generation of parameters for speech synthesis out of text G10L13/08)
- G11B 27/02: Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
- H04N 21/434: Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream {(demultiplexing of data packets for data networks, e.g. RTP/UDP H04L65/00)}
- H04N 21/439: Processing of audio elementary streams
- G10L 25/57: for processing of video signals
- G10L 25/81: for discriminating voice from music
- G10L 2021/02087: {the noise being separate speech, e.g. cocktail party}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

One aspect of the present disclosure provides a method for automatically removing a background sound source of audio data of a video, including separating the audio data including at least one sound source component into a first component related to a human voice and a second component related to sounds other than the human voice using a first separation model, separating the first component into a vocal component and a speech component using a second separation model, separating the second component into a music component and a noise component using a third separation model, and generating an audio data with the background sound source for the audio data of the video removed by synthesizing the speech component and the noise component.

## Claims

### Claim 1

A method of automatically removing a background sound source of audio data, comprising:
separating, using a first separation model, the audio data including at least one sound source component into a first component related to a human voice and a second component related to sounds other than the human voice;
separating, using a second separation model, the first component into a vocal component and a speech component; and
generating, based on the speech component, an audio data with the background sound source for the audio data removed.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
separating, using a third separation model, the second component into a music component and a noise component,
wherein the audio data with the background sound source removed is generated by synthesizing the speech component and the noise component.

### Claim 3 (depends on 1)

The method of claim 1, wherein the first separation model is trained using a training method including:
generating first training data based on a first component including at least one of a speech component and a vocal component for a first dataset and a second component including at least one of a music component and a noise component for the first dataset;
separating the first training data into the first component for the first training data and the second component for the first training data using the first separation model;
calculating a first separation loss based on the first component for the first training data, the second component for the first training data, and a corresponding ground truth; and
updating at least one weight of the first separation model based on the first separation loss.

### Claim 4 (depends on 3)

The method of claim 3, wherein the second separation model is trained using a training method including:
generating second training data based on a speech component and a vocal component for a second dataset;
separating the second training data into a speech component for the second training data and a vocal component for the second training data using the second separation model;
calculating quality data for the second training data using a pre-trained vocal detection model;
calculating a second detection loss related to a speech component for the second training data and a vocal component for the second training data using the vocal detection model;
calculating a second separation loss based on the speech component for the second training data and the vocal component for the second training data, which are separated by the second separation model, and a corresponding ground truth;
calculating a second combined loss based on the quality data for the second training data, the second detection loss, and the second separation loss; and
updating at least one weight of the second separation model based on the second combined loss.

### Claim 5 (depends on 2)

The method of claim 2, wherein the third separation model is trained using a training method including:
generating third training data based on a music component and a noise component for a third dataset;
separating the third training data into a music component for the third training data and a noise component for the third training data using the third separation model;
calculating quality data for the third training data using a pre-trained music detection model;
calculating a third detection loss related to a music component for the third training data and a noise component for the third training data using the music detection model;
calculating a third separation loss based on a music component for the third training data separated by the third separation model, a noise component for the third training data, and a corresponding ground truth;
calculating a third combined loss based on the quality data for the third training data, the third detection loss, and the third separation loss; and
updating at least one weight of the third separation model based on the third combined loss.

### Claim 6 (depends on 1)

The method of claim 1, wherein the second separation model is trained using a training method including:
separating a training data into a speech component for training data and a vocal component for the training data using the second separation model;
calculating a probability that the speech component for the training data is a vocal component using a pre-trained vocal detection model;
calculating a probability that the vocal component for the training data is a vocal component using the vocal detection model;
generating a first vocal detection loss based on the probability that the speech component for the training data is a vocal component;
generating a second vocal detection loss based on the probability that the vocal component for the training data is a vocal component; and
updating at least one weight of the second separation model based on the first vocal detection loss and the second vocal detection loss.

### Claim 7 (depends on 2)

The method of claim 2, wherein the third separation model is trained using a training method including:
separating a training data into a music component for the training data and a noise component for the training data using the third separation model;
calculating a probability that the music component for the training data is the music component using a trained music detection model;
calculating a probability that the noise component for the training data is the music component using the music detection model;
generating a first music detection loss based on the probability that the music component for the training data is the music component;
generating a second music detection loss based on the probability that the noise component for the training data is the music component; and
updating at least one weight of the third separation model based on the first music detection loss and the second music detection loss.

### Claim 8

A device for automatically removing a background sound source, comprising:
a memory configured to store one or more instructions; and
a processor configured to execute the one or more instructions stored in the memory, wherein the processor executes the one or more instructions to
separate, using a first separation model, an audio data including at least one sound source component into a first component related to a human voice and a second component related to sounds other than a human voice,
separate, using a second separation model, the first component into a vocal component and a speech component, and
generate, based on the speech component, an audio data with the background sound source for the audio data removed.

### Claim 9 (depends on 8)

The device of claim 8, wherein the processor separates the second component into a music component and a noise component using a third separation model, and
the audio data with the background sound source removed is generated by synthesizing the speech component and the noise component.
