# US 10,770,050 B2 · Audio data processing method and apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,770,050 B2
- Document ID: US10770050B2
- Status: Granted
- Kind: B2
- Application number: 15/775,460
- Filed: 2017-06-02
- Published: 2018-11-15
- Granted: 2020-09-08
- Earliest claimed: 2016-07-01
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Bi Lei Zhu; Ke Li; Yong Jian Wu; Fei Yue Huang
- Web page: https://patentgrep.com/patent/US10770050B2

## CPC

- G10H 1/366: {with means for modifying or correcting the external signal, e.g. pitch correction, reverberation, changing a singer's voice}
- G10H 2210/005: Musical accompaniment, i.e. complete instrumental rhythm synthesis added to a performed melody, e.g. as output by drum machines
- G10H 2210/056: for extraction or identification of individual instrumental parts, e.g. melody, chords, bass; Identification or separation of instrumental parts by their characteristic voices or timbres
- G10H 2210/066: for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental
- G10H 2250/031: Spectrum envelope processing
- G10H 2250/215: Transforms, i.e. mathematical transforms into domains appropriate for musical signal processing, coding or compression
- G10L 21/0272: Voice signal separating
- G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

An audio data processing method and apparatus are provided. The method includes obtaining audio data. An overall spectrum of the audio data is obtained and separated into a singing voice spectrum and an accompaniment spectrum. An accompaniment binary mask of the audio data is calculated according to the audio data. The singing voice spectrum and the accompaniment spectrum are processed using the accompaniment binary mask, to obtain accompaniment data and singing voice data.

## Claims

### Claim 1

A method comprising:
obtaining audio data;
obtaining an overall spectrum of the audio data;
separating the overall spectrum into a first singing voice spectrum and a first accompaniment spectrum;
adjusting the overall spectrum according to the first singing voice spectrum and the first accompaniment spectrum, to obtain a second singing voice spectrum and a second accompaniment spectrum;
calculating an accompaniment binary mask of the audio data according to the audio data; and
processing the second singing voice spectrum and the second accompaniment spectrum using the accompaniment binary mask, to obtain accompaniment data and singing voice data.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the processing the second singing voice spectrum and the second accompaniment spectrum comprises:
filtering the second singing voice spectrum using the accompaniment binary mask, to obtain a third singing voice spectrum and an accompaniment subspectrum;
performing calculation using the accompaniment subspectrum and the second accompaniment spectrum, to obtain a third accompaniment spectrum; and
performing mathematical transformation on the third singing voice spectrum and the third accompaniment spectrum, to obtain the accompaniment data and singing voice data.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the filtering comprises:
multiplying the second singing voice spectrum by the accompaniment binary mask, to obtain the accompaniment subspectrum; and
subtracting the accompaniment subspectrum from the second singing voice spectrum, to obtain the third singing voice spectrum.

### Claim 4 (depends on 2)

The method according to claim 2, wherein the performing calculation comprises:
adding the accompaniment subspectrum and the second accompaniment spectrum, to obtain the third accompaniment spectrum.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the adjusting comprises:
calculating a singing voice binary mask according to the first singing voice spectrum and the first accompaniment spectrum; and
adjusting the overall spectrum by using the singing voice binary mask, to obtain the second singing voice spectrum and the second accompaniment spectrum.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the calculating comprises:
performing independent component analysis (ICA) on the audio data, to obtain first singing voice data and first accompaniment data; and
calculating the accompaniment binary mask according to the first singing voice data and the first accompaniment data,
wherein the singing voice spectrum and the accompaniment spectrum are processed using the accompaniment binary mask, to obtain second accompaniment data and second singing voice data.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the calculating the accompaniment binary mask according to the first singing voice data and the first accompaniment data comprises:
performing mathematical transformation on the first singing voice data and the first accompaniment data, to obtain a corresponding fourth singing voice spectrum and fourth accompaniment spectrum; and
calculating the accompaniment binary mask according to the fourth singing voice spectrum and the fourth accompaniment spectrum.

### Claim 8

An apparatus comprising:
at least one memory configured to store computer program code; and
at least one processor configured to access the at least one memory and operate according to the computer program code, the computer program code including:
first obtaining code configured to cause the at least one processor to obtain audio data;
second obtaining code configured to cause the at least one processor to obtain an overall spectrum of the audio data;
separation code configured to cause the at least one processor to separate the overall spectrum, to obtain a first singing voice spectrum and a first accompaniment spectrum;
adjustment code configured to cause the at least one processor to adjust the overall spectrum according to the first singing voice spectrum and the first accompaniment spectrum, to obtain a second singing voice spectrum and a second accompaniment spectrum
calculation code configured to cause the at least one processor to calculate an accompaniment binary mask of the audio data according to the audio data; and
processing code configured to cause the at least one processor to process the second singing voice spectrum and the second accompaniment spectrum using the accompaniment binary mask, to obtain accompaniment data and singing voice data.

### Claim 9 (depends on 8)

The apparatus according claim 8, wherein the processing code comprises:
filtration subcode configured to cause the at least one processor to filter the second singing voice spectrum using the accompaniment binary mask, to obtain a third singing voice spectrum and an accompaniment subspectrum;
first calculation subcode configured to cause the at least one processor to perform calculation using the accompaniment subspectrum and the second accompaniment spectrum, to obtain a third accompaniment spectrum; and
inverse transformation subcode configured to cause the at least one processor to perform mathematical transformation on the third singing voice spectrum and the third accompaniment spectrum, to obtain the accompaniment data and singing voice data.

### Claim 10 (depends on 9)

The apparatus according to claim 9, wherein the filtration submodule is configured to cause the at least one processor to:
multiply the second singing voice spectrum by the accompaniment binary mask, to obtain the accompaniment subspectrum; and
subtract the accompaniment subspectrum from the second singing voice spectrum, to obtain the third singing voice spectrum; and
the first calculation submodule is configured to cause the at least one processor to add the accompaniment subspectrum and the second accompaniment spectrum, to obtain the third accompaniment spectrum.

### Claim 11 (depends on 8)

The apparatus according to claim 8, wherein the adjustment code is configured to cause the at least one processor to:
calculate a singing voice binary mask according to the first singing voice spectrum and the first accompaniment spectrum; and
adjust the overall spectrum by using the singing voice binary mask, to obtain the first singing voice spectrum and the first accompaniment spectrum.

### Claim 12 (depends on 8)

The apparatus according to claim 8, wherein the calculation code comprises:
analysis subcode configured to cause the at least one processor to perform independent component analysis (ICA) on the audio data, to obtain first singing voice data and first accompaniment data; and
second calculation subcode configured to cause the at least one processor to calculate the accompaniment binary mask according to the first singing voice data and the first accompaniment data,
wherein the processing code is configured to cause the at least one processor to process the singing voice spectrum and the accompaniment spectrum using the accompaniment binary mask, to obtain second accompaniment data and second singing voice data.

### Claim 13 (depends on 12)

The apparatus according to claim 12, wherein the second calculation submodule is configured to cause the at least one processor to:
perform mathematical transformation on the first singing voice data and the first accompaniment data, to obtain a corresponding fourth singing voice spectrum and fourth accompaniment spectrum; and
calculate the accompaniment binary mask according to the fourth singing voice spectrum and the fourth accompaniment spectrum.

### Claim 14

A method comprising:
separating audio data into a singing voice spectrum and an accompaniment spectrum using an Azimuth Discrimination and Resynthesis (ADRess) method;
adjusting an overall spectrum of the audio data according to the singing voice spectrum and the accompaniment spectrum, to obtain an adjusted singing voice spectrum and an adjusted accompaniment spectrum;
calculating an accompaniment binary mask from the audio data; and
processing the adjusted singing voice spectrum and the adjusted accompaniment spectrum using the accompaniment binary mask, to obtain accompaniment data and singing voice data.

### Claim 15 (depends on 14)

The method according to claim 14, wherein the adjusting comprises:
calculating a singing voice binary mask according to the singing voice spectrum and the accompaniment spectrum,
wherein the overall spectrum is adjusted using the singing voice binary mask to obtain the adjusted signing voice spectrum and the adjusted accompaniment spectrum.

### Claim 16 (depends on 14)

The method according to claim 14, wherein the calculating comprises:
performing independent component analysis (ICA) on the audio data, to obtain initial singing voice data and initial accompaniment data; and
calculating the accompaniment binary mask according to the initial signing voice data and the initial accompaniment data.

### Claim 17 (depends on 16)

The method according to claim 16, wherein the calculating the accompaniment binary mask according to the initial singing voice data and the initial accompaniment data comprises:
performing mathematical transformation on the initial singing voice data and the initial accompaniment data, to obtain a transformed singing voice spectrum and a transformed accompaniment spectrum; and
calculating the accompaniment binary mask according to the transformed singing voice spectrum and the transformed accompaniment spectrum.