# US 2022/0286781 A1 · Method and apparatus for listening scene construction and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0286781 A1
- Document ID: US20220286781A1
- Status: Published application
- Kind: A1
- Application number: 17/751,960
- Filed: 2022-05-24
- Published: 2022-09-08
- Earliest claimed: 2019-11-25
- Assignees: Tencent Music Entertainment Technology (Shenzhen) Co., Ltd.
- Applicants: Tencent Music Entertainment Technology (Shenzhen) Co., Ltd.
- Inventors: Zhenhai YAN
- Web page: https://patentgrep.com/patent/US20220286781A1

## CPC

- H04R 5/04: Circuit arrangements, {e.g. for selective connection of amplifier inputs/outputs to loudspeakers, for loudspeaker detection, or for adaptation of settings to personal preferences or hearing impairments (combinations of amplifiers H03F3/68; stereophonic systems H04S)}
- H04R 5/033: Headphones for stereophonic communication {(details thereof, e.g. relating to batteries, cables or control elements H04R1/10)}
- H04S 1/005: {For headphones}
- H04S 7/301: {Automatic calibration of stereophonic sound system, e.g. with test microphone}
- H04S 7/302: {Electronic adaptation of stereophonic sound system to listener position or orientation (H04S7/301 takes precedence)}
- H04S 2420/01: Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)
- H04S: STEREOPHONIC SYSTEMS

## Abstract

A method and an apparatus for virtual listening scene construction and a storage medium are provided. The method includes the following. Target audio is determined, where the target audio is used to characterize a sound feature in a target scene. A position of a sound source of the target audio is determined. Dual-channel audio of the target audio is obtained by performing audio-visual modulation on the target audio according to the position of the sound source, where the dual-channel audio of the target audio during simultaneous output is able to produce an effect that the target audio is from the position of the sound source. The dual-channel audio of the target audio is rendered into target music to produce an effect that the target music is played in the target scene.

## Claims

### Claim 1

A method for listening scene construction, comprising:
determining target audio, the target audio being used to characterize a sound feature in a target scene;
determining a position of a sound source of the target audio;
obtaining dual-channel audio of the target audio by performing audio-visual modulation on the target audio according to the position of the sound source, the dual-channel audio of the target audio during simultaneous output being able to produce an effect that the target audio is from the position of the sound source; and
rendering the dual-channel audio of the target audio into target music to produce an effect that the target music is played in the target scene.

### Claim 2 (depends on 1)

The method of claim 1, wherein
the target audio before a vocal part of the target music occurs or after the vocal part ends is audio matched according to type information or whole lyrics of the target music; and/or
the target audio in the vocal part of the target music is audio matched according to a lyric content of the target music.

### Claim 3 (depends on 1)

The method of claim 1, wherein
determining the position of the sound source of the target audio comprises:
determining a position of the sound source of the target audio at each of a plurality of time nodes; and
obtaining the dual-channel audio of the target audio by performing the audio-visual modulation on the target audio according to the position of the sound source comprises:
obtaining the dual-channel audio of the target audio by performing the audio-visual modulation on the target audio according to the position of the sound source at each of the plurality of time nodes.

### Claim 4 (depends on 1)

The method of claim 1, wherein obtaining the dual-channel audio of the target audio by performing the audio-visual modulation on the target audio according to the position of the sound source comprises:
dividing the target audio into a plurality of audio frames; and
obtaining the dual-channel audio of the target audio by convolving a head-related transfer function (HRTF) from a position of the sound source to a left ear and a right ear respectively for each of the plurality of audio frames according to the position of the sound source corresponding to the audio frame.

### Claim 5 (depends on 4)

The method of claim 4, wherein obtaining the dual-channel audio of the target audio by convolving the HRTF from the position of the sound source to the left ear and the right ear respectively for each of the plurality of audio frames according to the position of the sound source corresponding to the audio frame comprises:
obtaining a first position of the sound source corresponding to a first audio frame, the first audio frame being any one of the plurality of audio frames;
determining a first HRTF corresponding to the first position on condition that the first position falls within a preset measuring point range, each measuring point in the preset measuring point range corresponding to an HRTF; and
obtaining dual-channel audio of the first audio frame of the target audio by convolving the first HRTF from the first position to the left ear and the right ear respectively for the first audio frame.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
determining P measuring position points according to the first position on condition that the first position falls out of the preset measuring point range, the P measuring position points being P points falling within the preset measuring point range, P being an integer not less than 1;
obtaining a second HRTF corresponding to the first position by fitting according to HRTFs respectively corresponding to the P measuring position points; and
obtaining the dual-channel audio of the first audio frame of the target audio by convolving the second HRTF from the first position to the left ear and the right ear respectively for the first audio frame.

### Claim 7 (depends on 1)

The method of claim 1, wherein
the dual-channel audio of the target audio comprises left channel audio and right channel audio; and
rendering the dual-channel audio of the target audio into the target music comprises:
determining a modulation factor according to a root mean square (RMS) value of the left channel audio, an RMS value of the right channel audio, and an RMS value of the target music;
obtaining adjusted left channel audio and adjusted right channel audio by adjusting the RMS value of the left channel audio and the RMS value of the right channel audio according to the modulation factor, an RMS value of the adjusted left channel audio and an RMS value of the adjusted right channel audio each being not greater than the RMS value of the target music; and
mixing the adjusted left channel audio into a left channel of the target music as rendered audio of the left channel of the target music, and mixing the adjusted right channel audio into a right channel of the target music as rendered audio of the right channel of the target music.

### Claim 8 (depends on 7)

The method of claim 7, wherein
the RMS value of the left channel audio before adjustment is RMSA1, the RMS value of the right channel audio before adjustment is RMSB1, and the RMS value of the target music is RMSY; and
determining the modulation factor according to the RMS value of the left channel audio, the RMS value of the right channel audio, and the RMS value of the target music comprises:
adjusting the RMS value of the left channel audio to RMSA2, and adjusting the RMS value of the right channel audio to RMSB2, such that RMSA2, RMSB2, and RMSY satisfy:
RMSA2=alpha*RMSY; and
RMSB2=alpha*RMSY, alpha being a preset scale factor, and 0<alpha<1;
assigning a ratio of RMSA2 to RMSA1 as a first left channel modulation factor MA1, that is,
M
A
⁢
1
=
R
⁢
M
⁢
S
A
⁢
2
R
⁢
M
⁢
S
A
⁢
1
;
assigning a ratio of RMSB2 to RMSB1 as a first right channel modulation factor MB1, that is,
M
B
⁢
1
=
R
⁢
M
⁢
S
B
⁢
2
R
⁢
M
⁢
S
B
⁢
1
;
assigning a smaller value of MA1and MB1 as a first group value M1, that is, M1=min (MA1, MB1); and
determining the first group value as the modulation factor.

### Claim 9 (depends on 8)

The method of claim 8, wherein determining the modulation factor according to the RMS value of the left channel audio, the RMS value of the right channel audio, and the RMS value of the target music further comprises:
adjusting the RMS value of the left channel audio to RMSA3, and adjusting the RMS value of the right channel audio to RMSB3, such that RMSA3, RMSB3, and RMSY satisfy:
RMSA3=F−RMSY, F being a maximum number of numbers that a floating-point type is able to represent; and
RMSB3=F−RMSY;
assigning a ratio of RMSA3 to RMSA1 as a second left channel modulation factor MA2, that is,
M
A
⁢
2
=
R
⁢
M
⁢
S
A
⁢
3
R
⁢
M
⁢
S
A
⁢
1
;
and
assigning a ratio of RMSB3 to RMSB1 as a second right channel modulation factor MB2, that is,
M
B
⁢
2
=
R
⁢
M
⁢
S
B
⁢
3
R
⁢
M
⁢
S
B
⁢
1
;
and
assigning a smaller value of MA2 and MB2 as a second group value M2, that is, M2=min (MA2, MB2), the first group value being less than the second group value.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
after determining the target audio and before determining the position of the sound source of the target audio,
transferring a sampling rate of the target audio to a sampling rate of the target music on condition that the sampling rate of the target audio is different from the sampling rate of the target music.

### Claim 11

An apparatus for listening scene construction, comprising:
a memory configured to store computer programs;
a processor configured to invoke the computer programs to:
determine target audio, the target audio being used to characterize a sound feature in a target scene;
determine a position of a sound source of the target audio;
obtain dual-channel audio of the target audio by performing audio-visual modulation on the target audio according to the position of the sound source, the dual-channel audio of the target audio during simultaneous output being able to produce an effect that the target audio is from the position of the sound source; and
render the dual-channel audio of the target audio into target music to produce an effect that the target music is played in the target scene.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein
the target audio before a vocal part of the target music occurs or after the vocal part ends is audio matched according to type information or whole lyrics of the target music; and/or
the target audio in the vocal part of the target music is audio matched according to a lyric content of the target music.

### Claim 13 (depends on 11)

The apparatus of claim 11, wherein
the processor configured to determine the position of the sound source of the target audio is specifically configured to:
determine a position of the sound source of the target audio at each of a plurality of time nodes; and
the processor configured to obtain the dual-channel audio of the target audio by performing the audio-visual modulation on the target audio according to the position of the sound source is specifically configured to:
obtain the dual-channel audio of the target audio by performing the audio-visual modulation on the target audio according to the position of the sound source at each of the plurality of time nodes.

### Claim 14 (depends on 11)

The apparatus of claim 11, wherein the processor configured to obtain the dual-channel audio of the target audio by performing the audio-visual modulation on the target audio according to the position of the sound source is configured to:
divide the target audio into a plurality of audio frames; and
obtain the dual-channel audio of the target audio by convolving a head-related transfer function (HRTF) from a position of the sound source to a left ear and a right ear respectively for each of the plurality of audio frames according to the position of the sound source corresponding to the audio frame.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein the processor configured to obtain the dual-channel audio of the target audio by convolving the HRTF from the position of the sound source to the left ear and the right ear respectively for each of the plurality of audio frames according to the position of the sound source corresponding to the audio frame is configured to:
obtain a first position of the sound source corresponding to a first audio frame, the first audio frame being one of the plurality of audio frames;
determine a first HRTF corresponding to the first position on condition that the first position falls within a preset measuring point range, each measuring point in the preset measuring point range corresponding to an HRTF; and
obtain dual-channel audio of the first audio frame of the target audio by convolving the first HRTF from the first position to the left ear and the right ear respectively for the first audio frame.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the processor is further configured to:
determine P measuring position points according to the first position on condition that the first position falls out of the preset measuring point range, the P measuring position points being P points falling within the preset measuring point range, P being an integer not less than 1;
obtain a second HRTF corresponding to the first position by fitting according to HRTFs respectively corresponding to the P measuring position points; and
obtain the dual-channel audio of the first audio frame of the target audio by convolving the second HRTF from the first position to the left ear and the right ear respectively for the first audio frame.

### Claim 17 (depends on 11)

The apparatus of claim 11, wherein the processor configured to render the dual-channel audio of the target audio into the target music to produce the effect that the target music is played in the target scene specifically is configured to:
determine a modulation factor according to a root mean square (RMS) value of the left channel audio, an RMS value of the right channel audio, and an RMS value of the target music;
obtain adjusted left channel audio and adjusted right channel audio by adjusting the RMS value of the left channel audio and the RMS value of the right channel audio according to the modulation factor, an RMS value of the adjusted left channel audio and an RMS value of the adjusted right channel audio each being not greater than the RMS value of the target music; and
mix the adjusted left channel audio into a left channel of the target music as rendered audio of the left channel of the target music, and mix the adjusted right channel audio into a right channel of the target music as rendered audio of the right channel of the target music.

### Claim 18 (depends on 17)

The apparatus of claim 17, wherein
the RMS value of the left channel audio before adjustment is RMSA1, the RMS value of the right channel audio before adjustment is RMSB1, and the RMS value of the target music is RMSY; and
the processor configured to determine the modulation factor according to the RMS value of the left channel audio, the RMS value of the right channel audio, and the RMS value of the target music is specifically configured to:
adjust the RMS value of the left channel audio to RMSA2, and adjust the RMS value of the right channel audio to RMSB2, such that RMSA2, RMSB2, and RMSY satisfy:
RMSA2=alpha*RMSY; and
RMSB2=alpha* RMSY, alpha being a preset scale factor, and 0<alpha<1;
assign a ratio of RMSA2 to RMSA1 as a first left channel modulation factor MA1, that is,
M
A
⁢
1
=
R
⁢
M
⁢
S
A
⁢
2
R
⁢
M
⁢
S
A
⁢
1
;
assign a ratio of RMSB2 to RMSB1 as a first right channel modulation factor MB1, that is,
M
B
⁢
1
=
R
⁢
M
⁢
S
B
⁢
2
R
⁢
M
⁢
S
B
⁢
1
;
assign a smaller value of MA1 and MB1 as a first group value M1, that is, M1=min (MA1, MB1); and
determine the first group value as the modulation factor.

### Claim 19 (depends on 18)

The apparatus of claim 18, wherein the processor configured to determine the modulation factor according to the RMS value of the left channel audio, the RMS value of the right channel audio, and the RMS value of the target music is further configured to:
adjust the RMS value of the left channel audio to RMSA3, and adjust the RMS value of the right channel audio to RMSB3, such that RMSA3, RMSB3, and RMSY satisfy:
RMSA3=F−RMSY, F being a maximum number of numbers that a floating-point type is able to represent; and
RMSB3=F−RMSY;
assign a ratio of RMSA3 to RMSA1 as a second left channel modulation factor MA2, that is,
M
A
⁢
2
=
R
⁢
M
⁢
S
A
⁢
3
R
⁢
M
⁢
S
A
⁢
1
;
assign a ratio of RMSB3 to RMSB1 as a second right channel modulation factor MB2, that is
M
B
⁢
2
=
R
⁢
M
⁢
S
B
⁢
3
R
⁢
M
⁢
S
B
⁢
1
;
and
assign a smaller value of MA2 and MB2 as a second group value M2, that is, M2=min (MA2, MB2) the first group value being less than the second group value.

### Claim 20 (depends on 1)

A non-volatile computer storage medium comprising computer programs which, when running on an electronic device, are operable with the electronic device to perform the method of claim 1.
