# US 2021/0176584 A1 · Audio Processing Method and Apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0176584 A1
- Document ID: US20210176584A1
- Status: Published application
- Kind: A1
- Application number: 17/179,723
- Filed: 2021-02-19
- Published: 2021-06-10
- Earliest claimed: 2018-08-20
- Assignees: Huawei Technologies Co., Ltd.
- Applicants: Huawei Technologies Co., Ltd.
- Inventors: Cal Armstrong; Gavin Kearney; Bin Wang; Zexin Liu
- Web page: https://patentgrep.com/patent/US20210176584A1

## CPC

- H04S 7/302: {Electronic adaptation of stereophonic sound system to listener position or orientation (H04S7/301 takes precedence)}
- H04S 7/303: {Tracking of listener position or orientation}
- H04S 2400/01: Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
- H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence)
- H04S 2420/01: Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
- H04S 2420/11: Application of ambisonics in stereophonic audio systems
- H04S: STEREOPHONIC SYSTEMS

## Abstract

An audio processing method includes: M first audio signals are obtained by processing a to-be-processed audio signal by M first virtual speakers; N second audio signals are obtained by processing the to-be-processed audio signal by N second virtual speakers; M first head-related transfer functions (HRTFs) centered at a left ear position and N second HRTFs centered at a right ear position are obtained; a first target audio signal is obtained based on the M first audio signals and the M first HRTFs; and a second target audio signal is obtained based on the N second audio signals and the N second HRTFs.

## Claims

### Claim 1

An audio processing method, comprising:
obtaining M first audio signals by processing a to-be-processed audio signal by M first virtual speakers, and N second audio signals by processing the to-be-processed audio signal by N second virtual speakers, wherein the M first virtual speakers are in a one-to-one correspondence with the M first audio signals, the N second virtual speakers are in a one-to-one correspondence with the N second audio signals, and M and N are positive integers;
obtaining M first head-related transfer functions (HRTFs) and N second HRTFs, wherein all the M first HRTFs are centered at a left ear position, all the N second HRTFs are centered at a right ear position, the M first HRTFs are in a one-to-one correspondence with the M first virtual speakers, and the N second HRTFs are in a one-to-one correspondence with the N second virtual speakers; and
obtaining a first target audio signal based on the M first audio signals and the M first HRTFs, and obtaining a second target audio signal based on the N second audio signals and the N second HRTFs.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the obtaining a first target audio signal based on the M first audio signals and the M first HRTFs comprises:
convolving each of the M first audio signals with a corresponding first HRTF, to obtain M first convolved audio signals; and
obtaining the first target audio signal based on the M first convolved audio signals; or
wherein the obtaining a second target audio signal based on the N second audio signals and the N second HRTFs comprises:
convolving each of the N second audio signals with a corresponding second HRTF, to obtain N second convolved audio signals; and
obtaining the second target audio signal based on the N second convolved audio signals.

### Claim 3 (depends on 1)

The method according to claim 1, wherein correspondences between a plurality of preset positions and a plurality of HRTFs are prestored, and the obtaining M first HRTFs comprises:
obtaining M first positions of the M first virtual speakers relative to the current left ear position; and
determining, based on the M first positions and the correspondences, that M HRTFs corresponding to the M first positions are the M first HRTFs; or
the obtaining N second HRTFs comprises:
obtaining N second positions of the N second virtual speakers relative to the current right ear position; and
determining, based on the N second positions and the correspondences, that N HRTFs corresponding to the N second positions are the N second HRTFs.

### Claim 4 (depends on 1)

The method according to claim 1, wherein correspondences between a plurality of preset positions and a plurality of HRTFs are prestored, and the obtaining M first HRTFs comprises:
obtaining M third positions of the M first virtual speakers relative to a current head center, wherein the third position comprises a first azimuth and a first elevation of the first virtual speaker relative to the current head center, and comprises a first distance between the current head center and the first virtual speaker;
determining M fourth positions based on the M third positions, wherein the M third positions are in a one-to-one correspondence with the M fourth positions, one fourth position and a corresponding third position comprise a same elevation and a same distance, and a difference between an azimuth comprised in the one fourth position and a first value is a first azimuth comprised in the corresponding third position; and the first value is a difference between a first included angle and a second included angle, the first included angle is an included angle between a first straight line and a first plane, the second included angle is an included angle between a second straight line and the first plane, the first straight line is a straight line that passes through the current left ear and a coordinate origin of a three-dimensional coordinate system, the second straight line is a straight line that passes through the current head center and the coordinate origin, and the first plane is a plane constituted by an X axis and a Z axis of the three-dimensional coordinate system; and
determining, based on the M fourth positions and the correspondences, that M HRTFs corresponding to the M fourth positions are the M first HRTFs; or
the obtaining N second HRTFs comprises:
obtaining N fifth positions of the N second virtual speakers relative to the current head center, wherein the fifth position comprises a second azimuth and a second elevation of the second virtual speaker relative to the current head center, and comprises a second distance between the current head center and the second virtual speaker;
determining N sixth positions based on the N fifth positions, wherein the N fifth positions are in a one-to-one correspondence with the N sixth positions, one sixth position and a corresponding fifth position comprise a same elevation and a same distance, and a sum of an azimuth comprised in the one sixth position and a second value is a second azimuth comprised in the corresponding fifth position; and the second value is a difference between a third included angle and a second included angle, the second included angle is an included angle between a second straight line and a first plane, the third included angle is an included angle between a third straight line and the first plane, the second straight line is the straight line that passes through the current head center and the coordinate origin, the third straight line is a straight line that passes through the current right ear and the coordinate origin, and the first plane is the plane constituted by the X axis and the Z axis of the three-dimensional coordinate system; and
determining, based on the N sixth positions and the correspondences, that N HRTFs corresponding to the N sixth positions are the N second HRTFs.

### Claim 5 (depends on 1)

The method according to claim 1, wherein correspondences between a plurality of preset positions and a plurality of HRTFs are prestored, and the obtaining M first HRTFs comprises:
obtaining M third positions of the M first virtual speakers relative to a current head center, wherein the third position comprises a first azimuth and a first elevation of the first virtual speaker relative to the current head center, and comprises a first distance between the current head center and the first virtual speaker;
determining M seventh positions based on the M third positions, wherein the M third positions are in a one-to-one correspondence with the M seventh positions, one seventh position and a corresponding third position comprise a same elevation and a same distance, and a difference between an azimuth comprised in the one seventh position and a first preset value is a first azimuth comprised in the corresponding third position; and
determining, based on the M seventh positions and the correspondences, that M HRTFs corresponding to the M seventh positions are the M first HRTFs; or
the obtaining N second HRTFs comprises:
obtaining N fifth positions of the N second virtual speakers relative to the current head center, wherein the fifth position comprises a second azimuth and a second elevation of the second virtual speaker relative to the current head center, and comprises a second distance between the current head center and the second virtual speaker;
determining N eighth positions based on the N fifth positions, wherein the N fifth positions are in a one-to-one correspondence with the N eighth positions, one eighth position and a corresponding fifth position comprise a same elevation and a same distance, and a sum of an azimuth comprised in the one eighth position and the first preset value is a second azimuth comprised in the corresponding fifth position; and
determining, based on the N eighth positions and the correspondences, that N HRTFs corresponding to the N eighth positions are the N second HRTFs.

### Claim 6 (depends on 1)

The method according to claim 1, wherein before the obtaining M first audio signals by processing a to-be-processed audio signal by M first virtual speakers, the method further comprises:
obtaining a target virtual speaker group, wherein the target virtual speaker group comprises M target virtual speakers, and the M target virtual speakers are in a one-to-one correspondence with the M first virtual speakers; and
determining M tenth positions of the M first virtual speakers relative to the coordinate origin of the three-dimensional coordinate system based on M ninth positions of the M target virtual speakers relative to the coordinate origin, wherein the M ninth positions are in a one-to-one correspondence with the M tenth positions, one tenth position and a corresponding ninth position comprise a same elevation and a same distance, and a difference between an azimuth comprised in the one tenth position and a second preset value is an azimuth comprised in the corresponding ninth position; and
the obtaining M first audio signals by processing a to-be-processed audio signal by M first virtual speakers comprises: processing the to-be-processed audio signal based on the M tenth positions, to obtain the M first audio signals.

### Claim 7 (depends on 1)

The method according to claim 1, wherein M=N, and before the obtaining N second audio signals by processing the to-be-processed audio signal by N second virtual speakers, the method further comprises:
obtaining a target virtual speaker group, wherein the target virtual speaker group comprises M target virtual speakers, and the M target virtual speakers are in a one-to-one correspondence with the N second virtual speakers; and
determining N eleventh positions of the N second virtual speakers relative to the coordinate origin of the three-dimensional coordinate system based on M ninth positions of the M target virtual speakers relative to the coordinate origin, wherein the M ninth positions are in a one-to-one correspondence with the N eleventh positions, one eleventh position and a corresponding ninth position comprise a same elevation and a same distance, and a sum of an azimuth comprised in the one eleventh position and a second preset value is an azimuth comprised in the corresponding ninth position; and
the obtaining N second audio signals by processing the to-be-processed audio signal by N second virtual speakers comprises: processing the to-be-processed audio signal based on the N eleventh positions, to obtain the N second audio signals.

### Claim 8 (depends on 1)

The method according to claim 1, wherein the M first virtual speakers are speakers in a first speaker group, the N second virtual speakers are speakers in a second speaker group, and the first speaker group and the second speaker group are two independent speaker groups; or
the M first virtual speakers are speakers in a first speaker group, the N second virtual speakers are speakers in a second speaker group, and the first speaker group and the second speaker group are a same speaker group, wherein M=N.

### Claim 9

An audio processing apparatus, comprising:
at least one processor; and
a memory storing computer executable instructions for execution by the at least one processor, wherein the computer executable instructions instruct the at least one processor to:
obtain M first audio signals by processing a to-be-processed audio signal by M first virtual speakers, and N second audio signals by processing the to-be-processed audio signal by N second virtual speakers, wherein the M first virtual speakers are in a one-to-one correspondence with the M first audio signals, the N second virtual speakers are in a one-to-one correspondence with the N second audio signals, and M and N are positive integers; and
obtain M first head-related transfer functions (HRTFs) and N second HRTFs, wherein all the M first HRTFs are centered at a left ear position, all the N second HRTFs are centered at a right ear position, the M first HRTFs are in a one-to-one correspondence with the M first virtual speakers, and the N second HRTFs are in a one-to-one correspondence with the N second virtual speakers; and obtain a first target audio signal based on the M first audio signals and the M first HRTFs, and obtain a second target audio signal based on the N second audio signals and the N second HRTFs.

### Claim 10 (depends on 9)

The apparatus according to claim 9, wherein the computer executable instructions further instruct the at least one processor to:
convolve each of the M first audio signals with a corresponding first HRTF, to obtain M first convolved audio signals; and
obtain the first target audio signal based on the M first convolved audio signals; or
convolve each of the N second audio signals with a corresponding second HRTF, to obtain N second convolved audio signals; and
obtain the second target audio signal based on the N second convolved audio signals.

### Claim 11 (depends on 9)

The apparatus according to claim 9, wherein the computer executable instructions further instruct the at least one processor to:
obtain M first positions of the M first virtual speakers relative to the current left ear position; and
determine, based on the M first positions and correspondences, that M HRTFs corresponding to the M first positions are the M first HRTFs, wherein the correspondences are prestored correspondences between a plurality of preset positions and a plurality of HRTFs; or
obtain N second positions of the N second virtual speakers relative to the current right ear position; and
determine, based on the N second positions and correspondences, that N HRTFs corresponding to the N second positions are the N second HRTFs, wherein the correspondences are prestored correspondences between a plurality of preset positions and a plurality of HRTFs.

### Claim 12 (depends on 9)

The apparatus according to claim 9, wherein the computer executable instructions further instruct the at least one processor to:
obtain M third positions of the M first virtual speakers relative to a current head center, wherein the third position comprises a first azimuth and a first elevation of the first virtual speaker relative to the current head center, and comprises a first distance between the current head center and the first virtual speaker;
determine M fourth positions based on the M third positions, wherein the M third positions are in a one-to-one correspondence with the M fourth positions, one fourth position and a corresponding third position comprise a same elevation and a same distance, and a difference between an azimuth comprised in the one fourth position and a first value is a first azimuth comprised in the corresponding third position; and the first value is a difference between a first included angle and a second included angle, the first included angle is an included angle between a first straight line and a first plane, the second included angle is an included angle between a second straight line and the first plane, the first straight line is a straight line that passes through the current left ear and a coordinate origin of a three-dimensional coordinate system, the second straight line is a straight line that passes through the current head center and the coordinate origin, and the first plane is a plane constituted by an X axis and a Z axis of the three-dimensional coordinate system; and
determine, based on the M fourth positions and correspondences, that M HRTFs corresponding to the M fourth positions are the M first HRTFs, wherein the correspondences are prestored correspondences between a plurality of preset positions and a plurality of HRTFs; or
obtain N fifth positions of the N second virtual speakers relative to the current head center, wherein the fifth position comprises a second azimuth and a second elevation of the second virtual speaker relative to the current head center, and comprises a second distance between the current head center and the second virtual speaker;
determine N sixth positions based on the N fifth positions, wherein the N fifth positions are in a one-to-one correspondence with the N sixth positions, one sixth position and a corresponding fifth position comprise a same elevation and a same distance, and a sum of an azimuth comprised in the one sixth position and a second value is a second azimuth comprised in the corresponding fifth position; and the second value is a difference between a third included angle and a second included angle, the second included angle is an included angle between a second straight line and a first plane, the third included angle is an included angle between a third straight line and the first plane, the second straight line is the straight line that passes through the current head center and the coordinate origin, the third straight line is a straight line that passes through the current right ear and the coordinate origin, and the first plane is the plane constituted by the X axis and the Z axis of the three-dimensional coordinate system; and
determine, based on the N sixth positions and correspondences, that N HRTFs corresponding to the N sixth positions are the N second HRTFs, wherein the correspondences are prestored correspondences between a plurality of preset positions and a plurality of HRTFs.

### Claim 13 (depends on 9)

The apparatus according to claim 9, wherein the computer executable instructions further instruct the at least one processor to:
obtain M third positions of the M first virtual speakers relative to a current head center, wherein the third position comprises a first azimuth and a first elevation of the first virtual speaker relative to the current head center, and comprises a first distance between the current head center and the first virtual speaker;
determine M seventh positions based on the M third positions, wherein the M third positions are in a one-to-one correspondence with the M seventh positions, one seventh position and a corresponding third position comprise a same elevation and a same distance, and a difference between an azimuth comprised in the one seventh position and a first preset value is a first azimuth comprised in the corresponding third position; and
determine, based on the M seventh positions and correspondences, that M HRTFs corresponding to the M seventh positions are the M first HRTFs, wherein the correspondences are prestored correspondences between a plurality of preset positions and a plurality of HRTFs; or
obtain N fifth positions of the N second virtual speakers relative to the current head center, wherein the fifth position comprises a second azimuth and a second elevation of the second virtual speaker relative to the current head center, and comprises a second distance between the current head center and the second virtual speaker;
determine N eighth positions based on the N fifth positions, wherein the N fifth positions are in a one-to-one correspondence with the N eighth positions, one eighth position and a corresponding fifth position comprise a same elevation and a same distance, and a sum of an azimuth comprised in the one eighth position and the first preset value is a second azimuth comprised in the corresponding fifth position; and
determine, based on the N eighth positions and correspondences, that N HRTFs corresponding to the N eighth positions are the N second HRTFs, wherein the correspondences are prestored correspondences between a plurality of preset positions and a plurality of HRTFs.

### Claim 14 (depends on 9)

The apparatus according to claim 9, wherein the computer executable instructions further instruct the at least one processor to:
obtain a target virtual speaker group, wherein the target virtual speaker group comprises M target virtual speakers, and the M target virtual speakers are in a one-to-one correspondence with the M first virtual speakers; and
determine M tenth positions of the M first virtual speakers relative to the coordinate origin of the three-dimensional coordinate system based on M ninth positions of the M target virtual speakers relative to the coordinate origin, wherein the M ninth positions are in a one-to-one correspondence with the M tenth positions, one tenth position and a corresponding ninth position comprise a same elevation and a same distance, and a difference between an azimuth comprised in the one tenth position and a second preset value is an azimuth comprised in the corresponding ninth position; and
the processing module is specifically configured to process the to-be-processed audio signal based on the M tenth positions, to obtain the M first audio signals.

### Claim 15 (depends on 9)

The apparatus according to claim 9, wherein M=N, and the computer executable instructions further instruct the at least one processor to:
obtain a target virtual speaker group, wherein the target virtual speaker group comprises M target virtual speakers, and the M target virtual speakers are in a one-to-one correspondence with the N second virtual speakers; and
determine N eleventh positions of the N second virtual speakers relative to the coordinate origin of the three-dimensional coordinate system based on the M ninth positions of the M target virtual speakers relative to the coordinate origin, wherein the M ninth positions are in a one-to-one correspondence with the N eleventh positions, one eleventh position and a corresponding ninth position comprise a same elevation and a same distance, and a sum of an azimuth comprised in the one eleventh position and a second preset value is an azimuth comprised in the corresponding ninth position; and
the processing module is specifically configured to process the to-be-processed audio signal based on the N eleventh positions, to obtain the N second audio signals.

### Claim 16 (depends on 9)

The apparatus according to claim 9, wherein the M first virtual speakers are speakers in a first speaker group, the N second virtual speakers are speakers in a second speaker group, and the first speaker group and the second speaker group are two independent speaker groups; or
the M first virtual speakers are speakers in a first speaker group, the N second virtual speakers are speakers in a second speaker group, and the first speaker group and the second speaker group are a same speaker group, wherein M=N.

### Claim 17

A non-transitory computer-readable storage medium storing computer instructions, that when executed by one or more processors, cause the one or more processors to perform the steps of:
obtaining M first audio signals by processing a to-be-processed audio signal by M first virtual speakers, and N second audio signals by processing the to-be-processed audio signal by N second virtual speakers, wherein the M first virtual speakers are in a one-to-one correspondence with the M first audio signals, the N second virtual speakers are in a one-to-one correspondence with the N second audio signals, and M and N are positive integers;
obtaining M first head-related transfer functions (HRTFs) and N second HRTFs, wherein all the M first HRTFs are centered at a left ear position, all the N second HRTFs are centered at a right ear position, the M first HRTFs are in a one-to-one correspondence with the M first virtual speakers, and the N second HRTFs are in a one-to-one correspondence with the N second virtual speakers; and
obtaining a first target audio signal based on the M first audio signals and the M first HRTFs, and obtaining a second target audio signal based on the N second audio signals and the N second HRTFs.

### Claim 18 (depends on 17)

The non-transitory computer-readable storage medium according to claim 17, wherein the computer instructions further cause the one or more processors to perform the steps of:
convolving each of the M first audio signals with a corresponding first HRTF, to obtain M first convolved audio signals; and
obtaining the first target audio signal based on the M first convolved audio signals; or
convolving each of the N second audio signals with a corresponding second HRTF, to obtain N second convolved audio signals; and
obtaining the second target audio signal based on the N second convolved audio signals.

### Claim 19 (depends on 17)

The non-transitory computer-readable storage medium according to claim 17, wherein the computer instructions further cause the one or more processors to perform the steps of:
obtaining M first positions of the M first virtual speakers relative to the current left ear position; and
determining, based on the M first positions and correspondences between a plurality of preset positions and a plurality of HRTFs, that M HRTFs corresponding to the M first positions are the M first HRTFs; or
obtaining N second positions of the N second virtual speakers relative to the current right ear position; and
determining, based on the N second positions and the correspondences, that N HRTFs corresponding to the N second positions are the N second HRTFs.

### Claim 20 (depends on 17)

The non-transitory computer-readable storage medium according to claim 17, wherein the computer instructions further cause the one or more processors to perform the steps of:
obtaining M third positions of the M first virtual speakers relative to a current head center, wherein the third position comprises a first azimuth and a first elevation of the first virtual speaker relative to the current head center, and comprises a first distance between the current head center and the first virtual speaker;
determining M fourth positions based on the M third positions, wherein the M third positions are in a one-to-one correspondence with the M fourth positions, one fourth position and a corresponding third position comprise a same elevation and a same distance, and a difference between an azimuth comprised in the one fourth position and a first value is a first azimuth comprised in the corresponding third position; and the first value is a difference between a first included angle and a second included angle, the first included angle is an included angle between a first straight line and a first plane, the second included angle is an included angle between a second straight line and the first plane, the first straight line is a straight line that passes through the current left ear and a coordinate origin of a three-dimensional coordinate system, the second straight line is a straight line that passes through the current head center and the coordinate origin, and the first plane is a plane constituted by an X axis and a Z axis of the three-dimensional coordinate system; and
determining, based on the M fourth positions and correspondences between a plurality of preset positions and a plurality of HRTFs, that M HRTFs corresponding to the M fourth positions are the M first HRTFs; or
obtaining N fifth positions of the N second virtual speakers relative to the current head center, wherein the fifth position comprises a second azimuth and a second elevation of the second virtual speaker relative to the current head center, and comprises a second distance between the current head center and the second virtual speaker;
determining N sixth positions based on the N fifth positions, wherein the N fifth positions are in a one-to-one correspondence with the N sixth positions, one sixth position and a corresponding fifth position comprise a same elevation and a same distance, and a sum of an azimuth comprised in the one sixth position and a second value is a second azimuth comprised in the corresponding fifth position; and the second value is a difference between a third included angle and a second included angle, the second included angle is an included angle between a second straight line and a first plane, the third included angle is an included angle between a third straight line and the first plane, the second straight line is the straight line that passes through the current head center and the coordinate origin, the third straight line is a straight line that passes through the current right ear and the coordinate origin, and the first plane is the plane constituted by the X axis and the Z axis of the three-dimensional coordinate system; and
determining, based on the N sixth positions and the correspondences, that N HRTFs corresponding to the N sixth positions are the N second HRTFs.

### Claim 21 (depends on 17)

The non-transitory computer-readable storage medium according to claim 17, wherein the computer instructions further cause the one or more processors to perform the steps of:
obtaining M third positions of the M first virtual speakers relative to a current head center, wherein the third position comprises a first azimuth and a first elevation of the first virtual speaker relative to the current head center, and comprises a first distance between the current head center and the first virtual speaker;
determining M seventh positions based on the M third positions, wherein the M third positions are in a one-to-one correspondence with the M seventh positions, one seventh position and a corresponding third position comprise a same elevation and a same distance, and a difference between an azimuth comprised in the one seventh position and a first preset value is a first azimuth comprised in the corresponding third position; and
determining, based on the M seventh positions and correspondences between a plurality of preset positions and a plurality of HRTFs, that M HRTFs corresponding to the M seventh positions are the M first HRTFs; or
obtaining N fifth positions of the N second virtual speakers relative to the current head center, wherein the fifth position comprises a second azimuth and a second elevation of the second virtual speaker relative to the current head center, and comprises a second distance between the current head center and the second virtual speaker;
determining N eighth positions based on the N fifth positions, wherein the N fifth positions are in a one-to-one correspondence with the N eighth positions, one eighth position and a corresponding fifth position comprise a same elevation and a same distance, and a sum of an azimuth comprised in the one eighth position and the first preset value is a second azimuth comprised in the corresponding fifth position; and
determining, based on the N eighth positions and the correspondences, that N HRTFs corresponding to the N eighth positions are the N second HRTFs.