# US 10,873,826 B2 · Binaural rendering apparatus and method for playing back of multiple audio sources > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,873,826 B2 - Document ID: US10873826B2 - Status: Granted - Kind: B2 - Application number: 16/913,034 - Filed: 2020-06-26 - Published: 2020-10-15 - Granted: 2020-12-22 - Earliest claimed: 2016-10-28 - Assignees: Panasonic Intellectual Property Corporation of America - Applicants: Panasonic Intellectual Property Corporation of America - Inventors: Hiroyuki Ehara; Kai Wu; Sua Hong Neo - Web page: https://patentgrep.com/patent/US10873826B2 ## CPC - H04S 7/304: {For headphones} - G10L 19/008: Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing - H04S 1/005: {For headphones} - H04S 7/305: {Electronic adaptation of stereophonic audio signals to reverberation of the listening space (H04S7/301 takes precedence)} - H04S 2400/01: Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved - H04S 2420/01: Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD] - H04S: STEREOPHONIC SYSTEMS - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A method of generating binaural headphone playback signals given multiple audio source signals with an associated metadata and binaural room impulse response (BRIR) database is provided, wherein the audio source signals can be channel-based, object-based, or a mixture of both signals. The method includes grouping the audio source signals according to positions of the audio sources, parameterizing BRIR to be used for rendering, and dividing each audio source signal to be rendered into a number of blocks and frames. The method also includes averaging the parameterized BRIR sequences, downmixing the divided audio source signals using the diffuse blocks of BRIRs, and performing late reverberation processing on the downmixed version of the previous blocks of the audio source signals. ## Claims ### Claim 1 A method of generating binaural headphone playback signals given multiple audio source signals with an associated metadata and binaural room impulse response (BRIR) database, wherein the audio source signals can be channel-based, object-based, or a mixture of both signals, the method comprising: grouping the audio source signals according to positions of the audio sources; parameterizing BRIR to be used for rendering; dividing each audio source signal to be rendered into a number of blocks and frames; averaging the parameterized BRIR sequences; downmixing the divided audio source signals using the diffuse blocks of BRIRs; and performing late reverberation processing on the downmixed version of the previous blocks of the audio source signals, wherein, the late reverberation processing γ( ) of the previous blocks is a multiplication processing in the frequency domain of the average signal of K from the current to w blocks before (current-w) and the wth block of BRIR of hθave, the output of the late reverberation y(current-w) is denoted by Equation 1, y ( current - w ) = γ ⁡ ( 1 K ⁢ ∑ k = 1 K ⁢ ⁢ s k ( current - w ) ⁡ ( n ) , h θ ave ( w ) ⁡ ( n ) ) [ Equation ⁢ ⁢ 1 ] Current: index of current block W: index of diffuse blocks n: sample index (n=0, 1, 2, . . . , n) K: audio source (k=1, 2, . . . , k) Sk(current-w): current block of the kth source signal θave: averaged location of all the K sources hθave(w)(n): average of the diffuse blocks. ### Claim 2 (depends on 1) The method according to claim 1, wherein the audio-source position is computed for each time frame/block of the audio source signals given the source metadata and user head tracking data. ### Claim 3 (depends on 1) The method according to claim 1, wherein each BRIR filter signal in the BRIR database is divided into a direct block including a few frames and a number of diffuse blocks, and the frames and blocks are labelled using the target location of that BRIR filter signal. ### Claim 4 (depends on 1) The method according to claim 1, wherein the audio source signal is divided into the current block and a number of previous blocks, and the current block is further divided into a number of frames. ### Claim 5 (depends on 1) The method according to claim 1, wherein frame-by-frame binauralization processing is performed for the frames of the current block of the audio source signals using the selected BRIR frames, and the selection of each BRIR frame is based on searching for the nearest labelled BRIR frame which is closest to the computed position of each source. ### Claim 6 (depends on 1) The method according to claim 1, wherein frame-by-frame binauralization processing is performed with an incorporation of an audio source signal downmix module, such that the audio source signals can be downmixed according to the computed source grouping decision, and the binauralization processing is applied on that downmixed signal to reduce computational complexity. ### Claim 7 (depends on 1) The method according to claim 1, wherein calculating different cut-off frequencies for each block and the late reverberation processing are not performed on a downmixed version of the previous blocks above the cutoff frequencies. ### Claim 8 An integrated circuit (IC) for generating binaural headphone playback signals given multiple audio source signals with an associated metadata and binaural room impulse response (BRIR) database, wherein the audio source signals can be channel-based, object-based, or a mixture of both signals, the method comprising: one or more processors; and one or more memories, the integrated circuit configured to execute operations, including grouping the audio source signals according to positions of the audio sources; parameterizing BRIR to be used for rendering; dividing each audio source signal to be rendered into a number of blocks and frames; averaging the parameterized BRIR sequences; downmixing the divided audio source signals using the diffuse blocks of BRIRs; and performing late reverberation processing on the downmixed version of the previous blocks of the audio source signals, wherein, the late reverberation processing γ( ) of the previous blocks is a multiplication processing in the frequency domain of the average signal of K from the current to w blocks before (current-w) and the wth block of BRIR of hθave, the output of the late reverberation y(current-w) is denoted by Equation 1, y ( current - w ) = γ ⁡ ( 1 K ⁢ ∑ k = 1 K ⁢ ⁢ s k ( current - w ) ⁡ ( n ) , h θ ave ( w ) ⁡ ( n ) ) [ Equation ⁢ ⁢ 1 ] Current: index of current block W: index of diffuse blocks n: sample index (n=0, 1, 2, . . . , n) K: audio source (k=1, 2, . . . , k) Sk(current-w): current block of the kth source signal θave: averaged location of all the K sources hθave(w)(n): average of the diffuse blocks. ### Claim 9 (depends on 8) The integrated circuit according to claim 8, wherein the audio-source position is computed for each time frame/block of the audio source signals given the source metadata and user head tracking data. ### Claim 10 (depends on 8) The integrated circuit according to claim 8, wherein each BRIR filter signal in the BRIR database is divided into a direct block including a few frames and a number of diffuse blocks, and the frames and blocks are labelled using the target location of that BRIR filter signal. ### Claim 11 (depends on 8) The integrated circuit according to claim 8, wherein the audio source signal is divided into the current block and a number of previous blocks, and the current block is further divided into a number of frames. ### Claim 12 (depends on 8) The integrated circuit according to claim 8, wherein frame-by-frame binauralization processing is performed for the frames of the current block of the audio source signals using the selected BRIR frames, and the selection of each BRIR frame is based on searching for the nearest labelled BRIR frame which is closest to the computed position of each source. ### Claim 13 (depends on 8) The integrated circuit according to claim 8, wherein frame-by-frame binauralization processing is performed with an incorporation of an audio source signal downmix module, such that the audio source signals can be downmixed according to the computed source grouping decision, and the binauralization processing is applied on that downmixed signal to reduce computational complexity. ### Claim 14 (depends on 8) The integrated circuit according to claim 8, wherein calculating different cut-off frequencies for each block and the late reverberation processing are not performed on a downmixed version of the previous blocks above the cutoff frequencies.