# US 2018/0027351 A1 · Optimized virtual scene layout for spatial meeting playback

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0027351 A1
- Document ID: US20180027351A1
- Status: Published application
- Kind: A1
- Application number: 15/546,576
- Filed: 2016-02-03
- Published: 2018-01-25
- Earliest claimed: 2015-02-03
- Assignees: Dolby Laboratories Licensing Corporation
- Applicants: Dolby Laboratories Licensing Corporation
- Inventors: Richard J. CARTWRIGHT; Hannes MUESCH
- Web page: https://patentgrep.com/patent/US20180027351A1

## CPC

- H04S 7/303: {Tracking of listener position or orientation}
- G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10)
- H04M 3/56: Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities (video conference systems H04N7/15)
- H04R 1/1016: {Earpieces of the intra-aural type}
- H04S 3/008: {in which the audio signals are in digital form, i.e. employing more than two discrete digital channels (data reduction aspects thereof based on psychoacoustics G10L19/02)}
- H04M 3/42221: {Conversation recording systems (at the subscriber's set H04M1/656)}
- H04M 3/568: {audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants (echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30)}
- H04R 2420/07: Applications of wireless loudspeakers or wireless microphones
- H04S 7/302: {Electronic adaptation of stereophonic sound system to listener position or orientation (H04S7/301 takes precedence)}
- H04S 2400/01: Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
- H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence)
- H04S: STEREOPHONIC SYSTEMS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)

## Abstract

Various disclosed implementations involve processing and/or playback of a recording of a conference involving a plurality of conference participants. Some implementations involve receiving or determining conversational dynamics data. One or more variables of a cost function may be based, at least in part, on the conversational dynamics data. The cost function may be a spatial optimization cost function of a vector describing a virtual conference participant position for each of the conference participants in a virtual acoustic space. The virtual acoustic space may be determined relative to a listener's head. The virtual conference participant positions may be assigned according to a solution of the cost function.

## Claims

### Claim 1

A method for processing audio data, the method comprising:
receiving audio data corresponding to a recording of a conference involving a plurality of conference participants, the audio data including at least one of: (a) audio data from multiple endpoints, the audio data for each of the multiple endpoints having been recorded separately or (b) audio data from a single endpoint corresponding to multiple conference participants and including spatial information for each conference participant of the multiple conference participants;
analyzing the audio data to determine conversational dynamics data that includes at least one data type selected from a list of data types consisting of: data indicating the frequency and duration of conference participant speech; data indicating instances of conference participant doubletalk during which at least two conference participants are speaking simultaneously; and data indicating instances of conference participant conversations;
applying the conversational dynamics data as one or more variables of a spatial optimization cost function of a vector describing a virtual conference participant position for each of the conference participants in a virtual acoustic space;
applying an optimization technique to the spatial optimization cost function to determine a locally optimal solution; and
assigning the virtual conference participant positions in the virtual acoustic space based, at least in part, on the locally optimal solution.

### Claim 2 (depends on 1)

The method of claim 1, wherein the conference is a teleconference.

### Claim 3 (depends on 1)

The method of claim 1, wherein the virtual acoustic space is determined relative to a position of a virtual listener's head in the virtual acoustic space and wherein the spatial optimization cost function applies a penalty for placing conference participants who are involved in conference participant doubletalk at virtual conference participant positions that are on, or within a predetermined angular distance from, a cone of confusion defined relative to the position of the virtual listener's head, circular conical slices through the cone of confusion having identical inter-aural time differences.

### Claim 4 (depends on 1)

The method of claim 1, wherein the virtual acoustic space is determined relative to a position of a virtual listener's head in the virtual acoustic space and wherein the spatial optimization cost function applies a penalty for placing conference participants who are involved in a conference participant conversation with one another at virtual conference participant positions that are on, or within a predetermined angular distance from, a cone of confusion defined relative to the position of the virtual listener's head, circular conical slices through the cone of confusion having identical inter-aural time differences.

### Claim 5 (depends on 1)

The method of claim 1, wherein analyzing the audio data also involves determining which conference participants, if any, have perceptually similar voices.

### Claim 6 (depends on 5)

The method of claim 5, wherein the virtual acoustic space is determined relative to a position of a virtual listener's head in the virtual acoustic space and wherein the spatial optimization cost function applies a penalty for placing conference participants with perceptually similar voices at virtual conference participant positions that are on, or within a predetermined angular distance from, a cone of confusion defined relative to the position of the virtual listener's head, circular conical slices through the cone of confusion having identical inter-aural time differences.

### Claim 7 (depends on 1)

The method of claim 1, wherein the virtual acoustic space is determined relative to a position of a virtual listener's head in the virtual acoustic space and wherein the spatial optimization cost function applies a penalty for placing conference participants who speak frequently at virtual conference participant positions that are beside, behind, above, or below the position of the virtual listener's head.

### Claim 8 (depends on 1)

The method of claim 1, wherein the virtual acoustic space is determined relative to a position of a virtual listener's head in the virtual acoustic space and wherein the spatial optimization cost function applies a penalty for placing conference participants who speak frequently at virtual conference participant positions that are farther from the position of the virtual listener's head than the virtual conference participant positions of conference participants who speak less frequently.

### Claim 9 (depends on 1)

The method of claim 1, wherein the virtual acoustic space is determined relative to a position of a virtual listener's head in the virtual acoustic space and wherein the spatial optimization cost function applies a penalty for placing conference participants who speak infrequently at virtual conference participant positions that are not beside, behind, above or below the position of the virtual listener's head.

### Claim 10 (depends on 1)

The method of claim 1, wherein the optimization technique involves at least one technique selected from a group of optimization techniques consisting of a gradient descent technique, conjugate gradient technique, Newton's method, the Broyden-Fletcher-Goldfarb-Shanno algorithm; a genetic algorithm, an algorithm for simulated annealing, an ant colony optimization method or a Monte Carlo method.

### Claim 11 (depends on 1)

The method of claim 1, wherein assigning a virtual conference participant position involves selecting a virtual conference participant position from a set of predetermined virtual conference participant positions.

### Claim 12 (depends on 1)

The method of claim 1, wherein the audio data includes output of a voice activity detection process.

### Claim 13 (depends on 1)

The method of claim 1, wherein analyzing the audio data involves identifying speech corresponding to individual conference participants.

### Claim 14 (depends on 1)

The method of claim 1, wherein the audio data corresponds to a recording of a complete or substantially complete conference.

### Claim 15

A non-transitory medium having software stored thereon, the software including instructions for processing audio data by controlling at least one device for:
receiving audio data corresponding to a recording of a conference involving a plurality of conference participants, the audio data including at least one of: (a) audio data from multiple endpoints, the audio data for each of the multiple endpoints having been recorded separately or (b) audio data from a single endpoint corresponding to multiple conference participants and including spatial information for each conference participant of the multiple conference participants;
analyzing the audio data to determine conversational dynamics data that includes at least one data type selected from a list of data types consisting of: data indicating the frequency and duration of conference participant speech; data indicating instances of conference participant doubletalk during which at least two conference participants are speaking simultaneously; and data indicating instances of conference participant conversations;
applying the conversational dynamics data as one or more variables of a spatial optimization cost function of a vector describing a virtual conference participant position for each of the conference participants in a virtual acoustic space;
applying an optimization technique to the spatial optimization cost function to determine a locally optimal solution; and
assigning the virtual conference participant positions in the virtual acoustic space based, at least in part, on the locally optimal solution.

### Claim at position 16

-23. (canceled)

### Claim 24

An apparatus, comprising:
an interface system; and
a control system capable of:
receiving, via the interface system, audio data corresponding to a recording of a conference involving a plurality of conference participants, the audio data including at least one of: (a) audio data from multiple endpoints, the audio data for each of the multiple endpoints having been recorded separately or (b) audio data from a single endpoint corresponding to multiple conference participants and including spatial information for each conference participant of the multiple conference participants;
analyzing the audio data to determine conversational dynamics data that includes at least one data type selected from a list of data types consisting of: data indicating the frequency and duration of conference participant speech; data indicating instances of conference participant doubletalk during which at least two conference participants are speaking simultaneously; and data indicating instances of conference participant conversations;
applying the conversational dynamics data as one or more variables of a spatial optimization cost function of a vector describing a virtual conference participant position for each of the conference participants in a virtual acoustic space;
applying an optimization technique to the spatial optimization cost function to determine a locally optimal solution; and
assigning the virtual conference participant positions in the virtual acoustic space based, at least in part, on the locally optimal solution.

### Claim 25 (depends on 24)

The apparatus of claim 24, wherein the virtual acoustic space is determined relative to a position of a virtual listener's head in the virtual acoustic space and wherein the spatial optimization cost function applies a penalty for placing conference participants who are involved in conference participant doubletalk at virtual conference participant positions that are on, or within a predetermined angular distance from, a cone of confusion defined relative to the position of the virtual listener's head, circular conical slices through the cone of confusion having identical inter-aural time differences.

### Claim 26 (depends on 24)

The apparatus of claim 24, wherein the virtual acoustic space is determined relative to a position of a virtual listener's head in the virtual acoustic space and wherein the spatial optimization cost function applies a penalty for placing conference participants who are involved in a conference participant conversation with one another at virtual conference participant positions that are on, or within a predetermined angular distance from, a cone of confusion defined relative to the position of the virtual listener's head, circular conical slices through the cone of confusion having identical inter-aural time differences.

### Claim 27 (depends on 24)

The apparatus of claim 24, wherein the virtual acoustic space is determined relative to a position of a virtual listener's head in the virtual acoustic space and wherein the spatial optimization cost function applies a penalty for placing conference participants who speak frequently at virtual conference participant positions that are beside, behind, above, or below the position of the virtual listener's head.

### Claim 28 (depends on 24)

The apparatus of claim 24, wherein the virtual acoustic space is determined relative to a position of a virtual listener's head in the virtual acoustic space and wherein the spatial optimization cost function applies a penalty for placing conference participants who speak frequently at virtual conference participant positions that are farther from the position of the virtual listener's head than the virtual conference participant positions of conference participants who speak less frequently.

### Claim at position 22

-34. (canceled)
