# US 11,115,541 B2 · Post-teleconference playback using non-destructive audio transport > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,115,541 B2 - Document ID: US11115541B2 - Status: Granted - Kind: B2 - Application number: 16/691,487 - Filed: 2019-11-21 - Published: 2020-03-19 - Granted: 2021-09-07 - Earliest claimed: 2015-06-16 - Assignees: Dolby Laboratories Licensing Corporation - Applicants: Dolby Laboratories Licensing Corporation - Inventors: Glenn N. Dickins; Richard J. Cartwright - Web page: https://patentgrep.com/patent/US11115541B2 ## CPC - H04M 3/568: {audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants (echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30)} - G10L 21/0232: Processing in the frequency domain - G10L 21/0316: by changing the amplitude - H04M 3/2281: {Call monitoring, e.g. for law enforcement purposes; Call tracing; Detection or prevention of malicious calls} - H04M 3/42221: {Conversation recording systems (at the subscriber's set H04M1/656)} - G10L 2021/02082: {the noise being echo, reverberation of the speech} - H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Teleconference audio data including a plurality of individual uplink data packet streams, may be received during a teleconference. Each uplink data packet stream may corresponding to a telephone endpoint used by one or more teleconference participants. The teleconference audio data may be analyzed to determine a plurality of suppressive gain coefficients, which may be applied to first instances of the teleconference audio data during the teleconference, to produce first gain-suppressed audio data provided to the telephone endpoints during the teleconference. Second instances of the teleconference audio data, as well as gain coefficient data corresponding to the plurality of suppressive gain coefficients, may be sent to a memory system as individual uplink data packet streams. The second instances of the teleconference audio data may be less gain-suppressed than the first gain-suppressed audio data. ## Claims ### Claim 1 A method for processing audio data, the method comprising: receiving, by an analysis engine, audio data corresponding to a teleconference recording involving a plurality of conference participants, the audio data comprising an individual uplink data packet stream for each of the plurality of conference participants, each of the individual uplink data packet streams including gain coefficient data and at least one of: (a) conference participant speech data from multiple endpoints, recorded separately or (b) conference participant speech data from a single endpoint corresponding to multiple conference participants and including information for identifying conference participant speech for each conference participant of the multiple conference participants, the gain coefficient data corresponding to suppressive gain coefficients applied during the teleconference; analyzing, by the analysis engine, the audio data; determining, by the analysis engine, proposed modifications to at least some of the gain coefficient data, the proposed modifications to be applied when the teleconference recording is played back; and outputting indications of the proposed modifications, the indications of the proposed modifications corresponding to proposed selective changes to the attenuation of conference participant nuisance audio for playback, as compared to the attenuation of conference participant nuisance audio during the teleconference according to the suppressive gain coefficients, the conference participant nuisance audio corresponding to apparent non-voice activity. ### Claim 2 (depends on 1) The method of claim 1, wherein the gain coefficient data includes one or more types of gain coefficient data selected from a list of gain coefficient data types consisting of: gain coefficient data indicating gains could be applied to audio signals before and after instances of detected voice activity; gain coefficient data indicating gains that could be applied to level audio signals corresponding to voice activity; gain coefficient data indicating gains that could be applied to attenuate noise; gain coefficient data indicating gains that could be applied to attenuate conference participant nuisance audio corresponding to apparent non-voice activity; gain coefficient data indicating gains that could be applied to attenuate sibilance caused by voice capture and coding of fricatives; and gain coefficient data indicating gains that could be applied to attenuate reverberation. ### Claim 3 (depends on 1) The method of claim 1, wherein the analyzing involves: analyzing conversational dynamics of the conference recording to determine conversational dynamics data; searching the conference recording to determine instances of each of a plurality of segment classifications, each of the segment classifications based, at least in part, on the conversational dynamics data; and segmenting the conference recording into a plurality of segments, each of the segments corresponding with a time interval and at least one of the segment classifications, wherein the analyzing, searching and segmenting processes are performed by the analysis engine. ### Claim 4 (depends on 3) The method of claim 3, wherein instances of the segment classifications are determined according to a set of rules and wherein the rules are based on one or more conversational dynamics data types selected from a group of conversational dynamics data types consisting of: (a) a doubletalk ratio indicating a fraction of speech time in a time interval during which at least two conference participants are speaking simultaneously; (b) a speech density metric indicating a fraction of the time interval during which there is any conference participant speech; and (c) a dominance metric indicating a fraction of total speech uttered by a dominant conference participant during the time interval, the dominant conference participant being a conference participant who spoke the most during the time interval. ### Claim 5 (depends on 1) The method of claim 1, further comprising storing the indications of the proposed modifications as metadata. ### Claim 6 (depends on 1) The method of claim 1, further comprising modifying the gain coefficient data according to the proposed modifications. ### Claim 7 (depends on 1) The method of claim 1, wherein the indications of the proposed modifications correspond to proposed selective increases or reductions of noise attenuation for playback, as compared to noise attenuation applied during the teleconference according to the suppressive gain coefficients. ### Claim 8 (depends on 1) The method of claim 1, wherein: analyzing the audio data involves a post-teleconference voice activity detection process; and the indications of the proposed modifications include proposed changes to gains that could be applied before and after instances of detected voice activity during playback, as compared to gains that were applied during the teleconference, before and after instances of detected voice activity according to the suppressive gain coefficients. ### Claim 9 (depends on 1) The method of claim 1, wherein the audio data corresponding to the teleconference recording comprises audio data for at least one entire teleconference. ### Claim 10 A non-transitory medium having software stored thereon, the software including instructions for processing audio data by controlling at least one device for: receiving audio data corresponding to a teleconference recording involving a plurality of conference participants, the audio data comprising an individual uplink data packet stream for each of the plurality of conference participants, each of the individual uplink data packet streams including gain coefficient data and at least one of: (a) conference participant speech data from multiple endpoints, recorded separately or (b) conference participant speech data from a single endpoint corresponding to multiple conference participants and including information for identifying conference participant speech for each conference participant of the multiple conference participants, the gain coefficient data corresponding to suppressive gain coefficients applied during the teleconference; analyzing the audio data, wherein the analyzing involves: analyzing conversational dynamics of the conference recording to determine conversational dynamics data; searching the conference recording to determine instances of each of a plurality of segment classifications, each of the segment classifications based, at least in part, on the conversational dynamics data; and segmenting the conference recording into a plurality of segments, each of the segments corresponding with a time interval and at least one of the segment classifications; determining proposed modifications to at least some of the gain coefficient data, the proposed modifications to be applied when the teleconference recording is played back; and outputting indications of the proposed modifications. ### Claim 11 (depends on 10) The non-transitory medium of claim 10, wherein the gain coefficient data includes one or more types of gain coefficient data selected from a list of gain coefficient data types consisting of: gain coefficient data indicating gains could be applied to audio signals before and after instances of detected voice activity; gain coefficient data indicating gains that could be applied to level audio signals corresponding to voice activity; gain coefficient data indicating gains that could be applied to attenuate noise; gain coefficient data indicating gains that could be applied to attenuate conference participant nuisance audio corresponding to apparent non-voice activity; gain coefficient data indicating gains that could be applied to attenuate sibilance caused by voice capture and coding of fricatives; and gain coefficient data indicating gains that could be applied to attenuate reverberation. ### Claim 12 An apparatus, comprising: an interface system; and a control system capable of: receiving, via the interface system, audio data corresponding to a teleconference recording involving a plurality of conference participants, the audio data comprising an individual uplink data packet stream for each of the plurality of conference participants, each of the individual uplink data packet streams including gain coefficient data and at least one of: (a) conference participant speech data from multiple endpoints, recorded separately or (b) conference participant speech data from a single endpoint corresponding to multiple conference participants and including information for identifying conference participant speech for each conference participant of the multiple conference participants, the gain coefficient data corresponding to suppressive gain coefficients applied during the teleconference, the gain coefficient data including one or more types of gain coefficient data selected from a list of gain coefficient data types consisting of: gain coefficient data indicating gains could be applied to audio signals before and after instances of detected voice activity; gain coefficient data indicating gains that could be applied to level audio signals corresponding to voice activity; gain coefficient data indicating gains that could be applied to attenuate noise; gain coefficient data indicating gains that could be applied to attenuate conference participant nuisance audio corresponding to apparent non-voice activity; gain coefficient data indicating gains that could be applied to attenuate sibilance caused by voice capture and coding of fricatives; and gain coefficient data indicating gains that could be applied to attenuate reverberation; analyzing the audio data; determining proposed modifications to at least some of the gain coefficient data, the proposed modifications to be applied when the teleconference recording is played back; and outputting indications of the proposed modifications. ### Claim 13 A method for processing audio data, the method comprising: receiving, by an analysis engine, audio data corresponding to a teleconference recording involving a plurality of conference participants, the audio data comprising an individual uplink data packet stream for each of the plurality of conference participants, each of the individual uplink data packet streams including gain coefficient data and at least one of: (a) conference participant speech data from multiple endpoints, recorded separately or (b) conference participant speech data from a single endpoint corresponding to multiple conference participants and including information for identifying conference participant speech for each conference participant of the multiple conference participants, the gain coefficient data corresponding to suppressive gain coefficients applied during the teleconference; analyzing, by the analysis engine, the audio data; determining, by the analysis engine, proposed modifications to at least some of the gain coefficient data, the proposed modifications to be applied when the teleconference recording is played back; and outputting indications of the proposed modifications, wherein: analyzing the audio data involves a post-teleconference voice activity detection process; and the indications of the proposed modifications include proposed changes to gains that could be applied before and after instances of detected voice activity during playback, as compared to gains that were applied during the teleconference, before and after instances of detected voice activity according to the suppressive gain coefficients. ### Claim 14 (depends on 13) The method of claim 13, wherein the gain coefficient data includes one or more types of gain coefficient data selected from a list of gain coefficient data types consisting of: gain coefficient data indicating gains could be applied to audio signals before and after instances of detected voice activity; gain coefficient data indicating gains that could be applied to level audio signals corresponding to voice activity; gain coefficient data indicating gains that could be applied to attenuate noise; gain coefficient data indicating gains that could be applied to attenuate conference participant nuisance audio corresponding to apparent non-voice activity; gain coefficient data indicating gains that could be applied to attenuate sibilance caused by voice capture and coding of fricatives; and gain coefficient data indicating gains that could be applied to attenuate reverberation. ### Claim 15 (depends on 13) The method of claim 13, wherein the analyzing involves: analyzing conversational dynamics of the conference recording to determine conversational dynamics data; searching the conference recording to determine instances of each of a plurality of segment classifications, each of the segment classifications based, at least in part, on the conversational dynamics data; and segmenting the conference recording into a plurality of segments, each of the segments corresponding with a time interval and at least one of the segment classifications, wherein the analyzing, searching and segmenting processes are performed by the analysis engine. ### Claim 16 (depends on 15) The method of claim 15, wherein instances of the segment classifications are determined according to a set of rules and wherein the rules are based on one or more conversational dynamics data types selected from a group of conversational dynamics data types consisting of: (a) a doubletalk ratio indicating a fraction of speech time in a time interval during which at least two conference participants are speaking simultaneously; (b) a speech density metric indicating a fraction of the time interval during which there is any conference participant speech; and (c) a dominance metric indicating a fraction of total speech uttered by a dominant conference participant during the time interval, the dominant conference participant being a conference participant who spoke the most during the time interval. ### Claim 17 (depends on 13) The method of claim 13, further comprising modifying the gain coefficient data according to the proposed modifications.