# US 12,217,760 B2 · Metadata-based diarization of teleconferences > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,760 B2 - Document ID: US12217760B2 - Status: Granted - Kind: B2 - Application number: 17/588,296 - Filed: 2022-01-30 - Published: 2022-05-19 - Granted: 2025-02-04 - Earliest claimed: 2018-04-17 - Assignees: GONG.io Ltd. - Applicants: Gong.Io Ltd. - Inventors: Eilon Reshef; Hanan Shteingart; Zohar Shay; Shlomi Medalion - Web page: https://patentgrep.com/patent/US12217760B2 ## CPC - G10L 17/06: Decision making techniques; Pattern matching strategies - G06V 30/19: Recognition using electronic means - G06V 40/172: {Classification, e.g. identification} - G10L 17/02: Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction - G10L 17/04: Training, enrolment or model building - G10L 17/18: Artificial neural networks; Connectionist approaches - H04L 65/403: Arrangements for multi-party communication, e.g. for conferences (data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15) - H04L 65/765: {intermediate} - G10L 21/028: using properties of sound source - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract A method for audio processing includes receiving a recording of a teleconference among multiple participants over a network, including an audio stream containing speech uttered by the participants and information outside the audio stream. The method further includes processing the audio stream to identify speech segments interspersed with intervals of silence, extracting speaker identifications from the information outside the audio stream in the received recording, labeling a first set of the identified speech segments from the audio stream with the speaker identifications, extracting acoustic features from the speech segments in the first set, learning a correlation between the speaker identifications labelled to the segments in the first set and the extracted acoustic features, and labeling a second set of the identified speech segments using the learned correlation, to indicate the participants who spoke during the speech segments in the second set. ## Claims ### Claim 1 A method for audio processing, comprising: receiving, in a computer, a recording of a teleconference among multiple participants over a network, the recording including an audio stream containing speech uttered by the participants and information outside the audio stream; processing the audio stream by the computer to identify speech segments, in which one or more of the participants were speaking, interspersed with intervals of silence in the audio stream; extracting speaker identifications, which are indicative of the participants who spoke during periods of the teleconference, from the information outside the audio stream in the received recording, so as to provide for a plurality of periods the of the teleconference, corresponding speaker identifications; labeling a first set of the identified speech segments from the audio stream with the speaker identifications extracted from the information outside the audio stream in the received recording, wherein each speech segment from the audio stream, in the first set, is labelled with a speaker identification of a period corresponding to a time of the speech segment; extracting acoustic features from the speech segments in the first set; learning a correlation between the speaker identifications labelled to the segments in the first set, and the extracted acoustic features extracted from the corresponding segments of the first set; and relabeling one or more of the identified speech segments of the first set, using the learned correlation, to indicate the participants who spoke based on the learned correlation instead of the speaker identifications extracted from the information outside the audio stream. ### Claim 2 (depends on 1) The method according to claim 1, wherein the information outside the audio stream in the received recording comprises conference metadata and wherein extracting the speaker identifications comprises extracting from the conference metadata. ### Claim 3 (depends on 1) The method according to claim 1, wherein the information outside the audio stream in the received recording comprises a visual component of the teleconference recording, and wherein extracting the speaker identifications comprises extracting from the visual component of the teleconference. ### Claim 4 (depends on 3) The method according to claim 3, wherein the visual component of the teleconference recording comprises visual images of participants of the teleconference recording, and wherein extracting the speaker identifications comprises extracting from the visual images of the participants of the teleconference. ### Claim 5 (depends on 3) The method according to claim 3, wherein extracting the speaker identifications comprises optical character recognition (OCR) analysis of names on images of the visual component of the teleconference recording. ### Claim 6 (depends on 3) The method according to claim 3, wherein extracting the speaker identifications comprises analyzing a graphical representation indicating an active speaker, in the visual component of the teleconference recording. ### Claim 7 (depends on 3) The method according to claim 3, wherein the visual component of the teleconference comprise a video portion of the teleconference recording, and wherein extracting the speaker identifications comprises identifying an active speaker using facial recognition. ### Claim 8 (depends on 1) The method according to claim 1, wherein learning the correlation comprises training a neural network to predict the labelled speaker identifications of the segments in the first set, using the acoustic features extracted from the speech segments in the first set as inputs. ### Claim 9 (depends on 1) The method according to claim 1, wherein relabeling the one or more of the identified speech segments comprises applying a dynamic programming algorithm. ### Claim 10 (depends on 1) The method according to claim 1, wherein relabeling the one or more of the identified speech segments comprises measuring and compensating for a delay in transmission of the audio stream over the network relative to timestamps associated with the information outside the audio stream. ### Claim 11 (depends on 1) The method according to claim 1, further comprising labeling one or more of the speech segments for which the information outside the audio stream did not provide a speaker identification, using the learned correlation. ### Claim 12 (depends on 1) The method according to claim 1, wherein learning the correlation comprises building a respective statistical model of the speech of each participant based on the audio stream in the first set of the speech segments that were labeled as belonging to the participant, and wherein relabeling the one or more of the identified speech segments comprises comparing the statistical model to each of a sequence of time frames in the audio stream. ### Claim 13 (depends on 12) The method according to claim 12, further comprising building an updated statistical model using the one or more relabeled identified speech segments and updating the labeling of the first set of the identified speech segments using the updated statistical model. ### Claim 14 (depends on 12) The method according to claim 12, further comprising repeating the building of an updated statistical model and the updating of the labeling of the first set of the identified speech segments until a target overall confidence level of the labelling is reached. ### Claim 15 (depends on 1) The method according to claim 1, wherein learning the correlation comprises estimating transition probabilities between the speaker identifications based on the labeled speech segments in the first set, and wherein relabeling the one or more of the identified speech segments comprises applying the transition probabilities in relabeling of the speech segments. ### Claim 16 A method for audio processing, comprising: receiving, in a computer, a recording of a teleconference among multiple participants over a network, the recording including an audio stream containing speech uttered by the participants and a visual component of the teleconference recording; processing the audio stream by the computer to identify speech segments, in which one or more of the participants were speaking, interspersed with intervals of silence in the audio stream; extracting speaker identifications, which are indicative of the participants who spoke during periods of the teleconference from the visual component of the teleconference recording; labeling a first set of the identified speech segments from the audio stream with initial speaker identifications, wherein each speech segment from the audio stream, in the first set, is labelled with a speaker identification of a period corresponding to a time of the speech segment; extracting acoustic features from the speech segments in the first set; learning a correlation between the speaker identifications labelled to the segments in the first set, and the extracted acoustic features extracted from the corresponding segments of the first set; and relabeling one or more of the identified speech segments of the first set, using the learned correlation. ### Claim 17 (depends on 16) The method according to claim 16, wherein the visual component of the teleconference recording comprise visual images of participants of the teleconference wherein extracting the speaker identifications comprises extracting from the visual images the participants of the teleconference. ### Claim 18 (depends on 16) The method according to claim 16, wherein extracting the speaker identifications comprises optical character recognition (OCR) analysis of names on images of the visual component of the teleconference recording. ### Claim 19 (depends on 16) The method according to claim 16, wherein extracting the speaker identifications comprises analyzing a graphical representation indicating an active speaker, in the visual component of the teleconference recording. ### Claim 20 (depends on 16) The method according to claim 16, wherein the visual component of the teleconference comprise a video of the teleconference recording, and wherein portion extracting the speaker identifications comprises identifying an active speaker using facial recognition. ### Claim 21 Apparatus for audio processing, comprising: a memory, which is configured to store a recording of a teleconference among multiple participants over a network, the recording including an audio stream containing speech uttered by the participants and information outside the audio stream; and a processor, which is configured to: process the audio stream so as to identify speech segments, in which one or more of the participants were speaking, interspersed with intervals of silence in the audio stream, extract speaker identifications, which are indicative of the participants who spoke during periods of the teleconference from the information outside the audio stream in the received recording, so as to provide for a plurality of the periods of the teleconference, corresponding speaker identifications, label a first set of the identified speech segments from the audio stream with the speaker identifications extracted from the information outside the audio stream in the received recording, wherein each speech segment from the audio stream, in the first set, is labelled with a speaker identification of a period corresponding to a time of the speech segment; extract acoustic features from the speech segments in the first set; learn a correlation between the speaker identifications labelled to the segments in the first set, and the extracted acoustic features extracted from the corresponding segments of the first set; and relabel one or more of the identified speech segments of the first set, using the learned correlation, to indicate the participants who spoke based on the learned correlation instead of the speaker identifications extracted from the information outside the audio stream. ### Claim 22 (depends on 21) The apparatus according to claim 21, wherein the information outside the audio stream in the received recording comprises conference metadata and wherein the processor is configured to extract the speaker identifications from the conference metadata. ### Claim 23 (depends on 21) The apparatus according to claim 21, wherein the information outside the audio stream in the received recording comprises a visual component of the teleconference recording, and wherein the processor is configured to extract the speaker identifications from visual component of the teleconference. ### Claim 24 (depends on 23) The apparatus according to claim 23, wherein the visual component of the teleconference recording comprises visual images of participants of the teleconference recording, and wherein the processor is configured to extract the speaker identifications from the visual images of the participants of the teleconference. ### Claim 25 (depends on 23) The apparatus according to claim 23, wherein the processor is configured to extract the speaker identifications by optical character recognition (OCR) analysis of names on images of the visual component of the teleconference recording. ### Claim 26 (depends on 23) The apparatus according to claim 23, wherein the processor is configured to extract the speaker identifications by analyzing a graphical representation indicating an active speaker, in the visual component of the teleconference recording. ### Claim 27 (depends on 23) The apparatus according to claim 23, wherein the visual component of the teleconference comprise a video portion of the teleconference recording, and wherein the processor is configured to extract the speaker identifications by identifying an active speaker using facial recognition. ### Claim 28 (depends on 21) The apparatus according to claim 21, wherein the processor is configured to learn the correlation by training a neural network to predict the labelled speaker identifications of the segments in the first set, using the acoustic features extracted from the speech segments in the first set as inputs. ### Claim 29 (depends on 21) The apparatus according to claim 21, wherein the processor is configured to relabel the one or more of the identified speech segments by applying a dynamic programming algorithm. ### Claim 30 (depends on 21) The apparatus according to claim 21, wherein the processor is configured to relabel the one or more of the identified speech segments by measuring and compensating for a delay in transmission of the audio stream over the network relative to timestamps associated with the information outside the audio stream. ### Claim 31 (depends on 21) The apparatus according to claim 21, wherein the processor is further configured to label one or more of the speech segments for which the information outside the audio stream did not provide a speaker identification, using the learned correlation. ### Claim 32 (depends on 21) The apparatus according to claim 21, wherein the processor is configured to learn the correlation by building a respective statistical model of the speech of each participant based on the audio stream in the first set of the speech segments that were labeled as belonging to the participant, and wherein relabeling the one or more of the identified speech segments comprises comparing the statistical model to each of a sequence of time frames in the audio stream. ### Claim 33 (depends on 32) The apparatus according to claim 32, wherein the processor is additionally configured to build an updated statistical model using the one or more relabeled identified speech segments and to update the labeling of the first set of the identified speech segments using the updated statistical model. ### Claim 34 (depends on 32) The apparatus according to claim 32, wherein the processor is additionally configured to repeat the building of an updated statistical model and the updating of the labeling of the first set of the identified speech segments until a target overall confidence level of the labelling is reached. ### Claim 35 (depends on 21) The apparatus according to claim 21, wherein the processor is configured to learn the correlation by estimating transition probabilities between the speaker identifications based on the labeled speech segments in the first set, and to relabel the one or more of the identified speech segments by applying the transition probabilities in relabeling of the speech segments.