# US 2025/0097385 A1 · Multimodal meeting segmentation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2025/0097385 A1 - Document ID: US20250097385A1 - Status: Published application - Kind: A1 - Application number: 18/778,653 - Filed: 2024-07-19 - Published: 2025-03-20 - Earliest claimed: 2023-09-15 - Assignees: Zoom Video Communications, Inc. - Applicants: Zoom Video Communications, Inc. - Inventors: Bilung Lee; Vijay Venkataswamy Parthasarathy; Renjie Tao; Chih-Kai Ting; Yun Zhang - Web page: https://patentgrep.com/patent/US20250097385A1 ## CPC - H04N 7/155: {involving storage of or access to video conference sessions (tracking arrangements for later retrieval of a computer conference content or participants activities H04L12/1831)} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract One method for multimodal meeting segmentation includes receiving a meeting recording and a transcript, the meeting recording comprising a first media stream; generating, using a trained machine learning (“ML”) model, a first set of embeddings corresponding to the first media stream and a second set of embeddings corresponding to the transcript; generating a first set of segments based on the first set of embeddings and a second set of segments based on the second set of embeddings; generating a final set of segments based on the first and second sets of segments; and associating the final set of segments with the meeting recording and storing the final set of segments. ## Claims ### Claim 1 A method comprising: receiving a meeting recording and a transcript, the meeting recording comprising a first media stream; generating, using a trained machine learning (“ML”) model, a first set of embeddings corresponding to the first media stream and a second set of embeddings corresponding to the transcript; generating a first set of segments based on the first set of embeddings and a second set of segments based on the second set of embeddings; generating a final set of segments based on the first and second sets of segments; and associating the final set of segments with the meeting recording and storing the final set of segments. ### Claim 2 (depends on 1) The method of claim 1, wherein the meeting recording comprises a second media stream, the first media stream comprising an audio stream and the second media stream comprising a video stream, and further comprising: generating, using the trained ML model, a third set of embeddings corresponding to the second media stream; generating a third set of segments based on the third set of embeddings; and wherein generating the final set of segments is further based on the third set of segments. ### Claim 3 (depends on 1) The method of claim 1, wherein: generating the first set of segments is based on a similarity between consecutive embeddings within the first set of segments; and generating the second set of segments is based on a similarity between consecutive embeddings within the second set of segments. ### Claim 4 (depends on 3) The method of claim 3, wherein the similarity between consecutive embeddings within the first set of segments and the similarity between consecutive embeddings within the first set of segments is based on at least one of a Euclidean distance, cosine, or dot product between the consecutive embeddings within the respective sets of embeddings. ### Claim 5 (depends on 1) The method of claim 1, wherein generating the first and second sets of segments comprises: responsive to identifying, in a respective set of segments, a first segment having a length satisfying a first threshold, partitioning the first segment into a plurality of partitioned segments. ### Claim 6 (depends on 5) The method of claim 5, further comprising: for each partitioned segment, responsive to determining that the respective partitioned segment satisfies a second threshold, merging the respective partitioned segment into a preceding or succeeding segment. ### Claim 7 (depends on 6) The method of claim 6, further comprising iteratively, after the partitioning and merging: responsive to determining that none of the segments within the respective set of segments satisfies the first threshold, outputting the respective set of segments; and otherwise performing the partitioning and merging on each segment satisfying the first threshold. ### Claim 8 (depends on 7) The method of claim 7, further comprising, responsive to determining that a maximum number of iterations has been reached, outputting the respective set of segments. ### Claim 9 A system comprising: a non-transitory computer-readable medium; and one or more processors communicatively connected to the non-transitory computer-readable medium, the one or more processors configured to execute processor-executable instructions stored in the non-transitory computer-readable medium to cause the one or more processors to: receive a meeting recording and a transcript, the meeting recording comprising a first media stream; generate, using a trained machine learning (“ML”) model, a first set of embeddings corresponding to the first media stream and a second set of embeddings corresponding to the transcript; generate a first set of segments based on the first set of embeddings and a second set of segments based on the second set of embeddings; generate a final set of segments based on the first and second sets of segments; and associate the final set of segments with the meeting recording and storing the final set of segments. ### Claim 10 (depends on 9) The system of claim 9, wherein the meeting recording comprises a second media stream, the first media stream comprising an audio stream and the second media stream comprising a video stream, and wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to: generate, using the trained ML model, a third set of embeddings corresponding to the second media stream; generate a third set of segments based on the third set of embeddings; and generate the final set of segments further based on the third set of segments. ### Claim 11 (depends on 9) The system of claim 9, wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to: generate the first set of segments is based on a similarity between consecutive embeddings within the first set of segments; and generate the second set of segments is based on a similarity between consecutive embeddings within the second set of segments. ### Claim 12 (depends on 11) The system of claim 11, wherein the similarity between consecutive embeddings within the first set of segments and the similarity between consecutive embeddings within the first set of segments is based on at least one of a Euclidean distance, cosine, or dot product between the consecutive embeddings within the respective sets of embeddings. ### Claim 13 (depends on 9) The system of claim 9, wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to: responsive to identifying, in a respective set of segments, a first segment having a length satisfying a first threshold, partition the first segment into a plurality of partitioned segments. ### Claim 14 (depends on 13) The system of claim 13, wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to: for each partitioned segment, responsive to determining that the respective partitioned segment satisfies a second threshold, merge the respective partitioned segment into a preceding or succeeding segment. ### Claim 15 (depends on 14) The system of claim 14, wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to iteratively, after the partitioning and merging: responsive to determining that none of the segments within the respective set of segments satisfies the first threshold, output the respective set of segments; and otherwise perform the partitioning and merging on each segment satisfying the first threshold. ### Claim 16 (depends on 15) The system of claim 15, wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to, responsive to determining that a maximum number of iterations has been reached, output the respective set of segments. ### Claim 17 A non-transitory computer-readable medium comprising processor-executable instructions configured to cause one or more processors to: receive a meeting recording and a transcript, the meeting recording comprising a first media stream; generate, using a trained machine learning (“ML”) model, a first set of embeddings corresponding to the first media stream and a second set of embeddings corresponding to the transcript; generate a first set of segments based on the first set of embeddings and a second set of segments based on the second set of embeddings; generate a final set of segments based on the first and second sets of segments; and associate the final set of segments with the meeting recording and storing the final set of segments. ### Claim 18 (depends on 17) The non-transitory computer-readable medium of claim 17, wherein the meeting recording comprises a second media stream, the first media stream comprising an audio stream and the second media stream comprising a video stream, and further comprising processor-executable instructions configured to cause the one or more processors to: generate, using the trained ML model, a third set of embeddings corresponding to the second media stream; generate a third set of segments based on the third set of embeddings; and generate the final set of segments further based on the third set of segments. ### Claim 19 (depends on 17) The non-transitory computer-readable medium of claim 17, further comprising processor-executable instructions configured to cause the one or more processors to: generate the first set of segments is based on a similarity between consecutive embeddings within the first set of segments; and generate the second set of segments is based on a similarity between consecutive embeddings within the second set of segments. ### Claim 20 (depends on 19) The non-transitory computer-readable medium of claim 19, wherein the similarity between consecutive embeddings within the first set of segments and the similarity between consecutive embeddings within the first set of segments is based on at least one of a Euclidean distance, cosine, or dot product between the consecutive embeddings within the respective sets of embeddings.