# US 12,192,022 B1 · Video conference content auto-retrieval and focus based on learned relevance > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,192,022 B1 - Document ID: US12192022B1 - Status: Granted - Kind: B1 - Application number: 18/592,428 - Filed: 2024-02-29 - Granted: 2025-01-07 - Earliest claimed: 2019-12-30 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Victor Carbune; Daniel Keysers; Thomas Deselaers - Web page: https://patentgrep.com/patent/US12192022B1 ## CPC - H04L 12/1831: {Tracking arrangements for later retrieval, e.g. recording contents, participants activities or behavior, network status} - H04L 12/1818: {Conference organisation arrangements, e.g. handling schedules, setting up parameters needed by nodes to attend a conference, booking network resources, notifying involved parties} - H04L 12/1822: {Conducting the conference, e.g. admission, detection, selection or grouping of participants, correlating users to one or more conference sessions, prioritising transmission} - H04L 12/1827: {Network arrangements for conference optimisation or adaptation} - H04M 3/566: {relating to a participants right to speak (arrangements for multi-party communication with floor control, e.g. for conferences, H04L65/4038, H04L65/4046, H04L65/4053)} - H04M 3/568: {audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants (echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30)} - H04M 3/569: {using the instant speaker's algorithm (speech detection per se G10L25/78)} - H04M 2201/50: Telephonic communication in combination with video communication - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08) ## Abstract Systems and methods for video conference content auto-retrieval and focus based on learned relevance is provided. A method may include determining collaborative documents each associated with at least one user participating in a video conference, and providing one or more first inputs to a machine learning model, the one or more first inputs identifying the plurality of collaborative documents, and comprising an indication of a request to identify a collaborative document that is most relevant to a first subject discussed during the video conference at a first point in time. The method may further include determining, based on one or more first outputs of the machine learning model, the collaborative document that is the most relevant to the first subject discussed during the video conference at the first point in time, and providing the determined collaborative document for presentation on one or more client devices while the first subject is being discussed during the video conference. ## Claims ### Claim 1 A method comprising: determining a plurality of collaborative documents each associated with at least one user of at least one of a plurality of client devices participating in a video conference; providing one or more first inputs to a machine learning model, the one or more first inputs identifying the plurality of collaborative documents, and comprising an indication of a request to identify a collaborative document that is most relevant to a first subject discussed during the video conference at a first point in time; determining, based on one or more first outputs of the machine learning model, the collaborative document that is the most relevant to the first subject discussed during the video conference at the first point in time; providing the determined collaborative document for presentation on one or more of the plurality of client devices while the first subject is being discussed during the video conference; and responsive to determining, at a second point in time during the video conference, that the first subject has changed to a second subject: determining, based on one or more second outputs of the machine learning model, that none of the plurality of collaborative documents is relevant to the second subject discussed during the video conference at the second point in time; and providing, for presentation on the plurality of client devices while the second subject is being discussed during the video conference, a media stream associated with a current speaker of the video conference. ### Claim 2 (depends on 1) The method of claim 1, further comprising: receiving a first plurality of media streams of the plurality of client devices participating in the video conference, the first plurality of media streams associated with the first point in time; and generating a transcription of an audio component of each of the first plurality of media streams in relation to the first point in time. ### Claim 3 (depends on 2) The method of claim 2, further comprising: determining the first subject being discussed during the video conference at the first point in time based on generated transcriptions of audio components of the first plurality of media streams; and providing the first subject as an additional input to the machine learning model that is trained to identify a collaborative document that is most relevant to a subject provided as input to the machine learning model. ### Claim 4 (depends on 3) The method of claim 3, wherein the first subject is determined based on topical analysis of the generated transcriptions. ### Claim 5 (depends on 3) The method of claim 3, wherein the machine learning model is trained based on training data comprising information identifying collaborative documents presented during past video conferences, and subjects discussed at various points in time during the past video conferences. ### Claim 6 (depends on 2) The method of claim 2, further comprising: providing the generated transcriptions as additional input to the machine learning model that is trained to determine a subject based on transcriptions provided as input to the machine learning model, and to also to identify a collaborative document that is most relevant to a determined subject. ### Claim 7 (depends on 1) The method of claim 1, wherein the machine learning model is trained based on training data comprising transcriptions corresponding to a third plurality of media streams from past video conferences, and information identifying collaborative documents presented during the past video conferences. ### Claim 8 (depends on 1) The method of claim 1, wherein each of the plurality of collaborative documents has a user of one of the plurality of client devices as a collaborator and is determined based on access rights assigned to users of the plurality of client devices with respect to the plurality of collaborative documents. ### Claim 9 (depends on 1) The method of claim 1, wherein the one or more first outputs of the machine learning model comprise rankings of the plurality of collaborative documents, wherein each ranking indicates a probability of a respective document being relevant to the first subject. ### Claim 10 (depends on 1) The method of claim 1, wherein providing the determined collaborative document for presentation on one or more of the plurality of client devices comprises: providing, for presentation on one or more of the plurality of client devices, a video conference user interface (UI) including a suggestion pertaining to the determined collaborative document; and upon receiving an indication of an acceptance of the suggestion, providing a media stream associated with a presentation of the document in the video conference UI on the plurality of client devices while the first subject is being discussed during the video conference. ### Claim 11 A non-transitory computer-readable medium comprising instructions, which when executed by a processing device cause a processing device to perform operations comprising: determining a plurality of collaborative documents each associated with at least one user of at least one of a plurality of client devices participating in a video conference; providing one or more first inputs to a machine learning model, the one or more first inputs identifying the plurality of collaborative documents, and comprising an indication of a request to identify a collaborative document that is most relevant to a first subject discussed during the video conference at a first point in time; determining, based on one or more first outputs of the machine learning model, the collaborative document that is the most relevant to the first subject discussed during the video conference at the first point in time; providing the determined collaborative document for presentation on one or more of the plurality of client devices while the first subject is being discussed during the video conference; and responsive to determining, at a second point in time during the video conference, that the first subject has changed to a second subject: determining, based on one or more second outputs of the machine learning model, that none of the plurality of collaborative documents is relevant to the second subject discussed during the video conference at the second point in time; and providing, for presentation on the plurality of client devices while the second subject is being discussed during the video conference, a media stream associated with a current speaker of the video conference. ### Claim 12 (depends on 11) The non-transitory computer-readable medium of claim 11, the operations further comprising: receiving a first plurality of media streams of the plurality of client devices participating in the video conference, the first plurality of media streams associated with the first point in time; and generating a transcription of an audio component of each of the first plurality of media streams in relation to the first point in time. ### Claim 13 (depends on 12) The non-transitory computer-readable medium of claim 12, the operations further comprising: determining the first subject being discussed during the video conference at the first point in time based on generated transcriptions of audio components of the first plurality of media streams; and providing the first subject as an additional input to the machine learning model that is trained to identify a collaborative document that is most relevant to a subject provided as input to the machine learning model. ### Claim 14 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein the machine learning model is trained based on training data comprising information identifying collaborative documents presented during past video conferences, and subjects discussed at various points in time during the past video conferences. ### Claim 15 (depends on 12) The non-transitory computer-readable medium of claim 12, the operations further comprising: providing the generated transcriptions as additional input to the machine learning model that is trained to determine a subject based on transcriptions provided as input to the machine learning model, and to also to identify a collaborative document that is most relevant to a determined subject. ### Claim 16 A system comprising: a memory; and a processing device, coupled to the memory, to perform operations comprising: determining a plurality of collaborative documents each associated with at least one user of at least one of a plurality of client devices participating in a video conference; providing one or more first inputs to a machine learning model, the one or more first inputs identifying the plurality of collaborative documents, and comprising an indication of a request to identify a collaborative document that is most relevant to a first subject discussed during the video conference at a first point in time; determining, based on one or more first outputs of the machine learning model, the collaborative document that is the most relevant to the first subject discussed during the video conference at the first point in time; providing the determined collaborative document for presentation on one or more of the plurality of client devices while the first subject is being discussed during the video conference; and responsive to determining, at a second point in time during the video conference, that the first subject has changed to a second subject: determining, based on one or more second outputs of the machine learning model, that none of the plurality of collaborative documents is relevant to the second subject discussed during the video conference at the second point in time; and providing, for presentation on the plurality of client devices while the second subject is being discussed during the video conference, a media stream associated with a current speaker of the video conference. ### Claim 17 (depends on 16) The system of claim 16, the operations further comprising: receiving a first plurality of media streams of the plurality of client devices participating in the video conference, the first plurality of media streams associated with the first point in time; and generating a transcription of an audio component of each of the first plurality of media streams in relation to the first point in time. ### Claim 18 (depends on 17) The system of claim 17, the operations further comprising: determining the first subject being discussed during the video conference at the first point in time based on generated transcriptions of audio components of the first plurality of media streams; and providing the first subject as an additional input to the machine learning model that is trained to identify a collaborative document that is most relevant to a subject provided as input to the machine learning model. ### Claim 19 (depends on 18) The system of claim 18, wherein the machine learning model is trained based on training data comprising information identifying collaborative documents presented during past video conferences, and subjects discussed at various points in time during the past video conferences. ### Claim 20 (depends on 16) The system of claim 16, wherein providing the determined collaborative document for presentation on one or more of the plurality of client devices comprises: providing, for presentation on one or more of the plurality of client devices, a video conference user interface (UI) including a suggestion pertaining to the determined collaborative document; and upon receiving an indication of an acceptance of the suggestion, providing a media stream associated with a presentation of the document in the video conference UI on the plurality of client devices while the first subject is being discussed during the video conference.