# US 12,120,451 B2 · Augmenting video with interactive visualizations > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,120,451 B2 - Document ID: US12120451B2 - Status: Granted - Kind: B2 - Application number: 17/714,691 - Filed: 2022-04-06 - Published: 2023-10-12 - Granted: 2024-10-15 - Earliest claimed: 2022-04-06 - Assignees: Tableau Software, Inc. - Applicants: Tableau Software, LLC - Inventors: Matthew Michael Brehmer; Brian David Hall; Linda Ruth Bartram - Web page: https://patentgrep.com/patent/US12120451B2 ## CPC - H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/012: {Head tracking input arrangements} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/1454: {involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay} - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - H04L 65/401: wherein the services involve a main real-time session and one or more additional parallel real-time or time sensitive sessions, e.g. white board sharing or spawning of a subconference - H04L 65/403: Arrangements for multi-party communication, e.g. for conferences (data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15) - G09G 2340/10: Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels - G09G 2354/00: Aspects of interface with display user - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) ## Abstract Embodiments are directed to presenting visualizations over a network. A presentation specification. An overlay may be generated that includes visualizations of presentation objects based on the presentation specification. A video stream of a view of a presenter captured by a presenter station may be provided. A composited video stream may be generated based on the overlay and the video stream. The composited video stream may be provided to a video conference service that distributes the composited video stream to the presenter station and to audience stations. Movements of the presenter may be tracked based on the video stream captured by the presenter station. Gestures may be determined based on the tracked movements. In response to a determined gesture being associated with an interactive feature, activating the interactive feature to update the overlay such that the updated overlay may be included in a recomposited video stream. ## Claims ### Claim 1 A method for modifying visualizations presented over a network in a video conference application, comprising: displaying one or more interactive data visualizations in a first user interface of a video conference application on a first computer system and in a second user interface of the video conference application on a second computer system connected to the first computer system via the network; receiving, via a video camera of the first computer system, a first video stream; displaying the first video stream in the second user interface of the video conference application on the second computer system; generating an overlay that includes the one or more interactive data visualizations generating a composited video stream that includes the first video stream and the overlay; displaying the composited video stream in the second user interface of the video conference application on the second computer system, wherein the overlay is displayed in a foreground of the composited video stream and the first video stream is displayed in a background, relative to the overlay displayed in the foreground, of the composited video stream; detecting, via the video camera of the first computer system, one or more predefined gestures; and in response to detecting the one or more predefined gestures: in accordance with the one or more predefined gestures being associated with an interactive feature of at least one of the one or more interactive data visualizations: modifying the one or more interactive data visualizations; and updating the overlay in the composited video stream with the one or more modified interactive data visualizations. ### Claim 2 (depends on 1) The method of claim 1, wherein detecting the one or more predefined gestures further comprises: determining motion information from the first video stream, wherein the motion information includes position information associated with one or more of hands, fingers, eyes, head, or posture of a user. ### Claim 3 (depends on 1) The method of claim 1, wherein detecting the one or more predefined gestures further comprises: tracking one or more physical movements of a user in the first video stream; and mapping the one or more physical movements to the one or more predefined gestures based on motion information that corresponds to the one or more physical movements and one or more gesture models. ### Claim 4 (depends on 1) The method of claim 1, wherein modifying the one or more interactive data visualizations further comprises: determining one or more activation conditions for the interactive feature; and determining the interactive feature based on one or more coordinates associated with the one or more predefined gestures and the one or more activation conditions. ### Claim 5 (depends on 1) The method of claim 1, wherein updating the overlay in the composited video stream with the one or more modified interactive data visualizations further comprises: determining the interactive feature of the at least one of the one or more interactive data visualizations determining one or more actions associated with the interactive feature of the at least one of the one or more interactive data visualizations; and executing the one or more actions associated with the interactive feature of the at least one of the one or more interactive data visualizations. ### Claim 6 (depends on 1) The method of claim 1, further comprising: in response to detecting the one or more predefined gestures: in accordance with a determination that the one or more predefined gestures are directed to a first presentation object of the at least one of the one or more interactive data visualizations, executing one or more actions with respect to the first presentation object. ### Claim 7 A computer system having one or more processors and memory storing one or more programs, wherein the computer system is a first computer system connected to a second computer system via a network and the one or more programs are configured to be executed by the one or more processors, the one or more programs including instructions for: displaying one or more interactive data visualizations in a first user interface of a video conference application on the first computer system and in a second user interface of the video conference application on the second computer system connected to the first computer system via the network: receiving, via a video camera of the first computer system, a first video stream; displaying the first video stream in the second user interface of the video conference application on the second computer system; generating an overlay that includes the one or more interactive data visualizations; generating a composited video stream that includes the first video stream and the overlay; displaying the composited video stream in the second user interface of the video conference application on the second computer system, wherein the overlay is displayed in a foreground of the composited video stream and the first video stream is displayed in a background, relative to the overlay displayed in the foreground, of the composited video stream; detecting, via the video camera of the first computer system, one or more predefined gestures; and in response to detecting the one or more predefined gestures: in accordance with the one or more predefined gestures being associated with an interactive feature of at least one of the one or more interactive data visualizations; modifying the one or more interactive data visualizations; and updating the overlay in the composited video stream with the one or more modified interactive data visualizations. ### Claim 8 (depends on 7) The system of claim 7, wherein detecting the one or more predefined gestures further comprises: determining motion information that includes position information associated with one or more of hands, fingers, eyes, head, or posture of a user. ### Claim 9 (depends on 7) The system of claim 7, wherein detecting the one or more predefined gestures further comprises: tracking one or more physical movements of a user in the first video stream; and mapping the one or more physical movements to the one or more predefined gestures based on motion information that corresponds to the one or more physical movements and one or more gesture models. ### Claim 10 (depends on 7) The computer system of claim 7, wherein modifying the one or more interactive data visualizations further comprises: determining one or more activation conditions for the interactive feature; and determining the interactive feature based on one or more coordinates associated with the one or more predefined gestures and the one or more activation conditions. ### Claim 11 (depends on 7) The computer system of claim 7, wherein updating the overlay in the composited video stream with the one or more modified interactive data visualizations further comprises: determining the interactive feature of the at least one of the one or more interactive data visualizations; determining one or more actions associated with the interactive feature of the at least one of the one or more interactive data visualizations; and executing the one or more actions associated with the interactive feature of the at least one of the one or more interactive data visualizations. ### Claim 12 (depends on 7) The computer system of claim 7, wherein the one or more programs including instructions for: in response to detecting the one or more predefined gestures: in accordance with a determination that the one or more predefined gestures are directed to a first presentation object: executing one or more actions with respect to first presentation object. ### Claim 13 A non-transitory computer readable storage medium storing one or more programs comprising instructions which, when executed by a first computer system connected to a second computer system via a network, cause the first computer system to perform operations comprising: displaying one or more interactive data visualizations in a first user interface of a video conference application on a first computer system and in a second user interface of the video conference application on a second computer system connected to the first computer system via the network; receiving, via a video camera of the first computer system, a first video stream; displaying the first video stream in the second user interface of the video conference application on the second computer system; generating an overlay that includes the one or more interactive data visualizations; generating a composited video stream that includes the first video stream and the overlay; displaying the composited video stream in the second user interface of the video conference application on the second computer system, wherein the overlay is displayed in a foreground of the composited video stream and the first video stream is displayed in a background, relative to the overlay displayed in the foreground, of the composited video steam; detecting, via the video camera of the first computer system, one or more predefined gestures; and in response to detecting the one or more predefined gestures: in accordance with the one or more predefined gestures being associated with an interactive feature of at least one of the one or more interactive data visualizations: modifying the one or more interactive data visualizations; and updating the overlay in the composited video stream with the one or more modified interactive data visualizations. ### Claim 14 (depends on 13) The non-transitory computer readable storage medium of claim 13, wherein detecting the one or more predefined gestures further comprises: determining motion information from the first video stream, wherein the motion information includes position information associated with one or more of hands, fingers eyes, head, or posture of a user. ### Claim 15 (depends on 13) the non-transitory computer readable storage medium of claim 13, wherein detecting the one or more predefined gestures further comprises: tracking one or more physical movements of a user in the first video stream; and mapping the one or more physical movements to the one or more predefined gestures based on motion information that corresponds to the one or more physical movements and one or more gesture models. ### Claim 16 (depends on 13) The non-transitory computer readable storage medium of claim 13, wherein modifying the one or more interactive data visualizations further comprises: determining one or more activation conditions for the interactive feature; and determining the interactive feature based on one or more coordinates associated with the one or more predefined gestures and the one or more activation conditions. ### Claim 17 (depends on 13) The non-transitory computer readable storage medium of claim 13, wherein updating the overlay in the composited video stream with the one or more modified interactive data visualizations further comprises: determining the interactive feature of the at least one of the one or more interactive data visualizations; determining one or more actions associated with the interactive feature of the at least one of the one or more interactive data visualizations; and executing the one or more actions associated with the interactive feature of the at least one of the one or more interactive data visualizations. ### Claim 18 (depends on 13) The non-transitory computer readable storage medium of claim 13, wherein the operations further comprising: in response to detecting the one or more predefined gestures: in accordance with a determination that the one or more predefined gestures are directed a first presentation object of the at least one of the one or more interactive data visualizations, executing one or more actions with respect to the first presentation object.