# US 11,992,934 B2 · Stereo video in augmented reality > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,992,934 B2 - Document ID: US11992934B2 - Status: Granted - Kind: B2 - Application number: 17/961,423 - Filed: 2022-10-06 - Published: 2023-01-26 - Granted: 2024-05-28 - Earliest claimed: 2021-01-13 - Assignees: Medivis, Inc. - Applicants: MediVis, Inc. - Inventors: Long Qian; Christopher Morley; Osamah Choudhry - Web page: https://patentgrep.com/patent/US11992934B2 ## CPC - B25J 13/08: by means of sensing devices, e.g. viewing or touching devices - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - B25J 9/1635: {flexible-arm control} - G06F 3/012: {Head tracking input arrangements} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G09B 23/28: for medicine - G09B 23/285: {for injections, endoscopy, bronchoscopy, sigmoidscopy, insertion of contraceptive devices or enemas} - G09B 23/30: Anatomical models {(G09B23/281 - G09B23/288 take precedence)} - G06T 2210/41: Medical - G06T 2219/028: Multiple view windows (top-side-front-sagittal-orthogonal) - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) ## Abstract Various embodiments of an apparatus, methods, systems and computer program products described herein are directed to an Interaction Engine. According to various embodiments, Interaction Engine generates within a unified three-dimensional (3D) coordinate space, a virtual 3D medical model positioned according to a model pose. The Interaction Engine receives video data from a plurality of video sources. The Interaction Engine renders a first Augmented Reality (AR) display that includes concurrent display of the virtual 3D medical model and visualization of at least a portion of video data from a first video source. The Interaction Engine renders a second Augmented Reality (AR) display that includes concurrent display of the virtual 3D medical model and visualization of at least a portion of video data from a second video source. ## Claims ### Claim 1 A system comprising one or more processors, and a non-transitory computer-readable medium including one or more sequences of instructions that, when executed by the one or more processors, cause the system to perform operations comprising: generating within a unified three-dimensional (3D) coordinate space, a virtual 3D medical model positioned according to a model pose; receiving video data from a plurality of video sources; rendering a first Augmented Reality (AR) display that includes concurrent display of the virtual 3D medical model and visualization of at least a portion of video data from a first video source; rendering a second AR display that includes concurrent display of the virtual 3D medical model and visualization of at least a portion of video data from a second video source; detecting a first physical gesture indicative of identifying a selection to initiate a type of virtual interaction from a plurality of types of virtual interactions based on the detected first physical gesture; concurrently modifying the first AR display and the second AR display, via the AR headset device, by (a) initiating display of a selectable first virtual object in the first AR display that corresponds with the selected type of virtual interaction and by (b) initiating display of a selectable second virtual object in the second AR display that corresponds with the selected type of virtual interaction; wherein rendering the first AR display further comprises: rendering a first virtual screen portraying the visualization of the video from the first video source concurrently with display of the first virtual object; and wherein rendering the second AR display further comprises: rendering a second virtual screen portraying the visualization of the video from the second video source concurrently with display of the second virtual object, the first and the second virtual objects each comprising an instance of a same type of virtual object. ### Claim 2 (depends on 1) The system of claim 1, further comprising: wherein rendering the first AR display comprises: rendering the first AR display with respect to a first lens of an AR headset device; and wherein rendering the second AR display comprises: rendering the second AR display with respect to a second lens of the AR headset device. ### Claim 3 (depends on 2) The system of claim 2, further comprising: detecting the first physical gesture; concurrently modifying the first and the second AR display, via the AR headset device, by rendering in the first and the second AR display a type of movement of a virtual 3D body part representation based on the detected first physical gesture; identifying selection of a type of virtual interaction to be initiated based on the type of movement of the virtual 3D body part representation; and concurrently modifying the first and the second AR display, via the AR headset device, by initiating display in the first and the second AR of a selectable virtual object that corresponds with the selected type of virtual interaction. ### Claim 4 (depends on 1) The system of claim 1, wherein rendering a respective AR display comprises: rendering, in each respective AR display, controls comprising at least one of: one or more camera parameters and image parameters and robotic arm control. ### Claim 5 (depends on 4) The system of claim 4, further comprising: wherein receiving video data comprises: receiving video data based on visual imagery captured by one of a first endoscope and a first exoscope; receiving one or more instances of virtual coordinates based on respective physical poses of one of the first endoscope and the first exoscope. ### Claim 6 A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to: generating within a unified three-dimensional (3D) coordinate space, a virtual 3D medical model positioned according to a model pose; receiving video data from a plurality of video sources; rendering a first Augmented Reality (AR) display that includes concurrent display of the virtual 3D medical model and visualization of at least a portion of video data from a first video source; rendering a second AR display that includes concurrent display of the virtual 3D medical model and visualization of at least a portion of video data from a second video source; detecting a first physical gesture indicative of a selection to initiate a type of virtual interaction from a plurality of types of virtual interactions based on the detected first physical gesture; concurrently modifying the first AR display and the second AR display, via the AR headset device, by (a) initiating display of a selectable first virtual object in the first AR display that corresponds with the selected type of virtual interaction and by (b) initiating display of a selectable second virtual object in the second AR display that corresponds with the selected type of virtual interaction; wherein rendering the first AR display further comprises: rendering a first virtual screen portraying the visualization of the video from the first video source concurrently with display of the first virtual object; and wherein rendering the second AR display further comprises: rendering a second virtual screen portraying the visualization of the video from the second video source concurrently with display of the second virtual object, the first and the second virtual objects each comprising an instance of a same type of virtual object. ### Claim 7 (depends on 6) The computer program product of claim 6, further comprising: detecting the first physical gesture; concurrently modifying the first and the second AR display, via the AR headset device, by rendering in the first and the second AR display a type of movement of a virtual 3D body part representation based on the detected first physical gesture; identifying selection of a type of virtual interaction to be initiated based on the type of movement of the virtual 3D body part representation. ### Claim 8 (depends on 7) The computer program product of claim 7, further comprising: detecting a second physical gesture based on a first change of pose data of the AR headset device, the first change of pose data representing a change of a position and orientation of the AR headset device in the unified 3D coordinate space; determining a second display position of the selectable virtual object based on the first change of pose data of the AR headset device, the selectable virtual object comprising a virtual 3D clipping plane; determining a first intersection between the virtual 3D clipping plane at the second display position and the rendered virtual 3D medical model at a current model pose; identifying one or more portions of non-rendered 3D medical model data for the virtual 3D medical model in the current model pose that map to the first intersection; concurrently modifying the first and the second AR display, via the AR headset device, by rendering in the first and the second AR display the virtual clipping plane at the second display position; and wherein concurrently modifying the first and the second AR display comprises concurrently rendering: (i) a first instance of the virtual clipping plane at the second display position in the first AR display and a first instance of the virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the first intersection while the first AR display renders visualization of video from the first video source in the first virtual screen; and (ii) a second instance of the virtual clipping plane at the second display position in the second AR display and a second instance of the virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the first intersection while the second AR display renders visualization of video from the second video source in the second virtual screen.