# US 11,992,273 B2 · System and method of displaying images from imaging devices > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,992,273 B2 - Document ID: US11992273B2 - Status: Granted - Kind: B2 - Application number: 17/265,486 - Filed: 2019-08-02 - Published: 2021-10-14 - Granted: 2024-05-28 - Earliest claimed: 2018-08-03 - Assignees: Intuitive Surgical Operations, Inc. - Applicants: Intuitive Surgical Operations, Inc. - Inventors: Gabriel F. Brisson - Web page: https://patentgrep.com/patent/US11992273B2 ## CPC - A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis - H04N 13/344: with head-mounted left-right displays - A61B 1/00193: {adapted for stereoscopic vision} - A61B 90/37: {Surgical systems with images on a monitor during operation} - G09G 5/00: Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators (image data processing or generation, in general G06T) - H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00) - A61B 34/30: Surgical robots - A61B 2034/2048: {using an accelerometer or inertia sensor} - A61B 2034/2055: {Optical tracking systems} - A61B 2090/376: {using X-rays, e.g. fluoroscopy} - A61B 2090/378: {using ultrasound} - G09G 2340/0464: Positioning - G09G 2354/00: Aspects of interface with display user - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) ## Abstract A system and method of displaying images from an imaging device includes a display device and one or more processors. The one or more processors are configured to track movement of an imaging device, determine a display-device-to-imaging-window transform based on the tracked movement and a corresponding command to move the imaging device, and render an image received from the imaging device on the display device at a position and orientation on the display device based on the display-device-to-imaging-window transform. In some embodiments, the display-device-to-imaging-window transform is determined based on at least one of a difference between: a commanded and an actual position of the imaging device, a commanded and an actual orientation of the imaging device, a commanded and an estimated position of the imaging device, or a commanded and an estimated orientation of the imaging device. ## Claims ### Claim 1 A computer-assisted device comprising: a display device; and one or more processors coupled to the display device, the one or more processors being configured to: track movement of an imaging device to determine an actual position of the imaging device or an actual orientation of the imaging device; determine a display-device-to-imaging-window transform based on a difference, the difference selected from a group consisting of: a difference between a commanded position of the imaging device and the actual position of the imaging device, and a difference between a commanded orientation of the imaging device and the actual orientation of the imaging device; and display an imaging window on the display device at a position and orientation on the display device based on the display-device-to-imaging-window transform; and render an image from the imaging device in the imaging window. ### Claim 2 (depends on 1) The computer-assisted device of claim 1, wherein the one or more processors are further configured to: determine a command for moving the imaging device based on detected motion of the display device; and command movement of the imaging device based on the determined command. ### Claim 3 (depends on 2) The computer-assisted device of claim 2, wherein the display-device-to-imaging-window transform is determined based on a display device transform, wherein the display device transform is based on the detected motion of the display device. ### Claim 4 (depends on 1) The computer-assisted device of claim 1, wherein the one or more processors are further configured to determine an imaging device transform based on the tracked movement of the imaging device wherein the imaging device transform is determined based on a position and an orientation of a view plane of the imaging device relative to the imaging device. ### Claim 5 (depends on 1) The computer-assisted device of claim 1, wherein the one or more processors are further configured to: determine a display device transform based on detected movement of the display device; determine an imaging device transform based on the tracked movement of the imaging device; and determine the display-device-to-imaging-window transform based on the display device transform and the imaging device transform. ### Claim 6 (depends on 1) The computer-assisted device of claim 1, wherein the one or more processors are further configured to project the image onto the imaging window. ### Claim 7 (depends on 1) The computer-assisted device of claim 1, wherein a direction of view of the imaging device is at an angle relative to an insertion axis of the imaging device into a workspace. ### Claim 8 (depends on 1) The computer-assisted device of claim 1, wherein the display device comprises a tablet, a smart phone, or a head-mounted display. ### Claim 9 (depends on 1) The computer-assisted device of claim 1, wherein: the display device is a stereoscopic device; and the imaging device is a stereoscopic device; and the one or more processors being configured to render the image received from the imaging device includes rendering left and right images received from the imaging device. ### Claim 10 (depends on 1) The computer-assisted device of claim 1, wherein: the display device has six degrees of freedom; and the imaging device has four degrees of freedom. ### Claim 11 (depends on 1) The computer-assisted device of claim 1, wherein the position and orientation of the imaging window provides one or more cues to an operator regarding at least one of: an inability of the imaging device to follow a corresponding command to move the imaging device, a range of motion limit of the imaging device, or a mismatch in degrees of freedom between the imaging device and the display device. ### Claim 12 A method comprising: tracking movement of an imaging device to determine an actual position of the imaging device or an actual orientation of the imaging device; determining a display-device-to-imaging-window transform based on a difference, the difference selected from the group consisting of: a difference between a commanded position of the imaging device and the actual position of the imaging device, and a difference between a commanded orientation of the imaging device and the actual orientation of the imaging device; and displaying an imaging window on a display device at a position and orientation on the display device based on the display-device-to-imaging-window transform; and rendering an image from the imaging device in the imaging window. ### Claim 13 (depends on 12) The method of claim 12, further comprising: determining a command for moving the imaging device based on detected motion of the display device; and commanding movement of the imaging device based on the determined command. ### Claim 14 (depends on 12) The method of claim 12, further comprising: determining a display device transform based on detected movement of the display device; determining an imaging device transform based on the tracked movement of the imaging device; and determining the display-device-to-imaging-window transform based on the display device transform and the imaging device transform. ### Claim 15 (depends on 12) The method of claim 12, wherein rendering the image received from the imaging device comprises projecting the image onto the imaging window. ### Claim 16 (depends on 12) The method of claim 12, wherein the position and orientation of the imaging window provides one or more cues to an operator regarding at least one of: an inability of the imaging device to follow a corresponding command to move the imaging device, a range of motion limit of the imaging device, or a mismatch in degrees of freedom between the imaging device and the display device. ### Claim 17 A non-transitory machine-readable medium comprising a plurality of machine-readable instructions which when executed by one or more processors are adapted to cause the one or more processors to perform a method comprising: tracking movement of an imaging device to determine: an actual position of the imaging device or an actual orientation of the imaging device; determining a display-device-to-imaging-window transform based on a difference, the difference selected from the group consisting of: a difference between a commanded position of the imaging device and the actual position of the imaging device, and a difference between a commanded orientation of the imaging device and the actual orientation of the imaging device; and displaying an imaging window on a display device at a position and orientation on the display device based on the display-device-to-imaging-window transform; and rendering an image from the imaging device in the imaging window. ### Claim 18 (depends on 17) The non-transitory machine-readable medium of claim 17, wherein the method further comprises: determining a command for moving the imaging device based on detected motion of the display device; and commanding movement of the imaging device based on the determined command. ### Claim 19 (depends on 17) The non-transitory machine-readable medium of claim 17, wherein the method further comprises: determining a display device transform based on detected movement of the display device; determining an imaging device transform based on the tracked movement of the imaging device; and determining the display-device-to-imaging-window transform based on the display device transform and the imaging device transform. ### Claim 20 (depends on 17) The non-transitory machine-readable medium of claim 17, wherein rendering the image received from the imaging device on the display device comprises rendering the image on an imaging window displayed on the display device, and the method further comprises: displaying the imaging window at a position and an orientation on the display device based on the display-device-to-imaging-window transform.