# US 2021/0243427 A1 · Compensation for observer movement in robotic surgical systems having stereoscopic displays

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0243427 A1
- Document ID: US20210243427A1
- Status: Published application
- Kind: A1
- Application number: 17/048,896
- Filed: 2019-04-01
- Published: 2021-08-05
- Earliest claimed: 2018-04-20
- Assignees: Covidien LP
- Applicants: Covidien LP
- Inventors: Dwight Meglan; Albert Dvornik
- Web page: https://patentgrep.com/patent/US20210243427A1

## CPC

- A61B 1/00009: {of image signals during a use of endoscope}
- A61B 1/00042: {for mechanical operation}
- A61B 1/0005: {combining images e.g. side-by-side, superimposed or tiled}
- A61B 1/00149: {using articulated arms}
- A61B 1/00193: {adapted for stereoscopic vision}
- A61B 34/30: Surgical robots
- A61B 90/37: {Surgical systems with images on a monitor during operation}
- H04N 13/344: with head-mounted left-right displays
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/371: {with simultaneous use of two cameras}
- G02B 2027/0134: {of stereoscopic type}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0181: {Adaptation to the pilot/driver}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS

## Abstract

Systems and methods for compensating for observer movement during robotic surgery. An exemplary system includes an image capture device configured to capture images of a surgical site, a stereoscopic display device, a sensor configured to detect positions of an observer, and a computing device including a processor and a memory storing instructions. The instructions, when executed by the processor, cause the computing device to receive the images of the surgical site from the image capture device, receive data from the sensor indicating a position of the observer, process the received images of the surgical site based on the movement of the observer, and cause the stereoscopic display device to display the processed images of the surgical site.

## Claims

### Claim 1

A robotic surgical system with observer movement compensation, the system comprising:
an image capture device configured to capture images of a surgical site;
a stereoscopic display device;
a sensor configured to detect positions of an observer; and
a computing device including at least one processor and a memory storing instructions which, when executed by the at least one processor, cause the computing device to:
receive the images of the surgical site from the image capture device,
receive data from the sensor indicating a first position of the observer,
process the received images of the surgical site based on the first position of the observer, and
cause the stereoscopic display device to display the processed images of the surgical site.

### Claim 2 (depends on 1)

The system according to claim 1, wherein the images of the surgical site include left-eye image data and right-eye image data.

### Claim 3 (depends on 1)

The system according to claim 1, wherein the images of the surgical site have a frame size, and
wherein the processing the received images of the surgical site based on the first position of the observer includes determining a portion of the images to display based on the first position of the observer, the portion of the images to display being smaller than the frame size.

### Claim 4 (depends on 3)

The system according to claim 3, wherein the portion of the images to display corresponds to a number of pixels less than the number of pixels included in the images of the surgical site.

### Claim 5 (depends on 3)

The system according to claim 3, wherein the determining the portion of the images to display includes cropping at least a portion of the images.

### Claim 6 (depends on 3)

The system according to claim 3, wherein the determining the portion of the images to display includes shifting at least a portion of the images.

### Claim 7 (depends on 6)

The system according to claim 6, wherein the shifting at least the portion of the images includes:
determining a vector of movement of the observer based on the first position of the ob server;
determining a direction and an amount of pixels to shift based on the determined vector of movement of the observer; and
shifting at least a portion of the images based on the determined direction and amount of pixels to shift.

### Claim 8 (depends on 7)

The system according to claim 7, wherein the determining the vector of movement of the observer further includes determining a degree of movement, and
wherein the determining the direction and the amount of pixels to shift is further based on the determined degree of movement.

### Claim 9 (depends on 7)

The system according to claim 7, wherein the determining the direction and the amount of pixels to shift is further based on a relationship between the vector of movement of the observer and the direction and amount of pixels to shift.

### Claim 10 (depends on 9)

The system according to claim 9, wherein the relationship is based on a table.

### Claim 11 (depends on 9)

The system according to claim 9, wherein the relationship is based on a threshold.

### Claim 12 (depends on 7)

The system according to claim 7, wherein the determining the vector of movement of the observer includes:
determining whether the first position of the observer approaches a maximum threshold; and
providing an alert indicating that the first position of the observer approaches the maximum threshold.

### Claim 13 (depends on 7)

The system according to claim 7, wherein the determining the vector of movement of the observer includes:
determining whether the first position of the observer exceeds a maximum threshold; and
providing an alert indicating that the first position of the observer exceeds the maximum threshold.

### Claim 14 (depends on 1)

The system according to claim 1, further comprising:
a surgical robot, wherein the image capture device is coupled to the surgical robot, and
wherein the instructions, when executed by the at least one processor, further cause the computing device to:
determine a vector of movement of the observer based on the data received from the sensor; and
cause the surgical robot to reposition the image capture device based on the determined vector of movement of the observer.

### Claim 15 (depends on 1)

The system according to claim 1, wherein the stereoscopic display is an autostereoscopic display.

### Claim 16 (depends on 1)

The system according to claim 1, wherein the stereoscopic display is a passive stereoscopic display, and the system further comprises three-dimensional (3D) glasses worn by the observer.

### Claim 17 (depends on 16)

The system according to claim 16, wherein the 3D glasses cause a left-eye image to be displayed to a left eye of the observer, and a right-eye image to be displayed to a right eye of the ob server.

### Claim 18 (depends on 1)

The system according to claim 1, wherein the image capture device is a stereoscopic camera coupled to an endoscope.

### Claim 19 (depends on 1)

The system according to claim 1, wherein the sensor is a motion sensor.

### Claim 20 (depends on 1)

The system according to claim 1, wherein the sensor is a camera.

### Claim 21 (depends on 20)

The system according to claim 20, wherein the data received from the sensor indicating a first position of the observer includes an image of the observer, and
wherein the instructions, when executed by the at least one processor, further cause the computing device to:
generate second image data based on the image of the observer; and
detect the first position of the observer by processing the second image data.

### Claim 22 (depends on 21)

The system according to claim 21, wherein the detecting the first position of the observer includes detecting one or more of:
a distance of the observer relative to a vector normal to the stereoscopic display;
a direction of the observer relative to the vector normal to the stereoscopic display; or
an orientation of the observer relative to the stereoscopic display.

### Claim 23 (depends on 22)

The system according to claim 22, wherein the direction of the observer is one or more of a lateral direction or a vertical direction.

### Claim 24 (depends on 1)

The system according to claim 1, wherein the instructions, when executed by the at least one processor, further cause the computing device to:
receive additional data from the sensor indicating a second position of the observer;
process the received images of the surgical site based on the second position of the observer, and
cause the stereoscopic display device to display the processed images of the surgical site.

### Claim 25

A method for compensating for observer movement in a robotic surgical system, the method comprising:
receiving images of a surgical site from an image capture device;
receiving data from a sensor indicating a first position of an observer;
processing the received images of the surgical site based on the first position of the observer; and
causing a stereoscopic display device to display the processed images of the surgical site.

### Claim 26

A non-transitory computer-readable storage medium storing a program for compensating for observer movement in a robotic surgical system, the program including instructions which, when executed by a processor, cause a computing device to:
receive images of a surgical site from an image capture device;
receive data from a sensor indicating a first position of an observer;
process the received images of the surgical site based on the first position of the observer; and
cause a stereoscopic display device to display the processed images of the surgical site.
