# US 11,439,478 B2 · Immersive three-dimensional display for robotic surgery

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,439,478 B2
- Document ID: US11439478B2
- Status: Granted
- Kind: B2
- Application number: 17/009,644
- Filed: 2020-09-01
- Published: 2020-12-24
- Granted: 2022-09-13
- Earliest claimed: 2016-10-03
- Assignees: Verb Surgical Inc.
- Applicants: Verb Surgical Inc.
- Inventors: Kent Anderson; Joan Savall; Brent Nobles; Allegra Shum; Pablo E. Garcia Kilroy; Karen Shakespear Koenig
- Web page: https://patentgrep.com/patent/US11439478B2

## CPC

- A61B 90/37: {Surgical systems with images on a monitor during operation}
- A61B 34/25: {User interfaces for surgical systems}
- A61B 34/30: Surgical robots
- A61B 34/35: for telesurgery
- A61B 34/74: {Manipulators with manual electric input means}
- A61B 90/50: Supports for surgical instruments, e.g. articulated arms
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04N 13/344: with head-mounted left-right displays
- A61B 2017/00199: {with a console, e.g. a control panel with a display}
- A61B 2017/00203: {with speech control or speech recognition}
- A61B 2017/00207: {with hand gesture control or hand gesture recognition}
- A61B 2017/00216: {with eye tracking or head position tracking control}
- A61B 2017/00973: {pedal-operated}
- A61B 2034/2055: {Optical tracking systems}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/372: {Details of monitor hardware}
- A61B 2090/502: {Headgear, e.g. helmet, spectacles}
- A61B 2090/508: {with releasable brake mechanisms}
- H04N 2213/001: Constructional or mechanical details
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An immersive display for use in a robotic surgical system includes a support arm, a housing mounted to the support arm and configured to engage with a face of the user, at least two eyepiece assemblies disposed in the housing and configured to provide a three-dimensional display, and at least one sensor, wherein the sensor enables operation of the robotic surgical system, and wherein the support arm is actuatable to move the housing for ergonomic positioning.

## Claims

### Claim 1

An immersive display for use in a robotic surgical system, comprising:
a support arm;
a housing mounted to the support arm and configured to provide a three-dimensional display of at least one image from an endoscopic camera used in the robotic surgical system;
at least one sensor configured to detect a head gesture of the user; and
a controller that configures the three-dimensional display to display a modified image from the endoscopic camera in response to the at least one sensor detecting an intentional head gesture of the user, wherein the controller configures the immersive display to maintain a relative spatial relationship of the housing and a hand of the user by providing visual cues to position the hand of the user at a target location in space.

### Claim 2 (depends on 1)

The immersive display of claim 1, wherein the controller modifies an image from the endoscopic camera based on input from a sensor or a control operating as a clutch.

### Claim 3 (depends on 2)

The immersive display of claim 2 wherein the input, from the sensor or the control that is operating as a clutch, is generated responsive to depressing a foot pedal or holding onto a handle with sensors.

### Claim 4 (depends on 1)

The immersive display of claim 1, wherein the housing is coupled to a contoured face frame configured to engage with a face of the user.

### Claim 5 (depends on 1)

The immersive display of claim 1, wherein the relative spatial relationship is a first relative spatial relationship, wherein the immersive display provides visual cues for maintaining a correspondence between i) the first relative spatial relationship of the housing and the at least one user hand position and ii) a second relative spatial relationship of the endoscopic camera and a surgical instrument.

### Claim 6 (depends on 1)

The immersive display of claim 1, wherein in response to the at least one sensor detecting a forward head gesture, the controller configures the three-dimensional display to display a zoomed-in image from the endoscopic camera, and wherein in response to the sensor detecting a backward head gesture, the three-dimensional display is configured to display a zoomed-out image from the endoscopic camera.

### Claim 7 (depends on 1)

The immersive display of claim 1, wherein in response to the sensor detecting a lateral head gesture, the controller configures three-dimensional display to display a panning image from the endoscopic camera, and wherein in response to the sensor detecting a tilting head gesture, the three-dimensional display is configured to display a tilting image from the endoscopic camera.

### Claim 8 (depends on 1)

The immersive display of claim 1, wherein the controller configures the three-dimensional display to display a graphical user interface and uses the at least one sensor to detect a head gesture to navigate the graphical user interface.

### Claim 9 (depends on 1)

The immersive display of claim 1, wherein the controller configures the three-dimensional display to display a visual representation of at least one of a user hand position and a user foot position relative to at least one target position.

### Claim 10 (depends on 1)

The immersive display of claim 1, further comprising at least one external camera coupled to the housing, wherein the camera is configured to provide at least one image of an environment external to the housing.

### Claim 11

A method for operating a robotic surgical system that includes a three-dimensional display, the method comprising:
providing at least one image from an endoscopic camera used in the robotic surgical system, for three-dimensional display to a user;
detecting using at least one sensor a head gesture of the user;
configuring the three-dimensional display to display a modified image from the endoscopic camera in response to detecting an intentional head gesture of the user; and
maintaining a relative spatial relationship of the three-dimensional display and a hand of the user by providing visual cues to position the hand of the user at a target location in space.

### Claim 12 (depends on 11)

The method of claim 11 wherein configuring the three-dimensional display is based on the user activating a clutch and leaning in, to command a camera view zoom in.

### Claim 13 (depends on 12)

The method of claim 12 further comprising configuring the three-dimensional display to display a further modified image from the endoscopic camera in response to detecting a further intentional head gesture of the user when the user leans out, to command a camera view zoom out.

### Claim 14 (depends on 11)

The method of claim 11 wherein configuring the three-dimensional display is based on the user activating a clutch and the user leaning out, to command a camera view zoom out.

### Claim 15 (depends on 11)

The method of claim 11 wherein configuring the three-dimensional display is based on the user activating a clutch and turning their head left or right to command a camera view pan left or a camera view pan right.

### Claim 16 (depends on 11)

The method of claim 11 wherein configuring the three-dimensional display is based on the user activating a clutch and tilting their head forward or backward to command a camera view tilt forward or a camera view tilt backward.

### Claim 17 (depends on 11)

The method of claim 11 wherein the user's hand is holding a handheld user interface devices, while simultaneously controlling the display of modified images from the endoscopic camera via their head gestures.

### Claim 18 (depends on 11)

The method of claim 11 further comprising configuring the three-dimensional display to display a graphical user interface and using the at least one sensor to detect a head gesture to navigate the graphical user interface.

### Claim 19 (depends on 1)

The immersive display of claim 1 further comprising an outward-facing illuminator, wherein the immersive display provides visual cues by configuring the outward-facing illuminator to project one or more reference indicators to highlight the target location in space.

### Claim 20 (depends on 1)

The immersive display of claim 1, wherein the immersive display provides visual cues by displaying a graphical representation on the three-dimensional display for where the user's hand is to be positioned with respect to a current housing location and orientation.

### Claim 21 (depends on 5)

The immersive display of claim 5, wherein the robotic surgical system comprises a handheld user interface device (UID) that is configured to control the surgical instrument, wherein the first and second relative spatial relationships enable the robotic surgical system to perform a teleoperation during which a relative position of the user's head and the user's hands that are each holding a handheld UID is substantially similar to a relative position of the endoscopic camera and the surgical instrument.

### Claim 22 (depends on 11)

The method of claim 11, wherein the relative spatial relationship is a first relative spatial relationship, wherein method further comprises configuring the three-dimensional display to provide guidance using the provided visual cues for maintaining a correspondence between i) the first relative spatial relationship of the three-dimensional display and user hand positions and ii) a second relative spatial relationship of the endoscopic camera and a surgical instrument.

### Claim 23 (depends on 11)

The method of claim 11, wherein the robotic surgical system further comprises an outward-facing illuminator, wherein the visual cues are provided by configuring the outward-facing illuminator to project one or more reference indicators to highlight user hand positions in space.

### Claim 24 (depends on 11)

The method of claim 11, wherein the visual cues are provided by displaying a graphical representation for where a user's hand is to be positioned with respect to a current location and orientation of the three-dimensional display.

### Claim 25 (depends on 22)

The method of claim 22, wherein the robotic surgical system comprises a handheld user interface device (UID) that is configured to control the surgical instrument, wherein the first and second relative spatial relationships enable the robotic surgical system to perform a teleoperation during which a relative position of the user's head and the user's hands that are each holding a handheld UID is substantially similar to a relative position of the endoscopic camera and the surgical instrument.