# US 2021/0068907 A1 · Handheld User Interface Device for a Surgical Robot

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0068907 A1
- Document ID: US20210068907A1
- Status: Published application
- Kind: A1
- Application number: 16/566,336
- Filed: 2019-09-10
- Published: 2021-03-11
- Earliest claimed: 2019-09-10
- Assignees: Verb Surgical Inc.
- Applicants: Verb Surgical Inc.
- Inventors: Bernhard Adolf Fuerst; Pablo E. Garcia Kilroy
- Web page: https://patentgrep.com/patent/US20210068907A1

## CPC

- A61B 34/35: for telesurgery
- A61B 34/25: {User interfaces for surgical systems}
- A61B 34/32: operating autonomously
- A61B 34/74: {Manipulators with manual electric input means}
- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- A61B 90/50: Supports for surgical instruments, e.g. articulated arms
- A61B 17/00234: {for minimally invasive surgery (A61B17/0218, A61B17/0469, A61B17/12013, A61B17/1285, A61B17/29, A61B17/320016 take precedence)}
- A61B 2017/00477: {Coupling (A61B2017/0046 takes precedence)}
- A61B 2034/107: {Visualisation of planned trajectories or target regions}
- A61B 2034/2048: {using an accelerometer or inertia sensor}
- A61B 2034/2065: {Tracking using image or pattern recognition}
- A61B 2034/252: {indicating steps of a surgical procedure}
- A61B 2034/305: {Details of wrist mechanisms at distal ends of robotic arms}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/371: {with simultaneous use of two cameras}
- A61B 2090/372: {Details of monitor hardware}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)

## Abstract

Disclosed herein is a mobile interface device to control a robotically-assisted surgical system. The mobile interface device provides a surgeon with a direct view of the surgical robotic system and allows a surgeon to easily and intuitively select, control, or manipulate various target components of the surgical robotic system. The mobile interface device may capture a live image of the surgical robotic system to automatically identify a robotic arm that appears in the center of the captured live image as the target component selected by the surgeon. Based on the current pose or position of the target component, the mobile interface device may generate a list of target poses and control options. The surgeon may select a control option to select a target pose and to manipulate the target component to command a robotically-assisted movement of the selected robotic arm from the current pose to the target pose.

## Claims

### Claim 1

A mobile interface device for a surgical robotic system, comprising:
a camera configured to capture a live image of a portion of the surgical robotic system;
a processor configured to:
process the captured live image to identify a target component of the surgical robotic system to be controlled using the mobile interface device; and
determine a current pose of the target component;
a display configured to:
display the captured live image; and
display one or more target poses of the target component; and
one or more control interfaces configured to:
allow a user to select one of the target poses; and
generate an input command to drive the target component from the current pose to the selected target pose.

### Claim 2 (depends on 1)

The mobile interface of claim 1, wherein the display is further configured to present information to visually indicate the one or more target poses of the target component to the user.

### Claim 3 (depends on 1)

The mobile interface device of claim 1, wherein the processor is further configured to determine a plurality of command options for controlling the target component and the display is further configured to display the plurality of command options for the user to control the target component.

### Claim 4 (depends on 1)

The mobile interface device of claim 1, wherein the processor is further configured to process the captured live image to identify a second component of the surgical robotic system, and the display is further configured to display information to indicate a current pose of the second component as one of the target poses of the target component.

### Claim 5 (depends on 1)

The mobile interface device of claim 1, wherein the processor is further configured to determine a trajectory to drive the target component from the current pose to the selected target pose.

### Claim 6 (depends on 1)

The mobile interface device of claim 1, further comprising one or more sensors configured to measure a three-dimensional position relative to the interface device of a plurality of surface points of the surgical robotic system captured in the live image, and wherein the processor is further configured to process the plurality of measured three-dimensional positions of the surface points of the surgical robotic system captured in the live image to identify the target component.

### Claim 7 (depends on 1)

The mobile interface device of claim 1, wherein the camera comprises a 3-dimensional camera configured to scan a patient on an operating table of the surgical robotic system during a pre-operative setup, wherein the processor is further configured to process scanned information of the patient to determine a patient-specific movement of the target component.

### Claim 8 (depends on 1)

The mobile interface device of claim 1, wherein the one or more control interfaces is further configured to generate a second input command to freeze a movement of the target component when the target component moves off the captured live image.

### Claim 9 (depends on 1)

The mobile interface device of claim 1, wherein the display is further configured to display a visual indication of the target component to allow the user to confirm the target component.

### Claim 10 (depends on 1)

The mobile interface device of claim 1, wherein the display is a touch-screen display and wherein the touch-screen display is configured to allow the user to select a current pose of a second component of the surgical robotic system in the captured live image as a desired target pose of the target component.

### Claim 11

A method for controlling a surgical robotic system, comprising:
capturing, using a mobile device, a live image of a portion of the surgical robotic system;
identifying, by the mobile device or by a user, a target component of the surgical robotic system from the captured live image;
displaying, by the mobile device, the captured live image and one or more target poses of the target component;
receiving, by the mobile device from the user, a selected target pose from the one or more target poses displayed; and
generating, by the mobile device, an input command to drive the target component from a current pose to the selected target pose.

### Claim 12 (depends on 11)

The method of claim 11, wherein the displaying, by the mobile device, the one or more target poses of the target component comprises presenting information about the one or more target poses.

### Claim 13 (depends on 11)

The method of claim 11, further comprising:
generating, by the mobile device, a plurality of command options for controlling the target component; and
displaying, by the mobile device, the plurality of command options for the user to control the target component.

### Claim 14 (depends on 11)

The method of claim 11, further comprising:
identifying, by the mobile device, a second component of the surgical robotic system from the captured live image; and
displaying, by the mobile device, information to indicate a current pose of the second component as one of the target poses of the target component.

### Claim 15 (depends on 11)

The method of claim 11, further comprising:
determining, by the mobile device, a trajectory to drive the target component from the current pose to the selected target pose.

### Claim 16 (depends on 11)

The method of claim 11, further comprising:
scanning a patient on an operating table of the surgical robotic system during a pre-operative setup to obtain three-dimensional information of the patient; and
processing the three-dimensional information of the patient to determine a patient-specific movement of the target component.

### Claim 17 (depends on 11)

The method of claim 11, further comprising:
generating, by the mobile device, a second input command to freeze a movement of the target component when the target component moves off the captured live image.

### Claim 18 (depends on 11)

The method of claim 11, further comprising:
displaying, by the mobile device, a visual indication of the target component to allow the user to confirm the target component.

### Claim 19

A surgical robotic system, comprising:
an operating table;
a plurality of robotic arms coupled to the operating table;
a mobile interface device configured to:
capture a live image of the operating table or the plurality of robotic arms;
display a rendered image of the operating table or the plurality of robotic arms from the live image;
receive user input to control the operating table or one of the robotic arms; and
generate an input command to control the operating table or one of the robotic arms in response to the user input; and
a control tower communicatively coupled to the mobile interface device, the operating table and the plurality of robotic arms, wherein the control tower is configured to:
receive the input command from the mobile interface device; and
generate a control command to manipulate the operating table or one of the robotic arms in response to the input command.

### Claim 20 (depends on 19)

The surgical robotic system of claim 19, wherein the mobile interface device is further configured to present a plurality of input command options to control the operating table or one of the robotic arms.
