# US 2024/0189049 A1 · Systems and methods for point of interaction displays in a teleoperational assembly

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0189049 A1
- Document ID: US20240189049A1
- Status: Published application
- Kind: A1
- Application number: 18/537,354
- Filed: 2023-12-12
- Published: 2024-06-13
- Earliest claimed: 2017-08-10
- Assignees: Intuitive Surgical Operations, Inc.
- Applicants: Intuitive Surgical Operations, Inc.
- Inventors: Brandon D. Itkowitz; Timothy B. Hulford; Tabish Mustufa; Manqi Xu
- Web page: https://patentgrep.com/patent/US20240189049A1

## CPC

- A61B 34/25: {User interfaces for surgical systems}
- A61B 34/35: for telesurgery
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 34/37: Leader-follower robots (A61B34/35 takes precedence)
- G06F 3/1423: {controlling a plurality of local displays, e.g. CRT and flat panel display}
- G06F 3/147: using display panels
- A61B 2017/00973: {pedal-operated}
- A61B 2034/2051: {Electromagnetic tracking systems}
- A61B 2034/2055: {Optical tracking systems}
- A61B 2034/254: {being adapted depending on the stage of the surgical procedure}
- A61B 2034/256: {having a database of accessory information, e.g. including context sensitive help or scientific articles}
- A61B 2034/258: {providing specific settings for specific users}
- A61B 2034/741: {Glove like input devices, e.g. "data gloves"}
- A61B 2034/742: {Joysticks}
- A61B 2090/366: {using projection of images directly onto the body}
- A61B 2090/368: {changing the image on a display according to the operator's position}
- A61B 2090/371: {with simultaneous use of two cameras}
- A61B 2217/005: with suction drainage system
- A61B 2217/007: with irrigation system
- G09G 2380/08: Biomedical applications
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)

## Abstract

A teleoperational system in a surgical environment is provided. The system comprises a teleoperational assembly including a first teleoperational arm and a visual projection device coupled to the teleoperational assembly. The system further comprises a sensor and one or more processors. The one or more processors are configured to: receive first sensor information from the sensor; determine a first visual aid based upon the first sensor information; operate the visual projection device to project the first visual aid into the surgical environment; and operate the visual projection device to change the first visual aid to a second visual aid based on second sensor information received from the sensor.

## Claims

### Claim at position 1

1-47. (canceled)

### Claim 48

A teleoperational system in a surgical environment comprising:
a teleoperational assembly including a first teleoperational arm;
a visual projection device coupled to the teleoperational assembly;
a sensor; and
one or more processors configured to:
receive first sensor information from the sensor;
determine a first visual aid based upon the first sensor information;
operate the visual projection device to project the first visual aid into the surgical environment; and
operate the visual projection device to change the first visual aid to a second visual aid based on second sensor information received from the sensor.

### Claim 49 (depends on 48)

The teleoperational system of claim 48 wherein the sensor is a depth sensor.

### Claim 50 (depends on 48)

The teleoperational system of claim 48 wherein the first sensor information indicates an absence of an obstruction in a work zone of the teleoperational assembly.

### Claim 51 (depends on 50)

The teleoperational system of claim 50 wherein the first visual aid is a direction indicator for guidance of the teleoperational assembly into the work zone.

### Claim 52 (depends on 51)

The teleoperational system of claim 51 wherein the second sensor information indicates a presence of an operating table in the work zone and the second visual aid is a teleoperational assembly positioning indicator.

### Claim 53 (depends on 48)

The teleoperational system of claim 48 wherein at least one of a size of the second visual aid relative to an operating table or an orientation of the second visual aid relative to the operating table remains unchanged as the visual projection device moves with the teleoperational assembly.

### Claim 54 (depends on 48)

The teleoperational system of claim 48 wherein the teleoperational assembly includes an orienting platform to which the first teleoperational arm is coupled and wherein the second sensor information indicates selection of an orienting platform orientation mode, and wherein the second visual aid is an orienting platform orientation indicator.

### Claim 55 (depends on 48)

The teleoperational system of claim 48 wherein the second sensor information indicates a system error and wherein the second visual aid includes a warning or instructions for correcting the system error.

### Claim 56 (depends on 48)

The teleoperational system of claim 48 wherein the visual projection device is coupled to the first teleoperational arm and wherein the one or more processors are further configured to:
determine a change in a condition associated with the first teleoperational arm; and
operate the visual projection device based on the changed condition.

### Claim 57 (depends on 48)

The teleoperational system of claim 48 wherein responsive to the first sensor information indicating information about the first teleoperational arm, the first visual aid is directed toward the first teleoperational arm.

### Claim 58 (depends on 48)

The teleoperational system of claim 48 wherein the teleoperational assembly includes an orienting platform to which the first teleoperational arm is coupled and wherein an appearance of the first visual aid remains unchanged when the orienting platform is rotated.

### Claim 59

A method comprising:
receiving first sensor information from a sensor of a teleoperational system in a surgical environment;
determining a first visual aid based upon the first sensor information;
operating a visual projection device to project the first visual aid into the surgical environment, wherein the visual projection device is coupled to a teleoperational assembly of the teleoperational system;
receiving second sensor information from the sensor;
determining a second visual aid based upon the second sensor information; and
operating the visual projection device to change the first visual aid to the second visual aid.

### Claim 60 (depends on 59)

The method of claim 59 wherein the first sensor information is a first distance measurement from the sensor.

### Claim 61 (depends on 60)

The method of claim 60 wherein the first distance measurement exceeds a predetermined value, indicating an absence of an obstruction in a work zone of the teleoperational assembly.

### Claim 62 (depends on 61)

The method of claim 61 wherein the first visual aid is a direction indicator for guiding the teleoperational system into the work zone.

### Claim 63 (depends on 62)

The method of claim 62 wherein the second sensor information is a second distance measurement from the sensor that is less than the predetermined value, indicating a presence of an operating table in the work zone and wherein the second visual aid is a teleoperational assembly positioning indicator.

### Claim 64 (depends on 59)

The method of claim 59 wherein a size of the second visual aid relative to an operating table remains unchanged as the visual projection device moves with the teleoperational assembly.

### Claim 65 (depends on 59)

The method of claim 59 wherein an orientation of the second visual aid relative to an operating table remains unchanged as the visual projection device moves with the teleoperational assembly.

### Claim 66 (depends on 59)

The method of claim 59 further comprising:
responsive to the first sensor information indicating information about a first teleoperational arm of the teleoperational assembly, projecting the first visual aid toward the first teleoperational arm.

### Claim 67 (depends on 66)

The method of claim 66 further comprising:
responsive to the second sensor information indicating information about a second teleoperational arm of the teleoperational assembly, projecting the second visual aid toward the second teleoperational arm.
