# US 2021/0343088 A1 · Mixed reality systems and methods for indicating an extent of a field of view of an imaging device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0343088 A1
- Document ID: US20210343088A1
- Status: Published application
- Kind: A1
- Application number: 17/286,782
- Filed: 2019-10-24
- Published: 2021-11-04
- Earliest claimed: 2018-10-26
- Assignees: Intuitive Surgical Operations, Inc.
- Applicants: Intuitive Surgical Operations, Inc.
- Inventors: Govinda Payyavula; Cortney Jansen; Huan Lac Phan; Simon P. DiMaio
- Web page: https://patentgrep.com/patent/US20210343088A1

## CPC

- A61B 1/00006: {of control signals}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- A61B 1/00045: {Display arrangement}
- A61B 1/0005: {combining images e.g. side-by-side, superimposed or tiled}
- A61B 1/05: characterised by the image sensor, e.g. camera, being in the distal end portion
- A61B 1/0676: {at distal tip of an endoscope}
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 34/30: Surgical robots
- A61B 2034/2057: {Details of tracking cameras}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A mixed reality presentation system identifies an operating condition associated with an operation performed on a body while an active imaging device captures imagery of an internal view of the body. The mixed reality presentation system also determines, based on the identified operating condition, that a display device is to toggle a display of a shape overlay that is displayed together with an external view of the body and that is indicative of an extent of a field of view of the active imaging device relative to the body. Based on the determining that the display device is to toggle the display of the shape overlay, the mixed reality presentation system directs the display device to toggle the display of the shape overlay. Corresponding systems and methods are also disclosed.

## Claims

### Claim 1

A system comprising:
a memory storing instructions; and
a processor communicatively coupled to the memory and configured to execute the instructions to:
identify an operating condition associated with an operation performed on a body while an active imaging device captures imagery of an internal view of the body,
determine, based on the identified operating condition, that a display device is to toggle a display of a shape overlay that is displayed together with an external view of the body, the shape overlay indicative of an extent of a field of view of the active imaging device relative to the body, and
based on the determining that the display device is to toggle the display of the shape overlay, direct the display device to toggle the display of the shape overlay.

### Claim 2 (depends on 1)

The system of claim 1, wherein:
the processor is further configured to execute the instructions to determine, in response to the identifying of the operating condition, a current display status of the shape overlay; and
the directing of the display device to toggle the display of the shape overlay comprises:
directing the display device to begin displaying the shape overlay together with the external view if the current display status indicates that the shape overlay is not displayed when the operating condition is identified, or
directing the display device to cease displaying the shape overlay together with the external view if the current display status indicates that the shape overlay is displayed when the operating condition is identified.

### Claim 3

(canceled)

### Claim 4 (depends on 1)

The system of claim 1, wherein the identifying of the operating condition comprises detecting a position of a distal end of the active imaging device in relation to an internal area of the body from which the active imaging device captures the imagery of the internal view.

### Claim 5 (depends on 1)

The system of claim 1, wherein the identifying of the operating condition comprises detecting an operational status of at least one of:
an image sensor included in the active imaging device,
a communication link by way of which the active imaging device provides data associated with the imagery, and
a light source associated with the active imaging device and configured to illuminate an internal area of the body from which the active imaging device captures the imagery.

### Claim 6 (depends on 1)

The system of claim 1, wherein the identifying of the operating condition comprises determining a status of an object insertion process in which an object is inserted into an internal area of the body.

### Claim 7 (depends on 6)

The system of claim 6, wherein:
the determining of the status of the object insertion process comprises detecting that the object is visible within the field of view; and
the directing of the display device to toggle the display comprises directing the display device to cease displaying the shape overlay based on the detection that the object is visible within the field of view.

### Claim 8 (depends on 1)

The system of claim 1, wherein the processor is further configured to execute the instructions to direct the display device to display, together with the external view of the body and the shape overlay, a simulated depiction of an internal portion of the body, the simulated depiction displayed so as to appear to be behind the shape overlay.

### Claim at position 9

9-12. (canceled)

### Claim 13 (depends on 1)

The system of claim 1, wherein the processor is further configured to execute the instructions to:
determine that the display device is to display an auxiliary shape overlay together with the external view of the body, the auxiliary shape overlay indicative of an extent, relative to the body, of a potential field of view of an auxiliary imaging device that is distinct from the active imaging device; and
based on the determining that the display device is to display the auxiliary shape overlay, direct the display device to display the auxiliary shape overlay together with the external view of the body.

### Claim 14 (depends on 1)

The system of claim 1, wherein the processor is further configured to execute the instructions to:
determine that the display device is to display an auxiliary shape overlay together with the external view of the body, the auxiliary shape overlay indicative of an extent, relative to the body, of a potential field of view of the active imaging device, the potential field of view distinct from the field of view; and
based on the determining that the display device is to display the auxiliary shape overlay, direct the display device to display the auxiliary shape overlay together with the external view of the body.

### Claim 15 (depends on 14)

The system of claim 14, wherein the directing of the display device to display the auxiliary shape overlay comprises directing the display device to concurrently display the shape overlay and the auxiliary shape overlay.

### Claim 16 (depends on 15)

The system of claim 15, wherein the potential field of view indicated by the auxiliary shape overlay corresponds to a previous position at which the active imaging device was located prior to the displaying of the auxiliary shape overlay.

### Claim 17 (depends on 14)

The system of claim 14, wherein the directing of the display device to display the auxiliary shape overlay comprises directing the display device to abstain from displaying the shape overlay while displaying the auxiliary shape overlay.

### Claim 18 (depends on 14)

The system of claim 14, wherein:
the active imaging device is configured to provide the captured imagery at different zoom levels, the field of view corresponds to a first zoom level, and the potential field of view corresponds to a second zoom level distinct from the first zoom level; or
the active imaging device is configured to capture the imagery of the internal view from different viewing angles, the field of view corresponds to a first viewing angle, and the potential field of view corresponds to a second viewing angle distinct from the first viewing angle; or
the active imaging device comprises a pair of stereoscopic image sensors, the field of view corresponds to a first image sensor in the pair of stereoscopic image sensors, and the potential field of view corresponds to a second image sensor in the pair of stereoscopic image sensors; or
the active imaging device is configured to capture the imagery of the internal view by way of one imaging technology at a time from a plurality of imaging technologies, the field of view corresponds to a first imaging technology in the plurality of imaging technologies, and the potential field of view corresponds to a second imaging technology in the plurality of imaging technologies.

### Claim at position 16

19-21. (canceled)

### Claim 22 (depends on 1)

The system of claim 1, wherein:
the identifying of the operating condition comprises detecting an initiation, during performance of the operation, of a process to swap out the active imaging device for an additional imaging device;
the directing of the display device to toggle the display of the shape overlay comprises beginning to display the shape overlay at a static position with respect to the body; and
the processor is further configured to execute the instructions to:
determine, subsequent to the beginning to display the shape overlay at the static position, that the display device is to further display an additional shape overlay together with the shape overlay and the external view of the body, the additional shape overlay indicative of an extent, relative to the body, of an additional field of view corresponding to the additional imaging device as the process to swap out the active imaging device for the additional imaging device is performed,
based on the determining that the display device is to display the additional shape overlay, direct the display device to display the additional shape overlay together with the shape overlay and the external view of the body, and
direct the display device to persist in displaying the shape overlay at the static position at least until the additional shape overlay overlaps with the shape overlay at the static position.

### Claim 23 (depends on 1)

The system of claim 1, wherein:
the identifying of the operating condition comprises detecting an initiation, during performance of the operation, of a process to move the active imaging device from a first port into the body by way of which the active imaging device captures the imagery of the internal view to a second port into the body;
the directing of the display device to toggle the display of the shape overlay comprises beginning to display the shape overlay at a static position with respect to the body; and
the processor is further configured to execute the instructions to:
determine, subsequent to the beginning to display the shape overlay at the static position, that the display device is to further display an additional shape overlay together with the shape overlay and the external view of the body, the additional shape overlay indicative of an extent, relative to the body, of an additional field of view corresponding to the field of view of the active imaging device as the process to move the active imaging device is performed,
based on the determining that the display device is to display the additional shape overlay, direct the display device to display the additional shape overlay together with the shape overlay and the external view of the body, and
direct the display device to persist in displaying the shape overlay at the static position at least until the additional shape overlay overlaps with the shape overlay at the static position.

### Claim 24 (depends on 1)

The system of claim 1, wherein:
the identifying of the operating condition comprises detecting an initiation, during performance of the operation, of a process to move the active imaging device from being attached to a first manipulator arm to being attached to a second manipulator arm;
the directing of the display device to toggle the display of the shape overlay comprises beginning to display the shape overlay at a static position with respect to the body; and
the processor is further configured to execute the instructions to:
determine, subsequent to the beginning to display the shape overlay at the static position, that the display device is to further display an additional shape overlay together with the shape overlay and the external view of the body, the additional shape overlay indicative of an extent, relative to the body, of an additional field of view corresponding to the field of view of the active imaging device as the process to move the active imaging device is performed,
based on the determining that the display device is to display the additional shape overlay, direct the display device to display the additional shape overlay together with the shape overlay and the external view of the body, and
direct the display device to persist in displaying the shape overlay at the static position at least until the additional shape overlay overlaps with the shape overlay at the static position.

### Claim 25

A method comprising:
identifying, by a mixed reality presentation system, an operating condition associated with an operation performed on a body while an active imaging device captures imagery of an internal view of the body;
determining, by the mixed reality presentation system and based on the identified operating condition, that a display device is to toggle a display of a shape overlay that is displayed together with an external view of the body, the shape overlay indicative of an extent of a field of view of the active imaging device relative to the body; and
directing, by the mixed reality presentation system and based on the determining that the display device is to toggle the display of the shape overlay, the display device to toggle the display of the shape overlay.

### Claim at position 21

26-27. (canceled)

### Claim 28 (depends on 25)

The method of claim 25, wherein the identifying of the operating condition comprises detecting a position of a distal end of the active imaging device in relation to an internal area of the body from which the active imaging device captures the imagery of the internal view.

### Claim 29 (depends on 25)

The method of claim 25, wherein the identifying of the operating condition comprises detecting at least one status selected from the group consisting of:
an operational status of an image sensor included in the active imaging device, an operational status of a communication link by way of which the active imaging device provides data associated with the imagery,
an operational status of a light source associated with the active imaging device and configured to illuminate an internal area of the body from which the active imaging device captures the imagery, and
a status of an object insertion process in which an object is inserted into an internal area of the body.

### Claim 30

(canceled)

### Claim 31 (depends on 25)

The method of claim 25, wherein:
the identifying of the operating condition comprises detecting that the object is visible within the field of view; and
the directing of the display device to toggle the display comprises directing the display device to cease displaying the shape overlay based on the detection that the object is visible within the field of view.

### Claim 32 (depends on 25)

The method of claim 25, further comprising:
determining, by the mixed reality presentation system, that the display device is to display an auxiliary shape overlay together with the external view of the body, the auxiliary shape overlay indicative of an extent, relative to the body, of a potential field of view of an auxiliary imaging device or of the active imaging device; and
directing, by the mixed reality presentation system and based on the determining that the display device is to display the auxiliary shape overlay, the display device to display the auxiliary shape overlay together with the external view of the body.

### Claim 33

(canceled)

### Claim 34 (depends on 25)

The method of claim 25, wherein:
the identifying of the operating condition comprises detecting an initiation, during performance of the operation, of a process to swap out the active imaging device for an additional imaging device;
the directing of the display device to toggle the display of the shape overlay comprises beginning to display the shape overlay at a static position with respect to the body; and
the method further comprises:
determining, by the mixed reality presentation system subsequent to the beginning to display the shape overlay at the static position, that the display device is to further display an additional shape overlay together with the shape overlay and the external view of the body, the additional shape overlay indicative of an extent, relative to the body, of an additional field of view corresponding to the additional imaging device as the process to swap out the active imaging device for the additional imaging device is performed,
directing, by the mixed reality presentation system and based on the determining that the display device is to display the additional shape overlay, the display device to display the additional shape overlay together with the shape overlay and the external view of the body, and
directing, by the mixed reality presentation system, the display device to persist in displaying the shape overlay at the static position at least until the additional shape overlay overlaps with the shape overlay at the static position.

### Claim 35 (depends on 25)

The method of claim 25, wherein:
the identifying of the operating condition comprises detecting an initiation, during performance of the operation, of a process to move the active imaging device from a first port into the body by way of which the active imaging device captures the imagery of the internal view to a second port into the body;
the directing of the display device to toggle the display of the shape overlay comprises beginning to display the shape overlay at a static position with respect to the body; and
the method further comprises:
determining, by the mixed reality presentation system subsequent to the beginning to display the shape overlay at the static position, that the display device is to further display an additional shape overlay together with the shape overlay and the external view of the body, the additional shape overlay indicative of an extent, relative to the body, of an additional field of view corresponding to the field of view of the active imaging device as the process to move the active imaging device is performed,
directing, by the mixed reality presentation system and based on the determining that the display device is to display the additional shape overlay, the display device to display the additional shape overlay together with the shape overlay and the external view of the body, and
directing, by the mixed reality presentation system, the display device to persist in displaying the shape overlay at the static position at least until the additional shape overlay overlaps with the shape overlay at the static position.

### Claim 36 (depends on 25)

The method of claim 25, wherein:
the identifying of the operating condition comprises detecting an initiation, during performance of the operation, of a process to move the active imaging device from being attached to a first manipulator arm to being attached to a second manipulator arm;
the directing of the display device to toggle the display of the shape overlay comprises beginning to display the shape overlay at a static position with respect to the body; and
the method further comprises:
determining, by the mixed reality presentation system subsequent to the beginning to display the shape overlay at the static position, that the display device is to further display an additional shape overlay together with the shape overlay and the external view of the body, the additional shape overlay indicative of an extent, relative to the body, of an additional field of view corresponding to the field of view of the active imaging device as the process to move the active imaging device is performed,
directing, by the mixed reality presentation system and based on the determining that the display device is to display the additional shape overlay, the display device to display the additional shape overlay together with the shape overlay and the external view of the body, and
directing, by the mixed reality presentation system, the display device to persist in displaying the shape overlay at the static position at least until the additional shape overlay overlaps with the shape overlay at the static position.

### Claim 37

A non-transitory computer-readable medium storing instructions that, when executed, direct a processor of a computing device to:
identify an operating condition associated with an operation performed on a body while an active imaging device captures imagery of an internal view of the body,
determine, based on the identified operating condition, that a display device is to toggle a display of a shape overlay that is displayed together with an external view of the body, the shape overlay indicative of an extent of a field of view of the active imaging device relative to the body, and
based on the determining that the display device is to toggle the display of the shape overlay, direct the display device to toggle the display of the shape overlay.

### Claim at position 32

38-43. (canceled)
