# US 12,213,737 B2 · User interface for image guided surgery system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,213,737 B2
- Document ID: US12213737B2
- Status: Granted
- Kind: B2
- Application number: 17/194,466
- Filed: 2021-03-08
- Published: 2021-10-14
- Granted: 2025-02-04
- Earliest claimed: 2020-04-09
- Assignees: Acclarent, Inc.; Biosense Webster (Israel) Ltd.
- Applicants: Acclarent, Inc.; Biosense Webster (Israel) Ltd.
- Inventors: Fatemeh Akbarian; Athanasios Papadakis; Jordan R. Trott; Noam Racheli; Itamar Bustan; Jetmir Palushi
- Web page: https://patentgrep.com/patent/US12213737B2

## CPC

- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 34/25: {User interfaces for surgical systems}
- A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones}
- A61B 2034/107: {Visualisation of planned trajectories or target regions}
- A61B 2034/2051: {Electromagnetic tracking systems}
- A61B 2034/2057: {Details of tracking cameras}
- A61B 2034/2068: {using pointers, e.g. pointers having reference marks for determining coordinates of body points}
- A61B 2034/2072: {Reference field transducer attached to an instrument or patient}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)

## Abstract

A virtual camera may be positioned relative to a 3-D model of a patient anatomy in order to provide a virtual camera view of the surrounding anatomy and tracked surgical instruments being deployed to the anatomy. Visual context provided by the virtual camera may be limited in some cases, such as where a surgical instrument is being used within a very narrow anatomical passageway or cavity. To provide more placement flexibility, an IGS system providing a virtual camera receives inputs defining variable visual characteristics for different segments or regions of the 3-D model, which may include hiding certain segments or making certain segments semi-transparent. With such a system, the view of the 3-D model provided by the virtual camera view may be modified to remove or deemphasize less relevant segments, to display or emphasize more relevant segments (e.g., critical patient anatomy), or both.

## Claims

### Claim 1

An image guided surgery system comprising:
(a) a display;
(b) a set of preoperative images associated with a patient;
(c) a tracked surgical instrument configured to produce a set of instrument location information that is associated with a coordinate system; and
(d) a processor configured to:
(i) segment the set of preoperative images to produce a patient model and associate the patient model with the coordinate system, wherein the patient model comprises a three-dimensional model of the patient's anatomy comprised of a plurality of segments,
(ii) identify, for each segment of the plurality of segments, an anatomical structure of the patient corresponding to each segment,
(iii) determine a filter selection indicative of a surgical procedure to be performed on the patient using the image guided surgery system,
(iv) categorize each segment of the plurality of segments into two or more segment groups based upon the filter selection indicative of the surgical procedure and on the anatomical structure corresponding to the respective segment, wherein each of the two or more segment groups is associated with a different visual characteristic,
(v) render a modified patient model based on the different visual characteristic of each of the two or more segment groups,
(vi) receive a set of virtual camera configurations that define a position and an orientation of a virtual camera within the coordinate system, and
(vii) cause the display to show a virtual camera view of the modified patient model based on the set of virtual camera configurations and the set of instrument location information.

### Claim 2 (depends on 1)

The system of claim 1, wherein:
(i) a first visual characteristic associated with a first group of the two or more segment groups is configured to cause segments in the first group to be included in the modified patient model, and
(ii) a second visual characteristic associated with a second group of the two or more segment groups is configured to cause segments in the second group to be excluded from the modified patient model.

### Claim 3 (depends on 2)

The system of claim 2, wherein the filter selection further comprises an identification of a segment that is to be included in the second group, and wherein the processor is further configured to:
(i) receive the filter selection prior to a surgical procedure associated with the patient, and
(ii) receive the filter selection via a user input device of the image guided surgery system during the surgical procedure associated with the patient.

### Claim 4 (depends on 2)

The system of claim 2, wherein the filter selection further comprises a tissue type, and wherein the processor is further configured to identify the second group in the plurality of segments based on the tissue type matching a tissue type characteristic associated with each of the plurality of segments.

### Claim 5 (depends on 1)

The system of claim 1, wherein the processor is further configured to determine the filter selection by:
(i) identifying a set of proximate segments from the plurality of segments that are within a configured distance of the position of the virtual camera within the coordinate system, and
(ii) associating the set of proximate segments with a first group of the two or more segment groups, wherein the different visual characteristic of the first group is configured to cause segments in the first group to be excluded from the modified patient model.

### Claim 6 (depends on 1)

The system of claim 1, wherein the processor is further configured to determine the filter selection by:
(i) identifying a set of proximate segments from the plurality of segments that are within a configured distance of the position of the tracked surgical instrument within the coordinate system, and
(ii) associating the set of proximate segments with a first group of the two or more segment groups, wherein the different visual characteristic of the first group is configured to cause segments in the first group to be excluded from the modified patient model.

### Claim 7 (depends on 1)

The system of claim 1, wherein the processor is further configured to determine the filter selection by:
(i) identifying a set of intervening segments from the plurality of segments that are positioned along an axis between the virtual camera and a position in the coordinate system that is associated with a point of interest, and
(ii) associating the set of intervening segments with a first group of the two or more segment groups, wherein the different visual characteristic of the first group is configured to cause segments in the first group to be excluded from the modified patient model.

### Claim 8 (depends on 7)

The system of claim 7, wherein the point of interest comprises a surgical site associated with a procedure being performed on the patient.

### Claim 9 (depends on 8)

The system of claim 8, wherein the filter selection further comprises a segment whitelist that describes one or more whitelisted segments of the plurality of segments, and wherein the processor is further configured to prevent the whitelisted segments from being associated with the first group and excluded from the modified patient model.

### Claim 10 (depends on 9)

The system of claim 9, wherein the processor is further configured to:
(i) receive a user input configured to cause a one-touch virtual camera movement,
(ii) determine a path within the coordinate system based on the user input,
(iii) move the virtual camera from an origin point along the path until the virtual camera returns to the origin point while maintaining the virtual camera's orientation on the surgical site, and
(iv) update the patient model based on changes in the set of intervening segments during movement of the virtual camera along the path.

### Claim 11 (depends on 1)

The system of claim 1, wherein the plurality of segments comprises one or more of:
(i) voxels,
(ii) sets of adjacent voxels, or
(iii) anatomical structures identified based upon their shape and location in the patient model.

### Claim 12 (depends on 1)

The system of claim 1, wherein each different visual characteristic is selected from a set of visual characteristics comprising:
(i) opaque,
(ii) partially translucent
(iii) transparent, and
(iv) gradual increase and reduction of opacity.

### Claim 13

A method for selectively controlling segments shown by a virtual camera comprising:
with a processor,
(a) segmenting a set of preoperative images to produce a patient model and associating the patient model with a coordinate system, wherein the patient model comprises a three-dimensional model of a patient's anatomy comprised of a plurality of segments;
(b) identify, for each segment of the plurality of segments, an anatomical structure of the patient corresponding to each segment,
(c) determine a filter selection indicative of a surgical procedure to be performed on the patient using the image guided surgery system,
(d) categorizing each segment of the plurality of segments into two or more segment groups based upon the filter selection indicative of the surgical procedure and on the anatomical structure corresponding to the respective segment, wherein each of the two or more segment groups is associated with a different visual characteristic;
(e) rendering a modified patient model based on the different visual characteristic of each of the two or more segment groups;
(f) receiving a set of virtual camera configurations that define a position and an orientation of the virtual camera within the coordinate system;
(g) receiving a set of instrument location information that indicates a tracked surgical instrument within the coordinate system; and
(h) causing a display to show a virtual camera view of the modified patient model based on the set of virtual camera configurations and the set of instrument location information.

### Claim 14 (depends on 13)

The method of claim 13, further comprising receiving the filter selection prior to a surgical procedure associated with the patient, wherein:
(i) a first visual characteristic associated with a first group of the two or more segment groups is configured to cause segments in the first group to be included in the modified patient model,
(ii) a second visual characteristic associated with a second group of the two or more segment groups is configured to cause segments in the second group to be excluded from the modified patient model, and
(iii) the filter selection further comprises an identification of a segment of the plurality of segments to be included in the second group of the two or more segment groups.

### Claim 15 (depends on 13)

The method of claim 13, further comprising:
(a) determining a filtered tissue type based upon the filter selection; and
(b) categorizing the plurality of segments into an included segment group and an excluded segment group based upon the filtered tissue type matching a tissue type characteristic associated with each of the plurality of segments.

### Claim 16 (depends on 13)

The method of claim 13, further comprising determining the filter selection by:
(i) identifying a set of proximate segments from the plurality of segments that are within a configured distance of the position of the virtual camera within the coordinate system, and
(ii) associating the set of proximate segments with a first group of the two or more segment groups, wherein the different visual characteristic of the first group is configured to cause segments in the first group to be excluded from the modified patient model.

### Claim 17 (depends on 13)

The method of claim 13, further comprising determining the filter selection by:
(i) identifying a set of intervening segments from the plurality of segments that are positioned along an axis between the virtual camera and a surgical site in the coordinate system, and
(ii) associating the set of intervening segments with a first group of the two or more segment groups, wherein the different visual characteristic of the first group is configured to cause segments in the first group to be excluded from the modified patient model.

### Claim 18

An image guided surgery system comprising one or more processors configured to:
(i) maintain a coordinate system,
(ii) segment a set of preoperative images to produce a patient model and associate the patient model with the coordinate system, wherein the patient model comprises a three-dimensional model of the patient's anatomy comprised of a plurality of segments,
(iii) identify, for each segment of the plurality of segments, an anatomical structure of the patient corresponding to each segment,
(iii) determine a filter selection indicative of a surgical procedure to be performed on the patient using the image guided surgery system,
(iv) categorize each segment of the plurality of segments into one of two segment groups based upon the filter selection indicative of the surgical procedure and on the anatomical structure corresponding to the respective segment, wherein each of the two segment groups is associated with a different visual characteristic,
(v) render a modified patient model based on the different visual characteristic of each of the two or more segment groups, and
(vi) update the two segment groups by moving a segment categorized into a first segment group to the second segment group based on a present procedural step of the surgical procedure being performed on the patient.

### Claim 19 (depends on 18)

The image guided surgery system of claim 18, wherein:
(i) a first visual characteristic associated with the first segment group is configured to cause segments in the first segment group to be included in the modified patient model, and
(ii) a second visual characteristic associated with the second segment group is configured to cause segments in the second segment group to be excluded from the modified patient model.

### Claim 20 (depends on 18)

The image guided surgery system of claim 18, wherein to determine the filter selection indicative of a surgical procedure to be performed on the patient comprises to determine a first filter selection, and
wherein the one or more processors are further configured to determine a second filter selection indicative of a tissue type,
wherein to categorize each segment of the plurality of segments further comprises to categorize each segment of the plurality of segments into one of the two segment groups based upon the first filter selection and the second filter selection.