# US 12,002,147 B2 · Method and system for optimizing distance estimation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,147 B2
- Document ID: US12002147B2
- Status: Granted
- Kind: B2
- Application number: 17/659,905
- Filed: 2022-04-20
- Published: 2022-10-27
- Granted: 2024-06-04
- Earliest claimed: 2021-04-27
- Assignees: Siemens Healthineers AG
- Applicants: Siemens Healthineers AG
- Inventors: Christoph Vetter; Kaloian Petkov; Rishabh Shah; Sandra Sudarsky
- Web page: https://patentgrep.com/patent/US12002147B2

## CPC

- G06T 15/08: Volume rendering
- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 90/36: {Image-producing devices or illumination devices not otherwise provided for}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/11: Region-based segmentation
- G06T 15/00: Three-dimensional [3D] image rendering
- G06T 15/04: Texture mapping
- G06T 15/205: {Image-based rendering}
- G06T 15/50: Lighting effects
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- A61B 2034/107: {Visualisation of planned trajectories or target regions}
- A61B 2034/2051: {Electromagnetic tracking systems}
- A61B 2034/2055: {Optical tracking systems}
- A61B 2034/2065: {Tracking using image or pattern recognition}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/376: {using X-rays, e.g. fluoroscopy}
- A61B 2090/3762: {using computed tomography systems [CT]}
- G06T 2207/30056: Liver; Hepatic
- G06T 2210/41: Medical
- G06T 2215/16: Using real world measurements to influence rendering
- G06T 2219/2024: Style variation
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Distance estimation is optimized in virtual or augmented reality. A distance map of a surgical instrument to a region of interest is determined, at least at the beginning and when a position of the surgical instrument has changed. A render-image is rendered based on a medical 3D image and the position of the surgical instrument, at least at the beginning and when the position of the surgical instrument has changed. At least the region of interest and those parts of the surgical instrument positioned in the volume of the render-image are shown in the render-image. Based on the distance map, at least for a predefined area of the region of interest, visible, acoustic, and/or haptic distance-information is added.

## Claims

### Claim 1

A method for optimizing distance estimation in virtual or augmented reality, the method comprising:
a) providing a medical three-dimensional (3D) image of a volume comprising a segmented anatomical object within a region of interest,
b) providing 3D information of a position of a surgical instrument,
c) determining a distance map of the surgical instrument to the region of interest, at least at a beginning of the method and in the case the position of the surgical instrument has changed, the distance map comprising a field of distances from multiple locations of the region of interest to the surgical instrument at a given time,
d) rendering a render-image based on the 3D image and the information of the position of a surgical instrument, at least at the beginning of the method and in the case the position of the surgical instrument has changed,
wherein at least the region of interest is shown and those parts of the surgical instrument positioned in a volume of the render-image, and
wherein based on the distance map, at least for a predefined area of the region of interest, visible, acoustic, and/or haptic distance-information is added to the render-image, the render-image having the visible, acoustic, and/or haptic distance-information that varies by position based on the field of distances,
e) outputting the render-image and repeating at least acts b) to e).

### Claim 2 (depends on 1)

The method according to claim 1, wherein the 3D medical image of the volume comprising is provided by:
select the region of interest as a predefined region of interest comprising an organ in the 3D medical image, and
segmenting an anatomical structure within the region of interest.

### Claim 3 (depends on 1)

The method according to claim 1, wherein in a course of rendering the render-image, a shadow of the surgical instrument is calculated or shadows for a number of the surgical instruments are calculated.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the distance-information is distance-depending tinting of shadows.

### Claim 5 (depends on 4)

The method according to claim 4, wherein different tinting is used for the shadows of different surgical instruments, and wherein tinting is only applied in the region of interest and/or on a predefined segmented organ.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the distance-information is distance-depending contour lines.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the distance information further comprises numerical values (NV) of distances, wherein a modulo operator is used on the distance in order to create bands and for the contour lines, these bands are compressed into anti-aliased lines.

### Claim 8 (depends on 1)

The method according to claim 1, wherein the position of more than one surgical instrument is provided and, in course of the rendering, the visible distance-information is visualized differently for different surgical instruments, wherein for the different surgical instruments different colors, textures, and/or line-styles are used.

### Claim 9 (depends on 1)

The method according to claim 1, wherein a stochastic visibility map is computed as the distance map.

### Claim 10 (depends on 9)

The method according to claim 9, wherein the stochastic visibility map is updated in the case that a view on the rendering-image has been changed, a number of artificial light-sources has been changed, and/or a position of an artificial light-source has been changed.

### Claim 11 (depends on 1)

The method according to claim 1, wherein, based on the position of a surgical instrument, an additional artificial light-source is positioned behind the surgical instrument and shining on the region of interest.

### Claim 12 (depends on 1)

The method according to claim 1, wherein in determining the distance map of the surgical instrument to the region of interest, the region of interest is recomputed according to the position of the surgical instrument.

### Claim 13 (depends on 12)

The method according to claim 12, wherein the region of interest is recomputed according to the position of the surgical instrument and a position of the segmented anatomical object in the region of interest.

### Claim 14 (depends on 1)

The method according to claim 1, wherein path tracing or shadow mapping for shadows or for ambient lighting, and/or a computation of the visibility map are restricted to the region of interest.

### Claim 15 (depends on 14)

The method according to claim 14, wherein the path tracing or shadow mapping and/or computation of the visibility map are restricted to mesh-to-mesh visibility so that only a mesh of the surgical instrument and a mesh of the segmentation is taken into account while the volume itself is ignored.

### Claim 16

A system for optimizing distance estimation in virtual or augmented reality, the system comprising:
a first data interface configured to receive a medical three-dimensional (3D) image of a volume comprising a segmented anatomical object within a region of interest, and to receive 3D information of a position of a surgical instrument,
a processor configured to determine a distance map of the surgical instrument to the region of interest, the distance map comprising a field of distances from multiple locations of the region of interest to the surgical instrument at a given time,
a renderer configured to render a render-image based on the 3D image and the 3D information of the position of the surgical instrument, wherein at least the region of interest and those parts of the surgical instrument positioned in a volume of the render-image are shown in the render-image, and wherein, based on the distance map, at least for a predefined area of the region of interest, visible, acoustic, haptic distance-information is added, the render-image having the visible, acoustic, and/or haptic distance-information that varies by position based on the field of distances,
a second data interface configured to output the render-image.

### Claim 17

A non-transitory computer-readable medium on which is stored programming that can be read and executed by a computer to optimize distance estimation in virtual or augmented reality, the programming including instructions to:
a) determine a distance map of a surgical instrument to a region of interest, at least at a beginning of the virtual or augmented reality and in the case the position of the surgical instrument has changed, the distance map comprising a field of distances from multiple locations of the region of interest to the surgical instrument at a given time,
b) render a render-image based on a 3D image and a position of a surgical instrument, at least at the beginning and in the case the position of the surgical instrument has changed, wherein at least the region of interest and parts of the surgical instrument positioned in a volume of the render-image are shown, and wherein, based on the distance map, at least for a predefined area of the region of interest, visible, acoustic, and/or haptic distance-information is added, the render-image having the visible, acoustic, and/or haptic distance-information that varies by position based on the field of distances,
c) output the render-image, and
d) repeating acts a) to c).