# US 2019/0378276 A1 · Augmented reality patient positioning using an atlas

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0378276 A1
- Document ID: US20190378276A1
- Status: Published application
- Kind: A1
- Application number: 16/478,341
- Filed: 2017-03-22
- Published: 2019-12-12
- Earliest claimed: 2017-03-22
- Assignees: Brainlab AG
- Applicants: Brainlab AG
- Inventors: Sven Flossmann; Samuel Kerschbaumer; Nils Frielinghaus; Christoffer Hamilton
- Web page: https://patentgrep.com/patent/US20190378276A1

## CPC

- G06T 7/0016: {involving temporal comparison}
- G06T 7/344: {involving models}
- G06T 7/11: Region-based segmentation
- G06T 7/33: using feature-based methods
- G06T 7/60: Analysis of geometric attributes
- G06T 2200/04: involving 3D image data
- G06T 2207/10081: Computed x-ray tomography [CT]
- G06T 2207/10088: Magnetic resonance imaging [MRI]
- G06T 2207/10132: Ultrasound image
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30016: Brain
- G06T 2207/30028: Colon; Small intestine
- G06T 2207/30061: Lung
- G06T 2207/30092: Stomach; Gastric
- G06T 2210/41: Medical
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The disclosed method encompasses using an augmented reality device to blend in augmentation information including for example atlas information. The atlas information may be display separately from or in addition to a patient image (planning image). In order to display the atlas information in a proper position relative to the patient image, data the two data sets are registered to one another. This registration can serve for generating a diversity of atlas-based image supplements, for example alternatively or additionally to the foregoing for displaying a segmentation of the patient image in the augmented reality image. The disclosed method is usable in a medical environment such as for surgery or radiotherapy.

## Claims

### Claim at position 1

-26. (canceled)

### Claim 27

A computer-implemented method for generating augmentation information relating to a medical image, using the output of an augmented reality device having an output unit for outputting the augmentation information, comprising executing, on at least one processor of at least one computer, the steps of:
acquiring patient medical image data which describes a patient image of an anatomical body part of a patient;
acquiring geometric model data which describes a geometric model of at least part of the patient's body;
registering the patient image with the geometric model;
outputting augmentation information using the output unit of the augmented reality device based on a result of registering the patient image with the geometric model;
wherein the output unit of the augmented reality device includes a display for displaying the augmentation information;
the augmented reality device including a head-mounted distance measurement unit having a light source for emitting measurement light onto a physical object; and
wherein at least the part of the geometric model or at least part of the patient image describing at least part of the anatomical body part is registered, based on a result of a distance measurement by the distance measurement unit of the augmented reality device, with a real image acquired using the augmented reality device and describing a physical structure corresponding to the at least one of the at least part of the geometric model or the at least part of the patient image describing the at least part of the anatomical body part.

### Claim 28 (depends on 27)

The method according to claim 27 wherein the geometric model data is acquired from an image-based atlas data, describing the at least part of the patient's body or from a generic geometric model of the at least part of the patient's body, or from an individual synthetic template describing a synthesized geometric model of the at least part of the patient's body.

### Claim 29 (depends on 27)

The method according to claim 27 wherein the geometric model comprises a model of at last part of the anatomical body part, or wherein the geometric model does not comprise a model of the anatomical body part.

### Claim 30 (depends on 27)

The method according to claim 27 wherein the patient image is registered with the geometric model by establishing a positional mapping between a patient image coordinate system in which image coordinates of the patient image are defined and a model coordinate system in which positions of the geometric model are defined by applying an image fusion algorithm to the patient image and the geometric model.

### Claim 31 (depends on 27)

The method according to claim 27 wherein the augmentation information describes at least part of a geometric model of at least part of the anatomical body part.

### Claim 32 (depends on 27)

The method according to claim 27 wherein the display of the augmented reality device is configured to output augmentation information as an overlay on a real image acquired using the augmented reality device.

### Claim 33 (depends on 27)

The method according to claim 27 wherein the physical structure is at least part of a patient's body or a shape representative of at least part of a patient's body.

### Claim 34 (depends on 33)

The method according to claim 33 wherein the physical structure is positionally tracked using a result of distance measurements by the distance measurement unit of the augmented reality device, and
wherein the registration between the at least one of at least part of the geometric model or at least part of the anatomical body part with the physical structure is used to determine, based on the result of the positional tracking, the position of at least part of the at least one of at least part of the geometric model or at least part of the anatomical body part in a coordinate system used by the augmented reality device to output image information describing the at least one of at least part of the geometric model or at least part of the anatomical body part.

### Claim 35 (depends on 27)

The method according to claim 27 wherein the physical structure is positionally tracked using a result of distance measurements by the distance measurement unit of the augmented reality device, and
wherein the registration between the at least one of at least part of the geometric model or at least part of the anatomical body part with the physical structure is used to determine, based on the result of the positional tracking, control data for controlling movement of at least part of the physical structure.

### Claim 36 (depends on 27)

The method according to claim 27 further comprising:
acquiring medical procedure data which describes an envisaged medical procedure to be carried out on the patient;
acquiring device control data describing a control parameter of a medical device based on the medical procedure data;
outputting augmentation information corresponding to the device control data by the augmented reality device.

### Claim 37

A non-transitory computer-readable storage medium for generating augmentation information relating to a medical image, using the output of an augmented reality device having an output unit for outputting the augmentation information, comprising instructions, which when executed by at least one processor causes the at least one processor to:
acquire patient medical image data which describes a patient image of an anatomical body part of a patient;
acquire geometric model data which describes a geometric model of at least part of the patient's body;
register the patient image with the geometric model;
output augmentation information using the output unit of the augmented reality device based on a result of registering the patient image with the geometric model;
wherein the output unit of the augmented reality device includes a display for displaying the augmentation information;
the augmented reality device including a head-mounted distance measurement unit having a light source for emitting measurement light onto a physical object; and wherein at least the part of the geometric model or at least part of the patient image describing at least part of the anatomical body part is registered, based on a result of a distance measurement by the distance measurement unit of the augmented reality device, with a real image acquired using the augmented reality device and describing a physical structure corresponding to the at least one of the at least part of the geometric model or the at least part of the patient image describing the at least part of the anatomical body part.

### Claim 38

A medical system for generating augmentation information relating to a medical image, using the output of an augmented reality device having an output unit for outputting the augmentation information, the medical system comprising:
at least one computer having at least one processor with associated memory, the memory storing instructions which when executed, cause the at least one processor to:
acquire patient medical image data which describes a patient image of an anatomical body part of a patient;
acquire geometric model data which describes a geometric model of at least part of the patient's body;
register the patient image with the geometric model;
output augmentation information using the output unit of the augmented reality device based on a result of registering the patient image with the geometric model;
wherein the output unit of the augmented reality device includes a display for displaying the augmentation information;
the augmented reality device including a head-mounted distance measurement unit having a light source for emitting measurement light onto a physical object, and wherein at least the part of the geometric model or at least part of the patient image describing at least part of the anatomical body part is registered, based on a result of a distance measurement by the distance measurement unit of the augmented reality device, with a real image acquired using the augmented reality device and describing a physical structure corresponding to the at least one of the at least part of the geometric model or the at least part of the patient image describing the at least part of the anatomical body part.