# US 10,872,401 B2 · Method for merging an analysis data record with an image data record, positioning device, computer program and electronically readable data storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,872,401 B2
- Document ID: US10872401B2
- Status: Granted
- Kind: B2
- Application number: 16/199,538
- Filed: 2018-11-26
- Published: 2019-06-06
- Granted: 2020-12-22
- Earliest claimed: 2017-12-05
- Assignees: Siemens Healthcare GmbH
- Applicants: Siemens Healthcare GmbH
- Inventors: Sebastian Schmidt; Philipp Hoelzer
- Web page: https://patentgrep.com/patent/US10872401B2

## CPC

- G06T 5/50: using two or more images, e.g. averaging or subtraction
- A61B 6/03: Computed tomography [CT]
- A61B 6/032: {Transmission computed tomography [CT]}
- A61B 6/4417: {related to combined acquisition of different diagnostic modalities}
- A61B 6/463: {characterised by displaying multiple images or images and diagnostic data on one display}
- A61B 6/466: {adapted to display 3D data}
- A61B 6/469: {for selecting a region of interest [ROI]}
- A61B 6/482: {involving multiple energy imaging}
- A61B 6/5235: {combining images from the same or different ionising radiation imaging techniques, e.g. PET and CT}
- A61B 6/5247: {combining images from an ionising-radiation diagnostic technique and a non-ionising radiation diagnostic technique, e.g. X-ray and ultrasound}
- A61B 6/5294: {involving using additional data, e.g. patient information, image labeling, acquisition parameters}
- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/30: Determination of transform parameters for the alignment of images, i.e. image registration
- G06T 7/337: {involving reference images or patches}
- G06T 7/40: Analysis of texture (depth or shape recovery from texture G06T7/529)
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G16H 10/40: for data related to laboratory analysis, e.g. patient specimen analysis
- G16H 30/40: for processing medical images, e.g. editing
- A61B 6/4241: {using energy resolving detectors, e.g. photon counting}
- A61B 2505/05: Surgical care
- A61B 2576/00: Medical imaging apparatus involving image processing or analysis (A61B1/00009, A61B6/52 and A61B8/52 take precedence)
- G06T 2207/10056: Microscopic image
- G06T 2207/10081: Computed x-ray tomography [CT]
- G06T 2207/10088: Magnetic resonance imaging [MRI]
- G06T 2207/10116: X-ray image
- G06T 2207/10132: Ultrasound image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G06T 2207/30096: Tumor; Lesion
- G06T 2210/41: Medical
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

A method for merging a three-dimensional analysis data record of a tissue sample of a patient with a three-dimensional image data record of the patient indicating, prior to removal, the removal area of the tissue sample. The method includes registering a three-dimensional intermediate data record of the three-dimensional analysis data record. The registering includes transforming analysis data of the three-dimensional analysis data record into intermediate data, corresponding to image data recorded with a modality with which the three-dimensional image data record has been recorded and including a resolution reduced, to determine the three-dimensional intermediate data record corresponding, in terms of resolution, to the image data record; and registering the three-dimensional intermediate data record with the three-dimensional image data record. Finally, the method includes merging the three-dimensional analysis data record and the three-dimensional image data record using a registration rule obtained during the registering of the three-dimensional intermediate data record.

## Claims

### Claim 1

A method for merging a three-dimensional analysis data record of a tissue sample of a patient taken from a patient with a three-dimensional image data record of the patient indicating, prior to removal, a removal area from which the tissue sample has been taken, the method comprising:
registering a three-dimensional intermediate data record of the three-dimensional analysis data record, with the three-dimensional image data record, the registering including
transforming analysis data of the three-dimensional analysis data record into intermediate data, the intermediate date corresponding to image data recorded with a modality with which the three-dimensional image data record has been recorded, and having a reduced resolution, such that the three-dimensional intermediate data record corresponds, in terms of resolution, to the image data record,
assigning at least one of an intermediate item of data of the three-dimensional intermediate data record and an item of tissue data describing a tissue property which relates to the modality of the three-dimensional image data record, to each voxel of the three-dimensional analysis data record, wherein the three-dimensional intermediate data record is determined by simulation of an imaging process of the modality using the item of tissue data; and
merging the three-dimensional analysis data record and the three-dimensional image data record using a registration rule obtained during the registering of the three-dimensional intermediate data record.

### Claim 2 (depends on 1)

The method of claim 1, wherein the tissue sample, in at least one expansion direction, is at least ten times a size of a voxel of the three-dimensional image data record.

### Claim 3 (depends on 1)

The method of claim 1, wherein, to determine the three-dimensional analysis data record, the tissue sample is divided into layers, and examined microscopically.

### Claim 4

The method of claim ,1 wherein recording parameters of the three-dimensional image data record are used with the simulation of the imaging process.

### Claim 5 (depends on 4)

The method of claim 4, wherein the three-dimensional image data record includes a number of images, having been recorded with different recording parameters, and wherein the three-dimensional image data record, including the number of images, is used in determining intermediate images of the three-dimensional intermediate data record for all sets of recording parameters.

### Claim 6 (depends on 1)

The method of claim 1, wherein the assigning is carried out using an artificial intelligence assignment algorithm, trained by machine learning by way of registered three-dimensional analysis data records and three-dimensional image data records.

### Claim 7 (depends on 1)

The method of claim 1, wherein at least one part of at least one of the transforming to determine the three-dimensional intermediate data record and the registering is carried out by texture analysis.

### Claim 8 (depends on 7)

The method of claim 7, wherein at least one of texture data, describing a texture of the three-dimensional analysis data record, is determined by the texture analysis and is taken into account when assigning at least one of image data and the item of tissue data and
the registering is carried out by comparing textures of the three-dimensional image data record with a texture of the three-dimensional intermediate data record.

### Claim 9 (depends on 1)

The method of claim 1, wherein, for registration purposes, a rough positioning of the three-dimensional analysis data record with respect to the three-dimensional image data record is firstly performed using intraoperative auxiliary images recorded with the removal of the tissue sample, registered with the three-dimensional image data record, and wherein the rough positioning is used as a starting point for the registering.

### Claim 10 (depends on 1)

The method of claim 1, wherein at least one of a direct-conversion detector and a recording protocol, allowing for a higher resolution in an area containing the tissue sample on account of a higher radiation dose, is used as an x-ray detector to record the three-dimensional image data record during x-ray imaging.

### Claim 11 (depends on 1)

The method of claim 1, wherein at least one of
the merging is carried out by way of superimposition,
a zoom function is provided with a display of a three-dimensional merging data record obtained as a result of the merging, and
prior to the merging, a reformatting of the three-dimensional analysis data record is carried out to adjust to a type of visualization of the three-dimensional image data record.

### Claim 12

A merging device, comprising:
a control device, embodied to
register a three-dimensional intermediate data record of a three-dimensional analysis data record, with a three-dimensional image data record, the control device being embodied, to register three-dimensional intermediate data record, to
transform analysis data of the three-dimensional analysis data record into intermediate data, the intermediate data corresponding to image data recorded with a modality with which the three-dimensional image data record has been recorded and having a reduced resolution, such that the three-dimensional intermediate data record corresponds, in terms of resolution, to the image data record,
assign at least one of an intermediate item of data of the three-dimensional intermediate data record and an item of tissue data describing a tissue property which relates to the modality of the three-dimensional analysis data record, wherein the three-dimensional intermediate data record is determined by simulation of an imaging process of the modality using the item of tissue data; and
merge the three-dimensional analysis data record and the three-dimensional image data record using a registration rule obtained during the registering of the three-dimensional intermediate data record.

### Claim 13 (depends on 1)

A non-transitory computer readable medium storing a computer program, the computer program including program segments that, when executed on a control device of a merging device, perform the method of claim 1.

### Claim 14 (depends on 1)

A non-transitory electronically readable data storage medium, storing a computer program to, when executed on a control device of a merging device, perform the method of claim 1.

### Claim 15 (depends on 2)

The method of claim 2, wherein, to determine the three-dimensional analysis data record, the tissue sample is divided into layers, examined microscopically.

### Claim 16 (depends on 1)

The method of claim 1, wherein the three-dimensional analysis data record is already reduced in terms of resolution.

### Claim 17 (depends on 1)

The method of claim 1, wherein the tissue data includes at least one of instance beam density, relaxation times, and acoustic impedances, based on the modality.

### Claim 18 (depends on 12)

The merging device of claim 12, wherein the tissue data includes at least one of instance beam density, relaxation times, and acoustic impedances, based on the modality.