# US 2021/0174523 A1 · Method for registration of image data and for provision of corresponding trained facilities, apparatus for doing so and corresponding computer program product

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174523 A1
- Document ID: US20210174523A1
- Status: Published application
- Kind: A1
- Application number: 17/116,190
- Filed: 2020-12-09
- Published: 2021-06-10
- Earliest claimed: 2019-12-10
- Assignees: Siemens Healthineers AG
- Applicants: Siemens Healthcare GmbH
- Inventors: Christian Kaethner; Alois Regensburger
- Web page: https://patentgrep.com/patent/US20210174523A1

## CPC

- G16H 30/40: for processing medical images, e.g. editing
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/251: {of input or preprocessed data}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 3/096: Transfer learning
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/11: Region-based segmentation
- G06T 7/337: {involving reference images or patches}
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/776: Validation; Performance evaluation
- G06V 10/803: {of input or preprocessed data}
- G06V 10/82: using neural networks
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G06N 3/045: Combinations of networks
- G06N 3/048: Activation functions
- G06N 3/08: Learning methods
- G06T 2207/10081: Computed x-ray tomography [CT]
- G06T 2207/10088: Magnetic resonance imaging [MRI]
- G06T 2207/20081: Training; Learning
- G06T 2207/30101: Blood vessel; Artery; Vein; Vascular
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A system and method for registration of preoperative image data of a patient with less high-quality intra-operative image data relative to the data. Predetermined anatomical landmarks and/or an anatomical variant are identified in the image data by machine-learning facilities. At least one of the machine-learning facilities is trained patient-specifically and/or specifically for an anatomical variant of the patient to be examined. The registration is then carried out with the aid of the landmarks identified.

## Claims

### Claim 1

A method for registration of preoperative image data provided of a patient to be examined with less high-quality intra-operative image data compared thereto of the patient provided, the method comprising:
training a first machine-learning facility to identify predetermined anatomical landmarks or an anatomical variant present in preoperative image data;
training patient-specifically for an anatomical variant of the respective patient to be examined, a second machine-learning facility to identify the predetermined anatomical landmarks in correspondingly less high-quality image data;
wherein if the first facility is trained to identify the predetermined anatomical landmarks, in the pre-operative image data of the patient to be examined, the predetermined anatomical landmarks that are present are identified, or
wherein if the first facility is trained to identify the anatomical variant, with the aid of the pre-operative image data of the patient to be examined, the patient's anatomical variant is identified and in the pre-operative image data of the patient to be examined, the landmarks are identified by a third machine-learning facility that is trained specifically for pre-operative image data of the anatomical variant to identify the predetermined anatomical landmarks;
identifying the predetermined anatomical landmarks in the intra-operative image data of the patient by the second facility; and
registering the preoperative image data provided of a patient to be examined with less high-quality intra-operative image data with the aid of the identified predetermined anatomical landmarks.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first machine-learning facility, second machine-learning facility, and third machine-learning facility are trained to generate segmentations that are used when registering the preoperative image data provided of a patient to be examined with less high-quality intra-operative image data.

### Claim 3 (depends on 1)

The method of claim 1, wherein a plurality of datasets of a plurality of patients is provided, wherein each of the datasets contains preoperative image data in each case, that are at least annotated with the respective anatomical variants present therein, and intra-operative image data assigned to these, in which at least the predetermined landmarks are specified as annotation, of precisely one of the plurality of patients;
wherein on the basis thereof anatomical variants are determined, that occur in datasets with at least a predetermined frequency in each case,
wherein for each of the anatomical variants determined a respective variant-specific instance of the second facility is provided that is trained or retrained in each case only with the aid of the intra-operative image data of the corresponding datasets in which the respective anatomical variant occurs;
wherein for each of the anatomical variants determined a respective instance of the third facility is provided that is trained or retrained in each case only with the aid of the preoperative image data of the corresponding datasets in which the respective anatomical variant occurs;
wherein the anatomical variant of the patient to be examined is determined with the aid of the preoperative image data provided of the patient to be examined by the first facility; and
wherein for the recognition of the landmarks in the image data provided for the registration, trained instances of the second facility and of the third facility are selected and used for the anatomical variant determined of the patient to be examined.

### Claim 4 (depends on 1)

The method of claim 1, wherein a plurality of datasets of a plurality of patients is provided, wherein each of the datasets contains preoperative image data and intra-operative image data assigned in which at least the predetermined landmarks are specified as annotations of at least one of precisely one of the plurality of patients;
wherein the predetermined landmarks are identified by the correspondingly trained first facility in the preoperative image data of the patient to be examined; wherein correspondingly similar datasets are determined and selected automatically from the plurality of datasets, that in accordance with a predetermined similarity criterion based on the identified landmarks, describe an anatomy similar to the patient to be examined;
wherein the third facility is used for identifying the landmarks in the preoperative image data of the patient to be examined, wherein the facility is only trained or retrained with the aid of the preoperative image data from the selected similar datasets,
wherein the second facility is used for identifying the landmarks in the intra-operative image data of the patient to be examined, wherein the facility is only trained or retrained with the aid of the intra-operative image data from the selected similar datasets.

### Claim 5 (depends on 4)

The method of claim 4, wherein whether in the datasets the landmarks identified in the preoperative image data of the patient to be examined or at least of a predetermined portion thereof determined in the preoperative image data of the patient to be examined is contained in the respective dataset is used as the similarity criterion.

### Claim 6 (depends on 4)

The method of claim 4, wherein the similarity criterion used is a geometrical position relationship of a number of landmarks in relation to one another and/or to at least one predetermined anatomy feature or reference point.

### Claim 7 (depends on 1)

The method of claim 1, wherein the predetermined landmarks that are present in the preoperative image data of the patient to be examined are identified by the first facility, wherein on the basis thereof pre-operative image data annotated with the identified landmarks, wherein synthetic simulated intra-operative training image data is created automatically from the annotated preoperative image data of the patient to be examined, and wherein the second facility is used to identify the landmarks in the intra-operative image data of the patient to be examined, wherein the facility is trained or retrained with the aid of the synthetic simulated intra-operative training image data.

### Claim 8 (depends on 7)

The method of claim 7, wherein to create the synthetic simulated intra-operative training image data, starting from the annotated preoperative image data, a plurality of different deformations of anatomical objects or features imaged is simulated on the basis of at least one predetermined biomechanical model.

### Claim 9 (depends on 7)

The method of claim 7, wherein to create the synthetic simulated intra-operative training image data, starting from the annotated preoperative image data, a plurality of different kinds, arrangements, or versions of image artifacts or noise effects are simulated, which may occur during the recording of intra-operative image data.

### Claim 10 (depends on 7)

The method of claim 7, wherein to create the synthetic simulated intra-operative training image data, starting from the annotated preoperative image data, a weaker level of contrast or an incomplete filling of a vascular tree imaged with a contrast medium in a plurality of different variants, is simulated.

### Claim 11 (depends on 1)

The method of claim 1, wherein a deep neural network with an input layer, an output layer, and at least one intermediate layer lying between the input layer and output layer is used as at least one of the machine-learning facilities.

### Claim 12 (depends on 1)

The method of claim 1, wherein at least one of the machine learning facilities that is different from the first facility, is pre-trained with the aid of image data of a plurality of different patients and is then retrained overfitted specifically to the patient to be examined or to the anatomical variant.

### Claim 13 (depends on 1)

The method of claim 1, wherein a segmentation of at least one anatomical structure in the preoperative image data or in the intra-operative image data of the patient to be examined, is generated by at least one of the machine-learning facilities.

### Claim 14 (depends on 1)

The method of claim 1, further comprising:
providing annotated training image data that comprises preoperative image data and less high-quality intra-operative image data; and
inputting the annotated training image data into each of machine-learning facilities as input data;
capturing output data created by processing of the input data by the respective machine-learning facilities;
training the machine-learning facilities with a predetermined training function on the basis of the input data and the output data, and
providing the machine-learning facilities.

### Claim 15

A method for registration of preoperative image data provided of a patient to be examined with less high-quality intra-operative image data compared thereto of the patient provided, the method comprising:
training a first machine-learning facility to generate a segmented image of predetermined anatomical landmarks or an anatomical variant present in preoperative image data;
training patient-specifically for an anatomical variant of the respective patient to be examined, a second machine-learning facility to generate a segmented image of the predetermined anatomical landmarks in correspondingly less high-quality image data;
wherein if the first facility is trained to segment the predetermined anatomical landmarks, in the pre-operative image data of the patient to be examined, the predetermined anatomical landmarks that are present are segmented, or
wherein if the first facility is trained to segment the anatomical variant, with the aid of the pre-operative image data of the patient to be examined, the patient's anatomical variant is segmented and in the pre-operative image data of the patient to be examined, the landmarks are segmented by a third machine-learning facility that is trained specifically for pre-operative image data of the anatomical variant to segment the predetermined anatomical landmarks;
segmenting the predetermined anatomical landmarks in the intra-operative image data of the patient by the second facility; and
registering the preoperative image data provided of a patient to be examined with less high-quality intra-operative image data with the aid of the segmentations.

### Claim 16 (depends on 15)

The method of claim 15, wherein a deep neural network with an input layer, an output layer, and at least one intermediate layer lying between the input layer and output layer is used as at least one of the machine-learning facilities.

### Claim 17 (depends on 15)

The method of claim 15, further comprising:
providing annotated training image data that comprises preoperative image data and less high-quality intra-operative image data; and
inputting the annotated training image data into each of machine-learning facilities as input data;
capturing output data created by processing of the input data by the respective machine-learning facilities;
training the machine-learning facilities with a predetermined training function on the basis of the input data and the output data, and
providing the machine-learning facilities.

### Claim 18

A non-transitory computer-readable data storage medium that stores machine-readable instructions executable by at least one processor for registration of preoperative image data provided of a patient to be examined with less high-quality intra-operative image data compared thereto of the patient provided, the machine-readable instructions comprising:
training a first machine-learning facility to identify predetermined anatomical landmarks or an anatomical variant present in preoperative image data;
training patient-specifically for an anatomical variant of the respective patient to be examined, a second machine-learning facility to identify the predetermined anatomical landmarks in correspondingly less high-quality image data;
wherein if the first facility is trained to identify the predetermined anatomical landmarks, in the pre-operative image data of the patient to be examined, the predetermined anatomical landmarks that are present are identified, or
wherein if the first facility is trained to identify the anatomical variant, with the aid of the pre-operative image data of the patient to be examined, the patient's anatomical variant is identified and in the pre-operative image data of the patient to be examined, the landmarks are identified by a third machine-learning facility that is trained specifically for pre-operative image data of the anatomical variant to identify the predetermined anatomical landmarks;
identifying the predetermined anatomical landmarks in the intra-operative image data of the patient by the second facility; and
registering the preoperative image data provided of a patient to be examined with less high-quality intra-operative image data with the aid of the identified predetermined anatomical landmarks.
