# US 2021/0065388 A1 · Registering a two-dimensional image with a three-dimensional image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0065388 A1
- Document ID: US20210065388A1
- Status: Published application
- Kind: A1
- Application number: 16/994,943
- Filed: 2020-08-17
- Published: 2021-03-04
- Earliest claimed: 2019-08-28
- Assignees: Siemens Healthcare GmbH
- Applicants: Siemens Healthcare GmbH
- Inventors: Anja Borsdorf; Roman Schaffert; Jian Wang; Markus Weiss
- Web page: https://patentgrep.com/patent/US20210065388A1

## CPC

- G06T 7/33: using feature-based methods
- A61B 6/032: {Transmission computed tomography [CT]}
- G06T 7/0012: {Biomedical image inspection}
- G06T 2207/10081: Computed x-ray tomography [CT]
- G06T 2207/10088: Magnetic resonance imaging [MRI]
- G06T 2207/10116: X-ray image
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)

## Abstract

The disclosure relates to a computer-implemented method for the provision of a transformation instruction for registering a 2D image with a 3D image. The method includes: receiving the 2D image and the 3D image; generating input data based on the 2D image having contour pixels and the 3D image having contour voxels; applying a trained function to the input data for identification of contour pixels of the 2D image and contour voxels of the 3D image, wherein at least one parameter of the trained function is adjusted based on a comparison of training contour pixels with comparison contour pixels and a comparison of training contour voxels corresponding thereto with comparison contour voxels; determining the transformation instruction based on the identified contour pixels of the 2D image and the contour voxels corresponding thereto of the 3D image for registering the 2D image with the 3D image; and providing the transformation instruction.

## Claims

### Claim 1

A computer-implemented method for provision of a transformation instruction for registering a two-dimensional (2D) image with a three-dimensional (3D) image, the method comprising:
receiving the 2D image and the 3D image;
generating input data based on the 2D image comprising contour pixels and the 3D image comprising contour voxels, wherein the contour pixels are dependent on contour features of the 2D image, wherein the contour voxels are dependent on contour features of the 3D image, and wherein the contour voxels of the 3D image have a substantially perpendicular contour surface normal in respect of a specified projection direction;
applying a trained function to the input data for identification of contour pixels of the 2D image and contour voxels of the 3D image, which correspond with each other, wherein at least one parameter of the trained function is adjusted based on a comparison of training contour pixels with comparison contour pixels and a comparison of training contour voxels corresponding thereto with comparison contour voxels; and
determining the transformation instruction based on the identified contour pixels of the 2D image and the contour voxels corresponding thereto of the 3D image for registering the 2D image with the 3D image; and
providing the transformation instruction.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the 2D image comprises a projection image of an examination region of an examination object depicted in the 3D image, and
wherein the specified projection direction is specified as a function of the projection image.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising:
adjusting the specified projection direction as a function of the transformation instruction; and
repeating the generating of the input data, the applying of the trained function, the determining of the transformation function, and the adjusting of the specified projection direction until an occurrence of a termination condition.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the generating of the input data further comprises creating a gradient image from the 3D image by applying a projection mapping, and
wherein the input data is also based on the gradient image.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the generating of the input data further comprises creating a second gradient image from the 2D image, and
wherein the input data is also based on the second gradient image.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, further comprising:
receiving a movement parameter,
wherein the contour features of the 2D image are adjusted in the generating of the input data using the movement parameter.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprising:
registering the 2D image with the 3D image based on the transformation instruction; and
providing the registered 3D image.

### Claim 8

A computer-implemented method for provision of a trained function, the method comprising:
receiving a two-dimensional (2D) training image and a three-dimensional (3D) training image;
identifying comparison contour voxels of the 3D training image and comparison contour pixels of the 2D training image, which correspond with each other, by: identification of contour features of the 3D training image; identification of contour features of the 2D training image; selection of the contour features of the 2D training image and the contour features of the 3D training image, which correspond with each other; and selection of the comparison contour pixels of the 2D training image and the comparison contour voxels of the 3D training image as a function of the selected contour features of the 2D training image and the selected contour features of the 3D training image;
generating input data based on the 2D training image comprising training contour pixels and the 3D training image comprising training contour voxels, wherein the training contour pixels are dependent on training contour features of the 2D training image, wherein the training contour voxels are dependent on training contour features of the 3D training image, which training contour voxels of the 3D training image have a substantially perpendicular contour surface normal in respect of a specified projection direction;
applying the trained function to the input data for identification of training contour pixels of the 2D training image and training contour voxels of the 3D training image, which correspond with each other;
adjusting at least one parameter of the trained function based on a comparison of the training contour pixel with the comparison contour pixels and a comparison of training contour voxels corresponding thereto with the comparison contour voxels; and
providing the trained function.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein the 2D training image comprises a projection image of an examination region depicted in the 3D training image, wherein the specified projection direction is specified as a function of the projection image.

### Claim 10 (depends on 8)

The computer-implemented method of claim 8, wherein the generating of the input data further comprises generation of a training gradient image from the 3D training image by applying a projection mapping, and
wherein the input data is also based on the training gradient image.

### Claim 11 (depends on 10)

A computer-implemented method of claim 10, wherein the generating of the input data further comprises generation of a second training gradient image from the 2D training image, and
wherein the input data is also based on the second training gradient image.

### Claim 12 (depends on 8)

The computer-implemented method of claim 8, further comprising:
receiving a training movement parameter,
wherein the contour features of the 2D training image and the training contour features of the 2D training image are adjusted using the training movement parameter.

### Claim 13

A device for provision of a transformation instruction for registering a two-dimensional (2D) image with a three-dimensional (3D) image, the device comprising:
a processing unit having:
an arithmetic unit; and
an interface,
wherein the interface is configured to receive the 2D image and the 3D image,
wherein the arithmetic unit is configured to generate input data based on the 2D image comprising contour pixels and the 3D image comprising contour voxels, wherein the contour pixels are dependent on contour features of the 2D image, wherein the contour voxels are dependent on contour features of the 3D image, which contour voxels of the 3D image have a substantially perpendicular contour surface normal in respect of a specified projection direction,
wherein the arithmetic unit is further configured to apply a trained function to the input data for identification of contour pixels of the 2D image and contour voxels of the 3D image, which correspond with each other, wherein at least one parameter of the trained function is configured to be adjusted based on a comparison of training contour pixels with comparison contour pixels and a comparison of training contour voxels corresponding thereto with comparison contour voxels,
wherein the arithmetic unit is further configured to determine a transformation instruction based on the identified contour pixels of the 2D image and the contour voxels corresponding thereto of the 3D image for registering the 2D image with the 3D image, and
wherein the interface is further configured to provide the transformation instruction.

### Claim 14 (depends on 13)

The device of claim 13, wherein the device is a medical imaging device, and wherein the 2D image and the 3D image are images of an examination object.

### Claim 15

A training unit for provision of a trained function, the training unit comprising:
a training interface; and
a training arithmetic unit,
wherein the training interface is configured to receive a two-dimensional (2D) training image and a three-dimensional (3D) training image,
wherein the training arithmetic unit is configured to identify comparison contour voxels of the 3D training image and comparison contour pixels of the 2D training image, which correspond with each other, by: identification of contour features of the 3D training image; identification of contour features of the 2D training image; selection of the contour features of the 2D training image and the contour features of the 3D training image, which correspond with each other; and selection of the comparison contour pixels of the 2D training image and the comparison contour voxels of the 3D training image as a function of the selected contour features of the 2D training image and the selected contour features of the 3D training image,
wherein the training arithmetic unit is further configured to generate input data based on the 2D training image comprising training contour pixels and the 3D training image comprising training contour voxels, wherein the training contour pixels are dependent on training contour features of the 2D training image, wherein the training contour voxels are dependent on training contour features of the 3D training image, which training contour voxels of the 3D training image have a substantially perpendicular contour surface normal in respect of the specified projection direction,
wherein the training arithmetic unit is further configured to apply the trained function to the input data for identification of training contour pixels of the 2D training image and training contour voxels of the 3D training image, which correspond with each other,
wherein the training arithmetic unit is further configured to adjust at least one parameter of the trained function based on a comparison of the training contour pixels with the comparison contour pixels and a comparison of training contour voxels corresponding thereto with the comparison contour voxels, and wherein the training interface is further configured to provide the trained function.
