# US 2023/0410346 A1 · Object keypoint detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410346 A1
- Document ID: US20230410346A1
- Status: Published application
- Kind: A1
- Application number: 18/029,719
- Filed: 2021-09-22
- Published: 2023-12-21
- Earliest claimed: 2020-10-06
- Assignees: Koninklijke Philips N.V.
- Applicants: Koninklijke Philips N.V.
- Inventors: Sascha KRUEGER
- Web page: https://patentgrep.com/patent/US20230410346A1

## CPC

- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/764: using classification, e.g. of video objects
- G06V 20/647: {by matching two-dimensional images to three-dimensional objects}
- G06T 2207/10088: Magnetic resonance imaging [MRI]
- G06T 2207/20044: Skeletonization; Medial axis transform
- G06T 2207/20084: Artificial neural networks [ANN]
- G06V 2201/12: Acquisition of 3D measurements of objects
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A system for obtaining object keypoints for an object in a medical scanner, wherein the object keypoints are three dimensional, 3D, coordinates with respect to the medical scanner of pre-determined object parts. The system comprises a camera system for obtaining two dimensional, 2D, images of the object in the medical scanner, wherein the camera system comprises one or more cameras, and a processor. The processor is configured to obtain scanner variables from the medical scanner, wherein the scanner variables include the position of a part of the medical scanner which determines a relative position between the cameras in the camera system and the object. The processor determines object keypoint projections in 2D coordinates based on the 2D images from the camera system and determines the object keypoints in 3D coordinates by triangulating the object keypoint projections with respect to the camera system based on the scanner variables.

## Claims

### Claim 1

A system for obtaining object keypoints for an object in a medical scanner, wherein the object keypoints are three dimensional (3D) coordinates with respect to the medical scanner of pre-determined object parts, and wherein the object keypoints are defined inside the object parts, the system comprising:
a camera system for obtaining two dimensional (2D) images of the object in the medical scanner, wherein the camera system comprises:
one or more cameras;
a processor configured to:
obtain scanner variables from at least one of the medical scanner or the camera system, wherein the medical scanner contains a first moving part and/or wherein the camera system contains a second moving part and wherein the scanner variables contain the position of the first moving part of the medical scanner and/or the second moving part of the camera system;
determine object keypoint projections in 2D coordinates based in inputting the 2D images into a neural network trained to learn the spatial relationships between object keypoint projections corresponding to different parts of the object, wherein the neural network is configured to output the object keypoint projections in each one of the input images; and
determine the object keypoints in 3D coordinates by triangulating the object keypoint projections with respect to the camera system based on the scanner variables.

### Claim 2

(canceled)

### Claim 3 (depends on 2)

The system of claim 2, wherein the neural network is based on a U-Net neural network architecture.

### Claim 4 (depends on 1)

The system of claim 1, wherein the camera system comprises exactly one camera.

### Claim 5 (depends on 1)

The system of claim 1, wherein the medical scanner is a magnetic resonance imaging (MRI) scanner comprising:
a table for positioning a subject; and
an imaging section for obtaining medical images, and
wherein the scanner variables contain the position of the table (104) relative to the imaging section.

### Claim 6 (depends on 1)

The system of claim 1, wherein determining the object keypoints is further based on triangulating the object keypoint projections in a first image taken before a movement and the object keypoint projections in a second image taken after the movement based on the distance covered during the movement.

### Claim 7 (depends on 1)

The system of claim 1, wherein the processor is further configured to determine imaging parameters for the medical scanner based on the object keypoints.

### Claim 8

A method for obtaining object keypoints for an object in a medical scanner, wherein the object keypoints are three dimensional (3D) coordinates with respect to the medical scanner and wherein the object keypoints are defined inside the object parts, the method comprising:
obtaining two dimensional (2D) images of the object in the medical scanner from a camera system, wherein the camera system comprises one or more cameras;
obtaining scanner variables from at least one of the medical scanner or the camera system, wherein the medical scanner contains a first moving part and/or wherein the camera system contains a second moving part and wherein the scanner variables contain the position of the first moving part of the medical scanner and/or the second moving part of the camera system;
determining object keypoint projections in 2D coordinates based on inputting the 2D images into a neural network trained to learn the spatial relationships between object keypoint projections corresponding to different parts of the object, wherein the neural network is configured to output the object keypoint projections in each one of the input images; and
determining the object keypoints in 3D coordinates by triangulating the object keypoint projections with respect to the camera system based on the scanner variables.

### Claim 9

(canceled)

### Claim 10 (depends on 8)

The method of claim 8, wherein the neural network is based on a U-Net neural network architecture.

### Claim 11 (depends on 8)

The method of claim 8, wherein the camera system comprises exactly one camera.

### Claim 12 (depends on 8)

The method of claim 8, wherein the medical scanner is a magnetic resonance imaging, MRI, scanner comprising:
a table for positioning a subject; and
an imaging section for obtaining medical images, and
wherein the scanner variables contain the position of the table relative to the imaging section.

### Claim 13 (depends on 8)

The method of claim 8, wherein determining the object keypoints is further based on triangulating the object keypoint projections in a first image taken before a movement and the object keypoint projections in a second image taken after the movement based on the distance covered during the movement.

### Claim 14 (depends on 8)

The method of claim 8, further comprising determining imaging parameters for the medical scanner based on the object keypoints.

### Claim 15 (depends on 8)

A computer program product comprising computer program code stored on a non-transitory computer readable medium which, when executed on a computing device having a processing system, causes the processing system to perform all of the steps of the method according to claim 8.
