# US 10,373,313 B2 · Spatially consistent multi-scale anatomical landmark detection in incomplete 3D-CT data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,373,313 B2
- Document ID: US10373313B2
- Status: Granted
- Kind: B2
- Application number: 15/695,112
- Filed: 2017-09-05
- Published: 2018-09-06
- Granted: 2019-08-06
- Earliest claimed: 2017-03-02
- Assignees: Siemens Healthcare GmbH
- Applicants: Siemens Healthcare GmbH
- Inventors: Florin Cristian Ghesu; Bogdan Georgescu; Sasa Grbic; Dorin Comaniciu
- Web page: https://patentgrep.com/patent/US10373313B2

## CPC

- G06T 7/75: {involving models}
- G06T 7/0012: {Biomedical image inspection}
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30004: Biomedical image processing
- G06V 2201/031: of internal organs
- G06V 2201/033: of skeletal patterns
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method and system for automated spatially-consistent multi-scale detection of anatomical landmarks in medical images is disclosed. A discrete scale-space representation of a medical image of a patient is generated. A plurality of anatomical landmarks are detected at a coarsest scale-level of the discrete scale-space representation of the medical image using a respective trained search model trained at the coarsest scale-level for each of the plurality of anatomical landmarks. Spatial coherence of the detected anatomical landmarks is enforced by fitting a learned robust shape model of the plurality of anatomical landmarks to the detected anatomical landmarks at the coarsest scale-level to robustly determine a set of the anatomical landmarks within a field-of-view of the medical image. The detected landmark location for each of the landmarks in the set of anatomical landmarks is refined at each remaining scale-level of the discrete scale-space representation of the medical image using, for each landmark, a respective trained search model trained at each remaining scale-level and constrained based on the predicted landmark location at a previous scale-level.

## Claims

### Claim 1

A method for automated spatially-consistent detection of anatomical landmarks in a medical image of a patient, comprising:
generating a discrete scale-space representation of a medical image of a patient, wherein the discrete scale-space representation of the medical image includes a plurality of scale-levels;
detecting a plurality of anatomical landmarks at a coarsest scale-level of the discrete scale-space representation of the medical image using a respective trained search model trained to predict a trajectory from a starting location to a predicted landmark location at the coarsest scale-level for each of the plurality of anatomical landmarks;
enforcing spatial coherence of the detected anatomical landmarks by fitting a learned shape model of the plurality of anatomical landmarks to the detected anatomical landmarks at the coarsest scale-level to robustly determine a set of the anatomical landmarks within a field-of-view of the medical image; and
refining the predicted landmark location for each of the landmarks in the set of anatomical landmarks at each remaining scale-level of the discrete scale-space representation of the medical image using, for each landmark in the set of anatomical landmarks, a respective trained search model trained to predict a trajectory to the predicted landmark location at each remaining scale-level, wherein the trained search model for each remaining scale-level for each landmark is constrained based on a range surrounding the predicted landmark location for that landmark at a previous scale-level.

### Claim 2 (depends on 1)

The method of claim 1, wherein for each anatomical landmark, the trained search model for each scale-level is an intelligent artificial agent that predicts the trajectory to the predicted landmark location at the corresponding scale-level by iteratively using a trained deep neural network (DNN) that inputs a region of interest surrounding a current location in the corresponding scale-level of the discrete scale-space representation of the medical image to calculate action-values corresponding to actions that move the current location in different direction and selecting an action having the highest action-value to be applied to the current location.

### Claim 3 (depends on 2)

The method of claim 2, wherein the discrete scale-space representation of the medical image includes M scale-levels from 0 to M−1, where scale-level 0 of the discrete scale-space representation of the medical image is the medical image at its original resolution and scale-level M−1 is the coarsest scale level, and the trained search model for the coarsest scale level M−1 for each landmark is a global search model that starts the search for the trajectory to the predicted landmark location from the center of the coarsest scale-level of the discrete scale-space representation of the medical image.

### Claim 4 (depends on 3)

The method of claim 3, wherein the trained search model for the coarsest scale level M−1 for each landmark is trained to reward trajectories that leave an image space of a training image through a correct image border when the corresponding landmark is missing from the training image.

### Claim 5 (depends on 4)

The method of claim 4, wherein detecting a plurality of anatomical landmarks at a coarsest scale-level of the discrete scale-space representation of the medical image using a respective trained search model trained to predict a trajectory from a starting location to a predicted landmark location at the coarsest scale-level for each of the plurality of anatomical landmarks comprises:
for each of the plurality of anatomical landmarks, predicting a trajectory to a predicted landmark location in the coarsest scale-level of the discrete scale-space representation of the medical image or a trajectory that leaves the image space of the discrete scale-space representation of the medical image.

### Claim 6 (depends on 3)

The method of claim 3, wherein for each remaining scale-level t from t=M−2, . . . , 0, the trained search model for scale-level t for each anatomical landmark in the set of anatomical landmarks starts the search for the trajectory to the predicted landmark location from a convergence point of the trained search model for the previous scale level t+1 for that anatomical landmark.

### Claim 7 (depends on 1)

The method of claim 1, wherein enforcing spatial coherence of the detected anatomical landmarks by fitting a learned shape model of the plurality of anatomical landmarks to the detected anatomical landmarks at the coarsest scale-level to robustly determine a set of the anatomical landmarks within a field-of-view of the medical image comprises:
fitting the learned shape model of the plurality of anatomical landmarks to the detected anatomical landmarks at the coarsest scale-level using an M-estimator sampling consensus based on random 3-point samples from the detected anatomical landmarks at the coarsest scale-level.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
automatically determining a scan range of the medical image based on the set of anatomical landmarks at a final scale-level of the discrete scale-space representation of the medical image based on the learned shape model of the plurality of anatomical landmarks.

### Claim 9 (depends on 8)

The method of claim 8, wherein automatically determining a scan range of the medical image based on the set of anatomical landmarks at a final scale-level of the discrete scale-space representation of the medical image based on the learned shape model of the plurality of anatomical landmarks comprises:
interpolating scan range values between detected landmark locations in the in the final scale-level of the discrete scale-space representation of the medical image of a minimum anatomical landmark along a z-axis in the learned shape model and a maximum anatomical landmark along the z-axis in the learned shape model, wherein the detected landmark location of the minimum anatomical landmark defines a 0% point and the detected landmark location of the maximum anatomical landmark defines a 100% point; and
extrapolating scan range values above the detected landmark location of the maximum anatomical landmark or below the detected landmark location of the minimum anatomical landmark using an estimated scale parameter of the fitted shape model.

### Claim 10

An apparatus for automated spatially-consistent detection of anatomical landmarks in a medical image of a patient, comprising:
means for generating a discrete scale-space representation of a medical image of a patient, wherein the discrete scale-space representation of the medical image includes a plurality of scale-levels;
means for detecting a plurality of anatomical landmarks at a coarsest scale-level of the discrete scale-space representation of the medical image using a respective trained search model trained to predict a trajectory from a starting location to a predicted landmark location at the coarsest scale-level for each of the plurality of anatomical landmarks;
means for enforcing spatial coherence of the detected anatomical landmarks by fitting a learned shape model of the plurality of anatomical landmarks to the detected anatomical landmarks at the coarsest scale-level to robustly determine a set of the anatomical landmarks within a field-of-view of the medical image; and
means for refining the predicted landmark location for each of the landmarks in the set of anatomical landmarks at each remaining scale-level of the discrete scale-space representation of the medical image using, for each landmark in the set of anatomical landmarks, a respective trained search model trained to predict a trajectory to the predicted landmark location at each remaining scale-level, wherein the trained search model for each remaining scale-level for each landmark is constrained based on a range surrounding the predicted landmark location for that landmark at a previous scale-level.

### Claim 11 (depends on 10)

The apparatus of claim 10, wherein for each anatomical landmark, the trained search model for each scale-level is an intelligent artificial agent that predicts the trajectory to the predicted landmark location at the corresponding scale-level by iteratively using a trained deep neural network (DNN) that inputs a region of interest surrounding a current location in the corresponding scale-level of the discrete scale-space representation of the medical image to calculate action-values corresponding to actions that move the current location in different direction and selecting an action having the highest action-value to be applied to the current location.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein the discrete scale-space representation of the medical image includes M scale-levels from 0 to M−1, where scale-level 0 of the discrete scale-space representation of the medical image is the medical image at its original resolution and scale-level M−1 is the coarsest scale level, and the trained search model for the coarsest scale level M−1 for each landmark is a global search model that starts the search for the trajectory to the predicted landmark location from the center of the coarsest scale-level of the discrete scale-space representation of the medical image.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein the trained search model for the coarsest scale level M−1 for each landmark is trained to reward trajectories that leave an image space of a training image through a correct image border when the corresponding landmark is missing from the training image.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein the means for detecting a plurality of anatomical landmarks at a coarsest scale-level of the discrete scale-space representation of the medical image using a respective trained search model trained to predict a trajectory from a starting location to a predicted landmark location at the coarsest scale-level for each of the plurality of anatomical landmarks comprises:
means for predicting, for each of the plurality of anatomical landmarks, a trajectory to a predicted landmark location in the coarsest scale-level of the discrete scale-space representation of the medical image or a trajectory that leaves the image space of the discrete scale-space representation of the medical image.

### Claim 15 (depends on 12)

The apparatus of claim 12, wherein for each remaining scale-level t from t=M−2, . . . , 0, the trained search model for scale-level t for each anatomical landmark in the set of anatomical landmarks starts the search for the trajectory to the predicted landmark location from a convergence point of the trained search model for the previous scale level t+1 for that anatomical landmark.

### Claim 16 (depends on 10)

The apparatus of claim 10, further comprising:
means for automatically determining a scan range of the medical image based on the set of anatomical landmarks at a final scale-level of the discrete scale-space representation of the medical image based on the learned shape model of the plurality of anatomical landmarks.

### Claim 17

A non-transitory computer readable medium storing computer program instructions for automated spatially-consistent detection of anatomical landmarks in a medical image of a patient, the computer program instructions when executed by a processor cause the processor to perform operations comprising:
generating a discrete scale-space representation of a medical image of a patient, wherein the discrete scale-space representation of the medical image includes a plurality of scale-levels;
detecting a plurality of anatomical landmarks at a coarsest scale-level of the discrete scale-space representation of the medical image using a respective trained search model trained to predict a trajectory from a starting location to a predicted landmark location at the coarsest scale-level for each of the plurality of anatomical landmarks;
enforcing spatial coherence of the detected anatomical landmarks by fitting a learned shape model of the plurality of anatomical landmarks to the detected anatomical landmarks at the coarsest scale-level to robustly determine a set of the anatomical landmarks within a field-of-view of the medical image; and
refining the predicted landmark location for each of the landmarks in the set of anatomical landmarks at each remaining scale-level of the discrete scale-space representation of the medical image using, for each landmark in the set of anatomical landmarks, a respective trained search model trained to predict a trajectory to the predicted landmark location at each remaining scale-level, wherein the trained search model for each remaining scale-level for each landmark is constrained based on a range surrounding the predicted landmark location for that landmark at a previous scale-level.

### Claim 18 (depends on 17)

The non-transitory computer readable medium of claim 17, wherein for each anatomical landmark, the trained search model for each scale-level is an intelligent artificial agent that predicts the trajectory to the predicted landmark location at the corresponding scale-level by iteratively using a trained deep neural network (DNN) that inputs a region of interest surrounding a current location in the corresponding scale-level of the discrete scale-space representation of the medical image to calculate action-values corresponding to actions that move the current location in different direction and selecting an action having the highest action-value to be applied to the current location.

### Claim 19 (depends on 18)

The non-transitory computer readable medium of claim 18, wherein the discrete scale-space representation of the medical image includes M scale-levels from 0 to M−1, where scale-level 0 of the discrete scale-space representation of the medical image is the medical image at its original resolution and scale-level M−1 is the coarsest scale level, and the trained search model for the coarsest scale level M−1 for each landmark is a global search model that starts the search for the trajectory to the predicted landmark location from the center of the coarsest scale-level of the discrete scale-space representation of the medical image.

### Claim 20 (depends on 19)

The non-transitory computer readable medium of claim 19, wherein the trained search model for the coarsest scale level M−1 for each landmark is trained to reward trajectories that leave an image space of a training image through a correct image border when the corresponding landmark is missing from the training image.

### Claim 21 (depends on 20)

The non-transitory computer readable medium of claim 20, wherein detecting a plurality of anatomical landmarks at a coarsest scale-level of the discrete scale-space representation of the medical image using a respective trained search model trained to predict a trajectory from a starting location to a predicted landmark location at the coarsest scale-level for each of the plurality of anatomical landmarks comprises:
for each of the plurality of anatomical landmarks, predicting a trajectory to a predicted landmark location in the coarsest scale-level of the discrete scale-space representation of the medical image or a trajectory that leaves the image space of the discrete scale-space representation of the medical image.

### Claim 22 (depends on 19)

The non-transitory computer readable medium of claim 19, wherein for each remaining scale-level t from t=M−2, . . . , 0, the trained search model for scale-level t for each anatomical landmark in the set of anatomical landmarks starts the search for the trajectory to the predicted landmark location from a convergence point of the trained search model for the previous scale level t+1 for that anatomical landmark.

### Claim 23 (depends on 17)

The non-transitory computer readable medium of claim 17, wherein enforcing spatial coherence of the detected anatomical landmarks by fitting a learned shape model of the plurality of anatomical landmarks to the detected anatomical landmarks at the coarsest scale-level to robustly determine a set of the anatomical landmarks within a field-of-view of the medical image comprises:
fitting the learned shape model of the plurality of anatomical landmarks to the detected anatomical landmarks at the coarsest scale-level using an M-estimator sampling consensus based on random 3-point samples from the detected anatomical landmarks at the coarsest scale-level.

### Claim 24 (depends on 17)

The non-transitory computer readable medium of claim 17, wherein the operations further comprise:
automatically determining a scan range of the medical image based on the set of anatomical landmarks at a final scale-level of the discrete scale-space representation of the medical image based on the learned shape model of the plurality of anatomical landmarks.

### Claim 25 (depends on 24)

The non-transitory computer readable medium of claim 24, wherein automatically determining a scan range of the medical image based on the set of anatomical landmarks at a final scale-level of the discrete scale-space representation of the medical image based on the learned shape model of the plurality of anatomical landmarks comprises:
interpolating scan range values between detected landmark locations in the in the final scale-level of the discrete scale-space representation of the medical image of a minimum anatomical landmark along a z-axis in the learned shape model and a maximum anatomical landmark along the z-axis in the learned shape model, wherein the detected landmark location of the minimum anatomical landmark defines a 0% point and the detected landmark location of the maximum anatomical landmark defines a 100% point; and
extrapolating scan range values above the detected landmark location of the maximum anatomical landmark or below the detected landmark location of the minimum anatomical landmark using an estimated scale parameter of the fitted shape model.