# US 2026/0087349 A1 · Hierarchical self-supervised visual language model with medical image localization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0087349 A1
- Document ID: US20260087349A1
- Status: Published application
- Kind: A1
- Application number: 18/897,090
- Filed: 2024-09-26
- Published: 2026-03-26
- Earliest claimed: 2024-09-26
- Assignees: Siemens Healthineers AG
- Applicants: Siemens Healthineers AG
- Inventors: Bogdan Georgescu; Awais Mansoor; Sasa Grbic; Dorin Comaniciu
- Web page: https://patentgrep.com/patent/US20260087349A1

## CPC

- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06T 7/0014: {using an image reference approach}
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems and methods for determining a correction between an input text-based query and one or more input medical images are provided. An input text-based query and one or more input medical images are received. The input text-based query is encoded into text features using a machine learning based text encoder network. The one or more input medical images are encoded into a spatial hierarchy of image features using a machine learning based image encoder network. A correlation is determined between the text features and the image features from a top-down of the spatial hierarchy of image features using the machine learning based text encoder network. The correlation between the text features and the image features is output.

## Claims

### Claim 1

A computer-implemented method comprising:
receiving an input text-based query and one or more input medical images;
encoding the input text-based query into text features using a machine learning based text encoder network;
encoding the one or more input medical images into a spatial hierarchy of image features using a machine learning based image encoder network;
determining a correlation between the text features and the spatial hierarchy of image features according to a top-down analysis of the spatial hierarchy of image features using the machine learning based text encoder network; and
outputting the correlation between the text features and the spatial hierarchy of image features.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein determining a correlation between the text features and the spatial hierarchy of image features according to a top-down analysis of the spatial hierarchy of image features using the machine learning based text encoder network comprises:
determining a correlation score for each respective image feature of the spatial hierarchy of image features, the correlation score representing a correlation between the respective image feature and the text features.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein determining a correlation between the text features and the image features according to a top-down analysis of the spatial hierarchy of image features using the machine learning based text encoder network comprises:
determining highly correlated image features of the spatial hierarchy of image feature based on the correlation scores; and
mapping regions corresponding to the highly correlated image features to the one or more input medical images.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the machine learning based encoder network is trained by:
receiving a training text-based query and one or more training medical images;
encoding the training text-based query into training text features using the machine learning based text encoder network;
encoding the one or more training medical images into a spatial hierarchy of training image features using the machine learning based image encoder network;
selecting a subset of the spatial hierarchy of training image features based on correlations between particular features of the spatial hierarchy of training image features and the text features;
training the machine learning based text encoder network for determining a correlation between input text features and input image features based on the text features and the selected subset of the spatial hierarchy of training image features; and
outputting the trained machine learning based text encoder network.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein selecting a subset of the spatial hierarchy of training image features based on correlations between particular features of the spatial hierarchy of training image features and the text features comprises:
determining a correlation between each feature of the spatial hierarchy of training image features and the text features; and
selecting the features of the spatial hierarchy of training image features with a highest correlation as the subset of the spatial hierarchy of training image features.

### Claim 6 (depends on 4)

The computer-implemented method of claim 4, wherein selecting a subset of the spatial hierarchy of training image features based on correlations between particular features of the spatial hierarchy of training image features and the text features comprises:
determining a correlation between each feature of the spatial hierarchy of training image features and image features of one or more example images known to be correlated with the training text-based query; and
selecting the subset of the spatial hierarchy of training image features based on the correlation.

### Claim 7 (depends on 4)

The computer-implemented method of claim 4, wherein training the machine learning based text encoder network for determining a correlation between input text features and input image features based on the text features and the selected subset of the spatial hierarchy of training image features comprises:
training the machine learning based text encoder network using contrastive learning.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein training the machine learning based text encoder network using contrastive learning comprises:
training the machine learning based text encoder network using the selected subset of the spatial hierarchy of training image features and the text features as positive examples and unselected image features of the spatial hierarchy of training image features and the text features as negative examples.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the machine learning based text encoder network comprises a language model.

### Claim 10

An apparatus comprising:
means for receiving an input text-based query and one or more input medical images;
means for encoding the input text-based query into text features using a machine learning based text encoder network;
means for encoding the one or more input medical images into a spatial hierarchy of image features using a machine learning based image encoder network;
means for determining a correlation between the text features and the spatial hierarchy of image features according to a top-down analysis of the spatial hierarchy of image features using the machine learning based text encoder network; and
means for outputting the correlation between the text features and the spatial hierarchy of image features.

### Claim 11 (depends on 10)

The apparatus of claim 10, wherein the means for determining a correlation between the text features and the spatial hierarchy of image features according to a top-down analysis of the spatial hierarchy of image features using the machine learning based text encoder network comprises:
means for determining a correlation score for each respective image feature of the spatial hierarchy of image features, the correlation score representing a correlation between the respective image feature and the text features.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein the means for determining a correlation between the text features and the image features according to a top-down analysis of the spatial hierarchy of image features using the machine learning based text encoder network comprises:
means for determining highly correlated image features of the spatial hierarchy of image feature based on the correlation scores; and
means for mapping regions corresponding to the highly correlated image features to the one or more input medical images.

### Claim 13 (depends on 10)

The apparatus of claim 10, wherein the machine learning based encoder network is trained by:
receiving a training text-based query and one or more training medical images;
encoding the training text-based query into training text features using the machine learning based text encoder network;
encoding the one or more training medical images into a spatial hierarchy of training image features using the machine learning based image encoder network;
selecting a subset of the spatial hierarchy of training image features based on correlations between particular features of the spatial hierarchy of training image features and the text features;
training the machine learning based text encoder network for determining a correlation between input text features and input image features based on the text features and the selected subset of the spatial hierarchy of training image features; and
outputting the trained machine learning based text encoder network.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein selecting a subset of the spatial hierarchy of training image features based on correlations between particular features of the spatial hierarchy of training image features and the text features comprises:
determining a correlation between each feature of the spatial hierarchy of training image features and the text features; and
selecting the features of the spatial hierarchy of training image features with a highest correlation as the subset of the spatial hierarchy of training image features.

### Claim 15

A non-transitory computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out operations comprising:
receiving an input text-based query and one or more input medical images;
encoding the input text-based query into text features using a machine learning based text encoder network;
encoding the one or more input medical images into a spatial hierarchy of image features using a machine learning based image encoder network;
determining a correlation between the text features and the spatial hierarchy of image features according to a top-down analysis of the spatial hierarchy of image features using the machine learning based text encoder network; and
outputting the correlation between the text features and the spatial hierarchy of image features.

### Claim 16 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the machine learning based encoder network is trained by:
receiving a training text-based query and one or more training medical images;
encoding the training text-based query into training text features using the machine learning based text encoder network;
encoding the one or more training medical images into a spatial hierarchy of training image features using the machine learning based image encoder network;
selecting a subset of the spatial hierarchy of training image features based on correlations between particular features of the spatial hierarchy of training image features and the text features;
training the machine learning based text encoder network for determining a correlation between input text features and input image features based on the text features and the selected subset of the spatial hierarchy of training image features; and
outputting the trained machine learning based text encoder network.

### Claim 17 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein selecting a subset of the spatial hierarchy of training image features based on correlations between particular features of the spatial hierarchy of training image features and the text features comprises:
determining a correlation between each feature of the spatial hierarchy of training image features and image features of one or more example images known to be correlated with the training text-based query; and
selecting the subset of the spatial hierarchy of training image features based on the correlation.

### Claim 18 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein training the machine learning based text encoder network for determining a correlation between input text features and input image features based on the text features and the selected subset of the spatial hierarchy of training image features comprises:
training the machine learning based text encoder network using contrastive learning.

### Claim 19 (depends on 18)

The non-transitory computer-readable storage medium of claim 18, wherein training the machine learning based text encoder network using contrastive learning comprises:
training the machine learning based text encoder network using the selected subset of the spatial hierarchy of training image features and the text features as positive examples and unselected image features of the spatial hierarchy of training image features and the text features as negative examples.

### Claim 20 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the machine learning based text encoder network comprises a language model.

### Claim 21

A computer-implemented method comprising:
receiving a training text-based query and one or more training medical images;
encoding the training text-based query into training text features using a machine learning based text encoder network;
encoding the one or more training medical images into a spatial hierarchy of image features using a machine learning based image encoder network;
selecting a subset of the spatial hierarchy of image features based on correlations between particular features of the spatial hierarchy of image features and the text features;
training the machine learning based text encoder network for determining a correlation between input text features and input image features based on the text features and the selected subset of the spatial hierarchy of image features; and
outputting the trained machine learning based text encoder network.

### Claim 22 (depends on 21)

The computer-implemented method of claim 21, wherein selecting a subset of the spatial hierarchy of image features based on correlations between particular features of the spatial hierarchy of image features and the text features comprises:
determining a correlation between each feature of the spatial hierarchy of image features and the text features; and
selecting the features of the spatial hierarchy of image features with a highest correlation as the subset of the spatial hierarchy of image features.

### Claim 23 (depends on 21)

The computer-implemented method of claim 21, wherein selecting a subset of the spatial hierarchy of image features based on correlations between particular features of the spatial hierarchy of image features and the text features comprises:
determining a correlation between each feature of the spatial hierarchy of image features and image features of one or more example images known to be correlated with the training text-based query; and
selecting the subset of the spatial hierarchy of image features based the correlation.
