# US 2025/0166170 A1 · Machine learning based medical imaging analysis using few shot learning with task instructions

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0166170 A1
- Document ID: US20250166170A1
- Status: Published application
- Kind: A1
- Application number: 18/512,237
- Filed: 2023-11-17
- Published: 2025-05-22
- Earliest claimed: 2023-11-17
- Assignees: Siemens Healthineers AG
- Applicants: Siemens Healthineers AG
- Inventors: Bogdan Georgescu; Awais Mansoor; Sasa Grbic; Dorin Comaniciu
- Web page: https://patentgrep.com/patent/US20250166170A1

## CPC

- G06T 7/0012: {Biomedical image inspection}
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06T 2207/10081: Computed x-ray tomography [CT]
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20092: Interactive image processing based on input by user
- G06T 2207/30004: Biomedical image processing
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems and methods for performing a medical imaging analysis task using a machine learning based task network based on task instructions are provided. One or more input medical images of a patient and task instructions for performing a medical imaging analysis task are received. The one or more input medical images are encoded into imaging features using an image encoder network. The task instructions are encoded into text features using a text encoder network. The medical imaging analysis task is performed based on the imaging features and the text features using a machine learning based task network. Results of the medical imaging analysis task are output.

## Claims

### Claim 1

A computer-implemented method comprising:
receiving 1) one or more input medical images of a patient and 2) task instructions for performing a medical imaging analysis task;
encoding the one or more input medical images into imaging features using an image encoder network;
encoding the task instructions into text features using a text encoder network;
performing the medical imaging analysis task based on the imaging features and the text features using a machine learning based task network; and
outputting results of the medical imaging analysis task.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the task instructions comprise references to image regions in at least one of the one or more input medical images.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the task instructions comprise anatomical knowledge and task knowledge, the task knowledge comprising at least one of a description of an anatomical abnormality, how the anatomical abnormality is represented, and how the anatomical abnormality can be detected in the one or more input medical images.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the task instructions are user-defined.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the text features and the imaging features are aligned in a same latent space.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the machine learning based task network is trained with self-supervised learning based on unannotated training medical images and text.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the machine learning based task network is trained with few shot learning using annotated training medical images and annotated task descriptions.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the machine learning based task network comprises an LLM (large language model) based task network.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein:
receiving 1) one or more input medical images of a patient and 2) task instructions for performing a medical imaging analysis task comprises receiving text-based medical data of the patient; and
encoding the task instructions into text features using a text encoder network comprises encoding the task instructions and the text-based medical data into the text features using the text encoder network.

### Claim 10

An apparatus comprising:
means for receiving 1) one or more input medical images of a patient and 2) task instructions for performing a medical imaging analysis task;
means for encoding the one or more input medical images into imaging features using an image encoder network;
means for encoding the task instructions into text features using a text encoder network;
means for performing the medical imaging analysis task based on the imaging features and the text features using a machine learning based task network; and
means for outputting results of the medical imaging analysis task.

### Claim 11 (depends on 10)

The apparatus of claim 10, wherein the task instructions comprise references to image regions in at least one of the one or more input medical images.

### Claim 12 (depends on 10)

The apparatus of claim 10, wherein the task instructions comprise anatomical knowledge and task knowledge, the task knowledge comprising at least one of a description of an anatomical abnormality, how the anatomical abnormality is represented, and how the anatomical abnormality can be detected in the one or more input medical images.

### Claim 13 (depends on 10)

The apparatus of claim 10, wherein the task instructions are user-defined.

### Claim 14

A non-transitory computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out operations comprising:
receiving 1) one or more input medical images of a patient and 2) task instructions for performing a medical imaging analysis task;
encoding the one or more input medical images into imaging features using an image encoder network;
encoding the task instructions into text features using a text encoder network;
performing the medical imaging analysis task based on the imaging features and the text features using a machine learning based task network; and
outputting results of the medical imaging analysis task.

### Claim 15 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein the text features and the imaging features are aligned in a same latent space.

### Claim 16 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein the machine learning based task network is trained with self-supervised learning based on unannotated training medical images and text.

### Claim 17 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein the machine learning based task network is trained with few shot learning using annotated training medical images and annotated task descriptions.

### Claim 18

A computer-implemented method comprising:
receiving 1) one or more training medical images and 2) training task instructions for performing a medical imaging analysis task;
encoding the one or more training medical images into imaging features using a pretrained image encoder network;
encoding the training task instructions into text features using a pretrained text encoder network;
training a machine learning based task network for performing the medical imaging analysis task based on the imaging features and the text features; and
outputting the trained machine learning based task network.

### Claim 19 (depends on 18)

The computer-implemented method of claim 18, wherein the machine learning based task network is trained with self-supervised learning based on unannotated training medical images and text.

### Claim 20 (depends on 18)

The computer-implemented method of claim 18, wherein the machine learning based task network is trained with few shot learning using annotated training medical images and annotated task descriptions.
