# US 2026/0094693 A1 · Multi-modal multi-task foundational models for medical image manipulation and information retrieval

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0094693 A1
- Document ID: US20260094693A1
- Status: Published application
- Kind: A1
- Application number: 18/898,763
- Filed: 2024-09-27
- Published: 2026-04-02
- Earliest claimed: 2024-09-27
- Assignees: Siemens Healthineers AG
- Applicants: Siemens Healthineers AG
- Inventors: Alexandru Constantin Serban; Mehmet Akif Gulsun; Puneet Sharma; Dorin Comaniciu
- Web page: https://patentgrep.com/patent/US20260094693A1

## CPC

- G16H 30/40: for processing medical images, e.g. editing
- G06F 40/279: Recognition of textual entities
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/82: using neural networks
- G06V 10/945: {User interactive design; Environments; Toolboxes}
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 2015/223: {Execution procedure of a spoken command}
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Systems and methods for automatically performing one or more actions on one or more medical applications are provided. Text-based instructions are received. The text-based instructions are encoded into text features using a machine learning based text encoder network. One or more instructions for performing by one or more medical applications are determined using a policy module based on the text features. The one or more instructions are performed by the one or more medical applications to generate a response to the text-based instructions. The response to the text-based instructions is output.

## Claims

### Claim 1

A computer-implemented method comprising:
receiving text-based instructions;
encoding the text-based instructions into text features using a machine learning based text encoder network;
determining one or more instructions for performing by one or more medical applications using a policy module based on the text features;
performing the one or more instructions by the one or more medical applications to generate a response to the text-based instructions; and
outputting the response to the text-based instructions.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
receiving one or more medical images; and
encoding the one or more medical images into image features using a machine learning based image encoder network,
wherein determining one or more instructions for performing by one or more medical applications using a policy module based on the text features comprises determining the one or more actions further based on the image features.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein:
performing the one or more instructions by the one or more medical applications to generate a response to the text-based instructions comprises performing the one or more instructions the one or more medical applications to modify the one or more medical images, and
outputting the response to the text-based instructions comprises outputting the one or more modified medical images.

### Claim 4 (depends on 2)

The computer-implemented method of claim 2, wherein the machine learning based text encoder network and the machine learning based image encoder network are trained to generate similar features for associated text-based instructions and medical images.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, further comprising:
adapting the policy module based on user feedback to the response to the text-based instructions.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more instructions comprise at least one of: one or more medical image analysis tasks performed on one or more medical images, functions to derive findings from the one or more medical images, functions to apply transformations on the one or more medical images, functions to derive information from the one or more medical images, or outputting text to a machine learning based model.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more instructions comprise one or more APIs (application programming interfaces).

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein receiving text-based instructions comprises:
receiving spoken instructions from a user; and
converting the spoken instructions to the text-based instructions.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the machine learning based text encoder network comprises a language model.

### Claim 10

An apparatus comprising:
means for receiving text-based instructions;
means for encoding the text-based instructions into text features using a machine learning based text encoder network;
means for determining one or more instructions for performing by one or more medical applications using a policy module based on the text features;
means for performing the one or more instructions by the one or more medical applications to generate a response to the text-based instructions; and
means for outputting the response to the text-based instructions.

### Claim 11 (depends on 10)

The apparatus of claim 10, further comprising:
means for receiving one or more medical images; and
means for encoding the one or more medical images into image features using a machine learning based image encoder network,
wherein the means for determining one or more instructions for performing by one or more medical applications using a policy module based on the text features comprises means for determining the one or more actions further based on the image features.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein:
the means for performing the one or more instructions by the one or more medical applications to generate a response to the text-based instructions comprises means for performing the one or more instructions by the one or more medical applications to modify the one or more medical images, and
the means for outputting the response to the text-based instructions comprises means for outputting the one or more modified medical images.

### Claim 13 (depends on 11)

The apparatus of claim 11, wherein the machine learning based text encoder network and the machine learning based image encoder network are trained to generate similar features for associated text-based instructions and medical images.

### Claim 14 (depends on 10)

The apparatus of claim 10, further comprising:
means for adapting the policy module based on user feedback to the response to the text-based instructions.

### Claim 15

A non-transitory computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out operations comprising:
receiving text-based instructions;
encoding the text-based instructions into text features using a machine learning based text encoder network;
determining one or more instructions for performing by one or more medical applications using a policy module based on the text features;
performing the one or more instructions by the one or more medical applications to generate a response to the text-based instructions; and
outputting the response to the text-based instructions.

### Claim 16 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, the operations further comprising:
receiving one or more medical images; and
encoding the one or more medical images into image features using a machine learning based image encoder network,
wherein determining one or more instructions for performing by one or more medical applications using a policy module based on the text features comprises determining the one or more instructions further based on the image features.

### Claim 17 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the one or more instructions comprise at least one of: one or more medical image analysis tasks performed on one or more medical images, functions to derive findings from the one or more medical images, functions to apply transformations on the one or more medical images, functions to derive information from the one or more medical images, or outputting text to a machine learning based model.

### Claim 18 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the one or more instructions comprise one or more APIs (application programming interfaces).

### Claim 19 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein receiving text-based instructions comprises:
receiving spoken instructions from a user; and
converting the spoken instructions to the text-based instructions.

### Claim 20 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the machine learning based text encoder network comprises a language model.
