# US 2024/0202546 A1 · Input generation for multimodal learning based machine learning models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202546 A1
- Document ID: US20240202546A1
- Status: Published application
- Kind: A1
- Application number: 18/067,096
- Filed: 2022-12-16
- Published: 2024-06-20
- Earliest claimed: 2022-12-16
- Assignees: Sap SE
- Applicants: Sap SE
- Inventors: Tassilo Klein; Moin Nabi
- Web page: https://patentgrep.com/patent/US20240202546A1

## CPC

- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06N 20/00: Machine learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method and system for generating inputs specific to a multimodal learning based machine learning model. The generating of inputs specific to the multimodal learning based machine learning model from training dataset comprises image data, the generating including: determining a plurality of permutations based on the one or more images included in the image data, and applying orthogonal super-positioning relative to the plurality of permutations. The method further comprises providing the inputs that are generated, based on the orthogonal super-positioning, into the multimodal learning based machine learning model, and generating, by the multimodal learning based machine learning model, a prediction specific to at least one image of the one or more images.

## Claims

### Claim 1

A computer-implemented method comprising:
generating inputs specific to a multimodal learning based machine learning model from a training dataset comprising image data, the image data comprising a plurality of rules specific to one or more images and a plurality of characterizations representative of orientations of the one or more images, the generating comprising:
determining a plurality of permutations based on the one or more images included in the image data, and
applying orthogonal super-positioning relative to the plurality of permutations;
providing the inputs that are generated, based on the orthogonal super-positioning, into the multimodal learning based machine learning model; and
generating, by the multimodal learning based machine learning model, a prediction specific to at least one image of the one or more images.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the multimodal learning based machine learning model is a contrastive language-image pre-training model.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein at least one of the plurality of rules specific to the one or more images corresponds to a color alteration rule.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein at least one of the plurality of rules specific to the one or more images corresponds to a Boolean rule.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein at least one of the plurality of rules specific to the one or more images comprises an orientation specific to each of the one or more images.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the applying of the orthogonal super-positioning relative to the plurality of permutations comprises:
associating at least a first subset of the plurality of permutations with a first parameter; and
associating at least a second subset of the plurality of permutations with a second parameter, wherein the second parameter is oriented orthogonally with respect to the first parameter.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein the applying of the orthogonal super-positioning relative to the plurality of permutations further comprises associating at least a third subset of the plurality of permutations with a third parameter.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, further comprising associating at least a fourth subset of the plurality of permutations with a fourth parameter, wherein the fourth parameter is oriented orthogonally with respect to the third parameter.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, further comprising:
receiving a query regarding at least one image of the one or more images, wherein the image of the one or more images comprises one or more objects; and
wherein the generating of the prediction specific to the at least one image of the one or more images comprises identifying text that is representative of the one or more objects of the at least one image.

### Claim 10

A system comprising:
at least one data processor; and
at least one memory storing instructions, which when executed by the at least one data processor, cause operations comprising:
generating inputs specific to a multimodal learning based machine learning model from a training dataset comprising image data, the image data comprising a plurality of rules specific to one or more images and a plurality of characterizations representative of orientations of the one or more images, the generating comprising:
determining a plurality of permutations based on the one or more images included in the image data, and
applying orthogonal super-positioning relative to the plurality of permutations;
providing the inputs that are generated, based on the orthogonal super-positioning, into the multimodal learning based machine learning model; and
generating, by the multimodal learning based machine learning model, a prediction specific to at least one image of the one or more images.

### Claim 11 (depends on 10)

The system of claim 10, wherein the multimodal learning based machine learning model is a contrastive language-image pre-training model.

### Claim 12 (depends on 10)

The system of claim 10, wherein at least one of the plurality of rules specific to the one or more images corresponds to a color alteration rule.

### Claim 13 (depends on 10)

The system of claim 10, wherein at least one of the plurality of rules specific to the one or more images corresponds to a Boolean rule.

### Claim 14 (depends on 10)

The system of claim 10, wherein at least one of the plurality of rules specific to the one or more images comprises an orientation specific to each of the one or more images.

### Claim 15 (depends on 10)

The system of claim 10, wherein the one of the operations of applying of the orthogonal super-positioning relative to the plurality of permutations comprises:
associating at least a first subset of the plurality of permutations with a first parameter; and
associating at least a second subset of the plurality of permutations with a second parameter, wherein the second parameter is oriented orthogonally with respect to the first parameter.

### Claim 16 (depends on 15)

The system of claim 15, wherein the operations further comprise associating at least a third subset of the plurality of permutations with a third parameter.

### Claim 17 (depends on 16)

The system of claim 16, wherein the operations further comprise associating at least a fourth subset of the plurality of permutations with a fourth parameter, wherein the fourth parameter is oriented orthogonally with respect to the third parameter.

### Claim 18 (depends on 17)

The system of claim 17, wherein the operations further comprise:
receiving a query regarding at least one image of the one or more images, wherein the image of the one or more images comprises one or more objects; and
wherein the generating of the prediction specific to the at least one image of the one or more images comprises identifying text that is representative of the one or more objects of the at least one image.

### Claim 19

A non-transitory computer-readable storage medium comprising programming code, which when executed by at least one data processor, causes operations comprising:
generating inputs specific to a multimodal learning based machine learning model from a training dataset comprising image data, the image data comprising a plurality of rules specific to one or more images and a plurality of characterizations representative of orientations of the one or more images, the generating comprising:
determining a plurality of permutations based on the one or more images included in the image data, and
applying orthogonal super-positioning relative to the plurality of permutations;
providing the inputs that are generated, based on the orthogonal super-positioning, into the multimodal learning based machine learning model; and
generating, by the multimodal learning based machine learning model, a prediction specific to at least one image of the one or more images.

### Claim 20 (depends on 19)

The non-transitory computer-readable storage medium of claim 19, wherein the multimodal learning based machine learning model is a contrastive language-image pre-training model.
