US 2022/0067941 A1Published application

Multimodal self-supervised learning for image analysis

AssigneeSap SE

InventorsAiham Taleb, Moin Nabi, Tassilo Klein

Life of this patent

Estimating the term…

Loading patent…

Abstract

A machine learning model may be trained on a first task of puzzle solving before being tuned on a second task of image analysis. The training of the machine learning model may be self-supervised whereas the tuning of the machine learning model may be supervised. The training data may include a puzzle generated to include multiple imaging modalities. The puzzle may be generated by shuffling a position of the pieces forming an original image. The machine learning model may be trained to perform the first task by reassembling the pieces in the puzzle to generate a reconstruction of the original image. Upon being trained to perform the first task and tuned to perform the second task, the machine learning model may be deployed to perform the second task. The second task may be an image segmentation task such as tumor segmentation and a regression task such as survival prediction.

Claims20

Dependent claims

A system, comprising:

at least one data processor;
and
at least one memory storing instructions which, when executed by the at least one data processor, result in operations comprising:
training, based at least on a first training data, a machine learning model to perform a first task of puzzle solving, the first training data including a puzzle that is generated to include a first piece comprising a first portion of a first image having a first imaging modality and a second piece comprising a second portion of a second image having a second imaging modality, the puzzle being generated by at least shuffling a respective position of the first piece and the second piece in a third image including the first piece and the second piece, and the machine learning model being trained to generate a reconstruction of the third image by at least reassembling the first piece and the second piece in the puzzle;
tuning, based at least on a second training data, the machine learning model trained to perform the first task, the machine learning model being tuned to perform a second task of image analysis;
and
performing the second task by at least applying the machine learning model trained on the first task and tuned on the second task.

4 more elements

Dependencies are inferred from the claim text.