# US 2024/0203098 A1 · System and method for self-distilled vision transformer for domain generalization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0203098 A1
- Document ID: US20240203098A1
- Status: Published application
- Kind: A1
- Application number: 18/084,152
- Filed: 2022-12-19
- Published: 2024-06-20
- Earliest claimed: 2022-12-19
- Assignees: Mohamed bin Zayed University of Artificial Intelligence
- Applicants: Mohamed bin Zayed University of Artificial Intelligence
- Inventors: Maryam SULTANA; Muhammad Muzammal NASEER; Muhammad Haris KHAN; Salman KHAN; Fahad Shahbaz KHAN
- Web page: https://patentgrep.com/patent/US20240203098A1

## CPC

- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/776: Validation; Performance evaluation
- G06V 10/82: using neural networks
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An apparatus and method for a machine learning engine for domain generalization which trains a vision transformer neural network using a training dataset including at least two domains for diagnosis of a medical condition. Image patches and class tokens are processed through a sequence of feature extraction transformer blocks to obtain a predicted class token. In parallel, intermediate class tokens are extracted as outputs of each of the feature extraction transformer blocks, where each transformer block is a sub-model. One sub-model is randomly sampled from the sub-models to obtain a sampled intermediate class token. The intermediate class token is used to make a sub-model prediction. The vision transformer neural network is optimized based on a difference between the predicted class token and the sub-model prediction. Inferencing is performed for a target medical image in a target domain that is different from the at least two domains.

## Claims

### Claim 1

A method of processing in a vision transformer neural network for classification of a target medical image, the vision transformer neural network having a sequence of feature extraction transformer blocks and a classifier component, the method comprising:
a training cycle of
providing a training dataset of medical images obtained from at least two domains for a medical diagnosis:
receiving, in processing circuitry, a plurality of image patches and initial class tokens obtained from a medical image of the training dataset:
processing, by the processing circuitry, the image patches and the initial class tokens through the sequence of feature extraction transformer blocks to obtain a predicted class token:
extracting, by the processing circuitry, in parallel with the processing through the transformer blocks, intermediate class tokens as outputs of each of the feature extraction transformer blocks, where each transformer block is a sub-model:
randomly sampling, by the processing circuitry, one sub-model from the sub-models to obtain a sampled intermediate class token:
making a sub-model prediction, by the processing circuitry, by passing the sampled intermediate class token through the classifier component: and
determining, by the processing circuitry, a difference between the predicted class token and the sub-model prediction and optimizing the vision transformer neural network based on the difference: and
inferencing by
receiving, in the processing circuitry, a plurality of target image patches for the target medical image in a target domain that is different from the at least two domains; and
predicting, by the processing circuit, a class of the target medical image using the optimized vision transformer neural network.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
determining, using the processing circuitry, a prediction error from the predicted class token using a cross-entropy loss.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
the step of determining the difference includes determining the difference between the predicted class token and the randomly sampled intermediate class token to determine a Kullback-Leibler divergence (KL) loss; and
minimizing an overall loss based on the cross-entropy loss and the KL loss.

### Claim 4 (depends on 1)

The method of claim 1, wherein the at least two domains are multiple source domains.

### Claim 5 (depends on 4)

The method of claim 4, wherein the source domains are different CT devices.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
a token-to-token module which reduces progressively a length of each of the plurality of image patches to obtain final reduced image patches,
wherein a first of the transformer blocks receives the final reduced image patches.

### Claim 7 (depends on 1)

The method of claim 1, wherein the receiving the plurality of image patches and the initial class tokens includes receiving a distillation token, and
wherein the image patches, initial class tokens and distillation token are processed by the sequence of feature extraction transformer blocks to obtain the predicted class token and a predicted distillation token.

### Claim 8 (depends on 1)

The method of claim 1, wherein dimensions of the intermediate class tokens of each of the feature extraction transformer blocks are the same.

### Claim 9

An apparatus for classification of medical images obtained from a plurality of medical imaging source devices for diagnosis of a predetermined medical condition, the apparatus comprising:
an input for receiving one at a time a training medical image of the medical images:
processing circuitry configured to train a vision transformer neural network by
splitting the medical image into a plurality of image patches and receiving initial class tokens,
processing the image patches and the initial class tokens through a sequence of feature extraction transformer blocks to obtain a predicted class token,
extracting, in parallel with the processing through the transformer blocks, intermediate class tokens as outputs of each of the feature extraction transformer blocks, where each transformer block is a sub-model,
randomly sampling one sub-model from the sub-models to obtain a sampled intermediate class token,
making a sub-model prediction by passing the sampled intermediate class token through a classifier component, and
determining a difference between the predicted class token and the sub-model prediction and optimizing the vision transformer neural network based on the difference; and
the processing circuitry configured for inferencing by
receiving a plurality of target image patches for a target medical image received from a medical imaging source device that is different from the source devices for the training medical images; and
predicting a class of the target medical image using the optimized vision transformer neural network.

### Claim 10 (depends on 9)

The apparatus of claim 9, wherein the plurality of medical imaging source devices are different CT devices.

### Claim 11 (depends on 9)

The apparatus of claim 9, wherein the plurality of medical imaging source devices are different MRI devices.

### Claim 12 (depends on 9)

The apparatus of claim 9, the processing circuitry further comprising:
a token-to-token module which reduces progressively a length of each of the plurality of image patches to obtain final reduced image patches,
wherein a first of the transformer blocks receives the final reduced image patches.

### Claim 13 (depends on 9)

The apparatus of claim 9, wherein the splitting into the plurality of image patches and receiving the initial class tokens includes receiving a distillation token, and
wherein the processing circuitry processes the image patches, initial class tokens and distillation token by the sequence of feature extraction transformer blocks to obtain the predicted class token and a predicted distillation token.

### Claim 14 (depends on 9)

The apparatus of claim 9, wherein dimensions of the intermediate class tokens output by each of the feature extraction transformer blocks are the same.

### Claim 15

A non-transitory computer readable storage medium storing program instructions, which when processed by a machine learning engine perform a method comprising:
a training cycle for a vision transformer neural network of
providing a training dataset of medical images including at least two domains for a medical diagnosis:
receiving a plurality of image patches and initial class tokens obtained from a medical image of the training dataset:
processing the image patches and the initial class tokens through a sequence of feature extraction transformer blocks to obtain a predicted class token:
extracting, in parallel with the processing through the transformer blocks, intermediate class tokens as outputs of each of the feature extraction transformer blocks, where each transformer block is a sub-model:
randomly sampling one sub-model from the sub-models to obtain a sampled intermediate class token:
making a sub-model prediction by passing the sampled intermediate class token through a classifier component; and
determining a difference between the predicted class token and the sub-model prediction and optimizing the vision transformer neural network based on the difference; and
inferencing by
receiving a plurality of target image patches for a target medical image in a target domain that is different from the at least two domains; and
predicting a class of the target medical image using the optimized vision transformer neural network.

### Claim 16 (depends on 15)

The computer readable storage medium of claim 15, the method further comprising:
determining a prediction error from the predicted class token using a cross-entropy loss;
the step of determining the difference includes determining the difference between the predicted class token and the randomly sampled intermediate class token to determine a Kullback-Leibler divergence (KL) loss; and
minimizing an overall loss based on the cross-entropy loss and the KL loss.

### Claim 17 (depends on 15)

The computer readable storage medium of claim 15, further comprising:
a token-to-token module which reduces progressively a length of each of the plurality of image patches to obtain final reduced image patches,
wherein a first of the transformer blocks receives the final reduced image patches.

### Claim 18 (depends on 15)

The computer readable storage medium of claim 15, wherein the receiving the plurality of image patches and the initial class tokens includes receiving a distillation token, and
wherein the image patches, initial class tokens and distillation token are processed by the sequence of feature extraction transformer blocks to obtain the predicted class token and a predicted distillation token.

### Claim 19 (depends on 15)

The computer readable storage medium of claim 15, wherein dimensions of the intermediate class tokens of each of the feature extraction transformer blocks are the same.

### Claim 20 (depends on 15)

The computer readable storage medium of claim 15, wherein at least two domains for the medical image and the target domain are different medical imaging source devices of a same type of medical device.
