# US 2023/0401825 A1 · Method and System for Multi-Scale Vision Transformer Architecture

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0401825 A1
- Document ID: US20230401825A1
- Status: Published application
- Kind: A1
- Application number: 17/853,739
- Filed: 2022-06-29
- Published: 2023-12-14
- Earliest claimed: 2022-06-14
- Applicants: NavInfo Europe B.V.
- Inventors: Ibrahim Batuhan Akkaya; Senthilkumar Sockalingam Kathiresan; Elahe Arani; Bahram Zonooz
- Web page: https://patentgrep.com/patent/US20230401825A1

## CPC

- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06N 3/09: Supervised learning
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A computer-implemented method for processing images in deep neural networks by: breaking an input sample into a plurality of non-overlapping patches; converting said patches into a plurality of patch-tokens; processing said patch-tokens in at least one transformer block comprising a multi-head self-attention block; providing a multi-scale feature module block in the at least one transformer block; using said multi-scale feature module block for extracting features corresponding to a plurality of scales by applying a plurality of kernels having different window sizes; concatenating said features in the multi-scale feature module block; providing a plurality of hierarchically arranged convolution layers in the multi-scale feature module block; and processing said features in said hierarchically arranged convolution layers for generating at least three multiscale tokens containing multiscale information.

## Claims

### Claim 1

A computer-implemented method for image processing in a deep neural network comprising the steps of:
breaking an input sample into a plurality of non-overlapping patches;
converting said patches into a plurality of patch-tokens; and
processing said patch-tokens in at least one transformer block;
wherein the method further comprises the steps of:
providing a multi-scale feature module block in the at least one transformer block;
using said multi-scale feature module block for extracting features corresponding to a plurality of scales by applying a plurality of kernels having different window sizes;
concatenating said features in the multi-scale feature module block;
providing a plurality of hierarchically arranged convolution layers in the multi-scale feature module block; and
processing said features in said hierarchically arranged convolution layers for generating at least three multiscale-tokens comprising multiscale information.

### Claim 2 (depends on 1)

The computer-implemented method according to claim 1 further comprising the steps of:
providing a multi-headed self-attention block in the at least one transformer block; and
feeding the at least three multiscale tokens as query, key, and value into the multi-head self-attention block.

### Claim 3 (depends on 1)

The computer-implemented method according to claim 1 further comprising the steps of:
arranging the patch-tokens in an image format; and
processing said arranged patch-tokens in a first convolutional layer of the multi-scale feature module.

### Claim 4 (depends on 1)

The computer-implemented method according to claim 1 further comprising the step of processing a classification token along with the plurality of patch-tokens in the hierarchical convolutional layers of the multi-scale feature module block using a depth-wise separable convolution comprising a depth-wise convolution followed by a pointwise convolution, wherein the classification token and the plurality of patch-tokens are concatenated before the pointwise convolution layers, and wherein the classification token and the plurality of patch-tokens are separated before the depth-wise convolution layers.

### Claim 5 (depends on 1)

The computer-implemented method according to claim 1 further comprising the step of rearranging and/or regrouping outputs of the hierarchical convolutional layers for providing the at least three multiscale tokens.

### Claim 6 (depends on 2)

The computer-implemented method according to claim 2 further comprising the step of providing a multi-layer perceptron block in the at least one transformer block for processing outputs of the multi-head self-attention block.

### Claim 7 (depends on 6)

The computer-implemented method according to claim 6 further comprising the step of applying residual connections after the multi-head self-attention and after multi-layer perceptron blocks.

### Claim 8 (depends on 4)

The computer-implemented method according to claim 4 further comprising the step of using a classification head for the classification token to category space for making a prediction.

### Claim 9 (depends on 1)

A computer-readable medium provided with a computer program, wherein when said computer program is loaded and executed by a computer, said computer program causes the computer to carry out the steps of the computer-implemented method according to claim 1.

### Claim 10 (depends on 1)

A data processing system comprising a computer loaded with a computer program, wherein said program is arranged for causing the computer to carry out the steps of the computer-implemented method according to claim 1.
