# US 12,217,382 B2 · Multi-scale transformer for image analysis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,382 B2
- Document ID: US12217382B2
- Status: Granted
- Kind: B2
- Application number: 18/527,528
- Filed: 2023-12-04
- Published: 2024-04-11
- Granted: 2025-02-04
- Earliest claimed: 2023-12-04
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Junjie Ke; Feng Yang; Qifei Wang; Yilin Wang; Peyman Milanfar
- Web page: https://patentgrep.com/patent/US12217382B2

## CPC

- G06T 3/04: Context-preserving transformations, e.g. by using an importance map (panospheric to cylindrical image transformations G06T3/12)
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06T 2207/10004: Still image; Photographic image
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30168: Image quality inspection
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The technology employs a patch-based multi-scale Transformer (300) that is usable with various imaging applications. This avoids constraints on image fixed input size and predicts the quality effectively on a native resolution image. A native resolution image (304) is transformed into a multi-scale representation (302), enabling the Transformer's self-attention mechanism to capture information on both fine-grained detailed patches and coarse-grained global patches. Spatial embedding (316) is employed to map patch positions to a fixed grid, in which patch locations at each scale are hashed to the same grid. A separate scale embedding (318) is employed to distinguish patches coming from different scales in the multiscale representation. Self-attention (508) is performed to create a final image representation. In some instances, prior to performing self-attention, the system may prepend a learnable classification token (322) to the set of input tokens.

## Claims

### Claim 1

A method for processing imagery, the method comprising:
encoding, by one or more processors for a set of images including a native resolution image and one or more resized variants of the native resolution image, a corresponding spatial embedding for each patch associated with a respective region of either the native resolution image or a given one of the one or more resized variants, to form a set of spatially encoded patches;
applying, by one or more processors, a set of scale embeddings to the set of spatially encoded patches to capture scale information associated with the native resolution image and the given one of the one or more resized variants to form a set of input tokens; and
creating, by one or more processors a final image representation via a self-attention process on the set of input tokens.

### Claim 2 (depends on 1)

The method of claim 1, wherein the final image representation corresponds to a predicted quality score of the native resolution image.

### Claim 3 (depends on 1)

The method of claim 1, further comprising constructing a multi-scale representation of the native resolution image, the multi-scale representation including the native resolution image and the one or more resized variants.

### Claim 4 (depends on 3)

The method of claim 3, wherein the one or more resized variants form a set of aspect ratio preserving resized variants.

### Claim 5 (depends on 3)

The method of claim 3, wherein constructing the multi-scale representation includes splitting the native resolution image and each resized variant into fixed-size patches, wherein each patch represents a region of either the native resolution image or one of the resized variants.

### Claim 6 (depends on 3)

The method of claim 3, wherein the multi-scale representation includes the native resolution image with height H, width W and one or more channels C, and one or more resized variants are derived using a Gaussian kernel, wherein each channel represents a color component of the native resolution image.

### Claim 7 (depends on 1)

The method of claim 1, wherein encoding the corresponding spatial embedding includes hashing a patch position for each patch within a grid of learnable embeddings.

### Claim 8 (depends on 1)

The method of claim 1, wherein each resized variant is formed so that an aspect ratio of that variant is sized according to a longer side of the native resolution image.

### Claim 9 (depends on 1)

The method of claim 1, further comprising aligning the set of spatially encoded patches across scales, by mapping patch locations from all scales to a grid.

### Claim 10 (depends on 1)

The method of claim 1, further comprising, prior to the self-attention process, prepending a learnable classification token to the set of input tokens.

### Claim 11 (depends on 1)

The method of claim 1, wherein the self-attention process includes performing multi-head self-attention on the set of input tokens.

### Claim 12 (depends on 11)

The method of claim 11, wherein performing the multi-head self-attention comprises:
performing a first normalization on the set of input tokens to generate a set of first normalizations;
performing a multi-head self-attention operation on the set of first normalizations;
performing a second normalization on output of the multi-head self-attention operation to generate a second normalization; and
performing a multi-layer perceptron process on the second normalization to create a final image representation.

### Claim 13 (depends on 12)

The method of claim 12, wherein the second normalization is performed on the output of the multi-head self-attention operation added to a residual connection from the set of input tokens.

### Claim 14 (depends on 13)

The method of claim 13, wherein the final image representation is obtained by adding input from the second normalization to output from the multi-layer perceptron process.

### Claim 15 (depends on 1)

The method of claim 1, further comprising pretraining a transformer on a selected image dataset.

### Claim 16 (depends on 1)

The method of claim 1, wherein a patch size is selected based on an average resolution across the native resolution image and the one or more resized variants.

### Claim 17

An image processing system, comprising:
memory configured to store imagery; and
one or more processors operatively coupled to the memory, the one or more processors being configured to
encode, for a set of images including a native resolution image and one or more resized variants of the native resolution image, a corresponding spatial embedding for each patch associated with a respective region of either the native resolution image or a given one of the one or more resized variants, to form a set of spatially encoded patches;
apply a set of scale embeddings to the set of spatially encoded patches to capture scale information associated with the native resolution image and the given one of the one or more resized variants, to form a set of input tokens; and
create a final image representation via self-attention on the set of input tokens.

### Claim 18 (depends on 17)

The image processing system of claim 17, wherein the one or more processors are further configured to store in the memory at least one of the final image representation, the native resolution image, or the one or more resized variants.

### Claim 19 (depends on 17)

The image processing system of claim 17, wherein the final image representation corresponds to a predicted quality score of the native resolution image.

### Claim 20 (depends on 17)

The image processing system of claim 17, wherein the one or more resized variants form a set of aspect ratio preserving resized variants.

### Claim 21 (depends on 17)

The image processing system of claim 17, wherein the one or more processors are further configured, prior to the self-attention, to prepend a learnable classification token to the set of input tokens.

### Claim 22 (depends on 17)

The image processing system of claim 17, wherein each resized variant is formed so that an aspect ratio of that variant is sized according to a longer side of the native resolution image.