US 2023/0144637 A1 · MULTI-STAGE ATTENTION MODEL FOR TEXTURE SYNTHESIS

Abstract

Systems and methods for image processing are described. Embodiments of the present disclosure include an image processing apparatus configured to efficiently perform texture synthesis (e.g., increase the size of, or extend, texture in an input image while preserving a natural appearance of the synthesized texture pattern in the modified output image). In some aspects, the image processing apparatus implements an attention mechanism with a multi-stage attention model where different stages (e.g., different transformer blocks) progressively refine image feature patch mapping at different scales, while utilizing repetitive patterns in texture images to enable network generalization. One or more embodiments of the disclosure include skip connections and convolutional layers (e.g., between transformer block stages) that combine high-frequency and low-frequency features from different transformer stages and unify attention to micro-structures, meso-structures and macro-structures. In some aspects, the skip connections enable information propagation in the transformer network.

Claims20

Dependent claims

A method for image processing, comprising:

segmenting input features of an image to obtain a first input sequence of feature patches, wherein the image comprises a texture and a region to be in-painted with the texture;
transforming the first input sequence using a first attention network to obtain a first output sequence of feature patches;
computing a second input sequence of feature patches based on the first output sequence, wherein the second input sequence comprises smaller scale patches than the first input sequence;
transforming the second input sequence using a second attention network to obtain a second output sequence of feature patches;
computing a third input sequence of feature patches based on the second output sequence, wherein the third input sequence comprises same scale patches as the first input sequence;
transforming the third input sequence using a third attention network to obtain a third output sequence of feature patches;
and
generating a modified image based on the third output sequence, wherein the modified image comprises the texture within the region to be in-painted.

5 more elements

A method for training a machine learning model, the method comprising:

receiving a ground truth image comprising a texture, a region to be in-painted with the texture, and ground truth pixel information for the region to be in-painted;
segmenting features of the training image to obtain a first input sequence of feature patches;
transforming the first input sequence using a first attention network to obtain a first output sequence of feature patches;
computing a second input sequence of feature patches based on the first output sequence, wherein the second input sequence comprises smaller scale patches than the first input sequence;
transforming the second input sequence using a second attention network to obtain a second output sequence of feature patches;
computing a third input sequence of feature patches based on the second output sequence, wherein the third input sequence comprises same scale patches as the first input sequence;
transforming the third input sequence using a third attention network to obtain a third output sequence of feature patches;
generating a modified image based on the third output sequence, wherein the modified image comprises the texture within the region to be in-painted;
comparing the pixel information of the modified image within the region to be in-painted with the ground truth pixel information to obtain a structural loss;
and
updating parameters of the first attention network, the second attention network, and the third attention network based on the structural loss.

8 more elements

An apparatus for image processing, comprising:

a first attention network configured to transform a first input sequence of feature patches of an image to obtain a first output sequence of feature patches, wherein the image comprises a texture and a region to be in-painted with the texture;
a first convolution network configured to perform a first convolution operation on first combined output features to obtain second combined input features, wherein the first combined output features comprise an arrangement of the first output sequence;
a second attention network transforming a second input sequence of feature patches to obtain a second output sequence of feature patches, wherein the second input sequence is based on the second combined input features;
a second convolution network configured to perform a second convolution operation on second combined output features to obtain third combined input features, wherein the second combined output features comprise an arrangement of the second output sequence;
a third attention network configured to transform a third input sequence to obtain a third output sequence of feature patches, wherein the third input sequence is based on the third combined input features;
and
a decoder configured to generate a modified image based on the third output sequence, wherein the modified image comprises the texture within the region to be in-painted.

4 more elements

Dependencies are inferred from the claim text.

Multi-stage attention model for texture synthesis

Life of this patent

Multi-stage attention model for texture synthesis

Life of this patent

Abstract

Claims20