Conditional axial transformer layers for high-fidelity image transformation

A computing device, comprising:

• one or more processors;
• and
• data storage, wherein the data storage has stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing device to carry out functions comprising:
• receiving an input grayscale image comprising an array of pixels;
• applying a neural network to transform the input grayscale image to a colorized high resolution version output image, wherein the applying of the neural network comprises:
• generating a low resolution coarse coloring of the input grayscale image, wherein the neural network comprises a plurality of conditional self-attention layers, wherein the generating of the low resolution coarse coloring is performed by one or more self-attention layers of an outer decoder of the neural network, and comprises applying one of:
• row-wise self-attention to a given pixel, wherein a row attention is an unmasked row attention, wherein a column attention is a masked column attention, and wherein one or more previously decoded pixels are in rows preceding a row associated with the given pixel, or
• column-wise self-attention to a given pixel, wherein a row attention is a masked row attention, wherein a column attention is an unmasked column attention, and wherein one or more previously decoded pixels are in columns preceding a column associated with the given pixel, and generating, by applying in parallel a color upsampler and a spatial upsampler to the low resolution coarse coloring, the colorized high resolution version output image, wherein each of the color upsampler and the spatial upsampler comprises layers corresponding respectively to the plurality of conditional self-attention layers;
• and providing the generated colorized high resolution version output image.

An article of manufacture comprising one or more non-transitory computer readable media having computer-readable instructions stored thereon that, when executed by one or more processors of a computing device, cause the computing device to carry out functions comprising:

• receiving an input grayscale image comprising an array of pixels;
• applying a neural network to transform the input grayscale image to a colorized high resolution version output image, wherein the applying of the neural network comprises:
• generating a low resolution coarse coloring of the input grayscale image, wherein the neural network comprises a plurality of conditional self-attention layers, wherein the generating of the low resolution coarse coloring is performed by one or more self-attention layers of an outer decoder of the neural network, and comprises applying one of:
• row-wise self-attention to a given pixel, wherein a row attention is an unmasked row attention, wherein a column attention is a masked column attention, and wherein one or more previously decoded pixels are in rows preceding a row associated with the given pixel, or
• column-wise self-attention to a given pixel, wherein a row attention is a masked row attention, wherein a column attention is an unmasked column attention, and wherein one or more previously decoded pixels are in columns preceding a column associated with the given pixel, and generating, by applying in parallel a color upsampler and a spatial upsampler to the low resolution coarse coloring, the colorized high resolution version output image, wherein each of the color upsampler and the spatial upsampler comprises layers corresponding respectively to the plurality of conditional self-attention layers;
• and providing the generated colorized high resolution version output image.