Super-resolution on text-to-image synthesis with GANS
AssigneeAdobe Inc.
InventorsTaesung Park, Minguk Kang, Richard Zhang, Junyan Zhu, Elya Shechtman, Sylvain Paris
Life of this patent
Estimating the term…
Abstract
Systems and methods for image processing are described. Embodiments of the present disclosure obtain a low-resolution image and a text description of the low-resolution image. A mapping network generates a style vector representing the text description of the low-resolution image. An adaptive convolution component generates an adaptive convolution filter based on the style vector. An image generation network generates a high-resolution image corresponding to the low-resolution image based on the adaptive convolution filter.
Claims19
A method comprising:
- obtaining a low-resolution image and a text description, wherein the text description indicates a target style;
- generating a text embedding representing the target style based on the text description;
- modifying an image generation network by generating an adaptive convolution filter for the low-resolution image based on the text embedding, wherein the adaptive convolution filter comprises a convolution kernel for performing a convolution on the low-resolution image;
- and
- generating, using the modified image generation network, a high-resolution image corresponding to the low-resolution image by performing a convolution process on the low-resolution image using the adaptive convolution filter, wherein the high-resolution image depicts content from the low-resolution image with fine details based on the target style from the text description.
A system comprising:
- a memory component;
- and
- a processing device coupled to the memory component, the processing device configured to perform operations comprising:
- obtaining a low-resolution image and a text description, wherein the text description indicates a target style;
- generating a text embedding representing the target style based on the text description;
- modifying an image generation network by generating an adaptive convolution filter for the low-resolution image based on the text embedding, wherein the adaptive convolution filter comprises a convolution kernel for performing a convolution on the low-resolution image;
- and
- generating, using the modified image generation network, a high-resolution image corresponding to the low-resolution image by performing a convolution process on the low-resolution image using the adaptive convolution filter, wherein the high-resolution image depicts content from the low-resolution image with fine details based on the target style from the text description.
A non-transitory computer readable medium storing code for image processing, the code comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
- obtaining a low-resolution image and a text description, wherein the text description indicates a target style;
- generating a text embedding representing the target style based on the text description;
- modifying an image generation network by generating an adaptive convolution filter for the low-resolution image based on the text embedding, wherein the adaptive convolution filter comprises a convolution kernel for performing a convolution on the low-resolution image;
- and
- generating, using the modified image generation network, a high-resolution image corresponding to the low-resolution image by performing a convolution process on the low-resolution image using the adaptive convolution filter, wherein the high-resolution image depicts content from the low-resolution image with fine details based on the target style from the text description.
Dependencies are inferred from the claim text.