Image text translation with style matching

A system comprising:

• one or more processors;
• and
• memory coupled to the one or more processors storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
• obtaining an original image that includes text in a source language;
• recognizing content of the text in the original image and generating translated text comprising a translation of the content of the text, wherein the translated text is in a target language;
• determining a text region of the text in the original image, wherein the text region includes a subset of pixels of the original image;
• determining a style encoding for the text in the original image, wherein the style encoding is a mathematical representation of a visual style of the text in the original image;
• generating a masked version of the original image by setting the subset of pixels corresponding to the text region to a fixed value;
• generating a noisy version of the original image;
• providing the noisy version of the original image, the masked version of the original image, and the text region as direct inputs to a pre-trained diffusion model as part of an encoding and decoding process of the pre-trained diffusion model;
• providing the translated text and the style encoding as conditioning inputs to the pre-trained diffusion model, wherein the translated text and the style encoding are each projected via respective parameterized linear transformations that are trained during a training phase of the diffusion model;
• and
• obtaining, as output of the diffusion model, an output image that includes the translated text, wherein a visual style of the translated text in the output image is same as the visual style of the text in the original image, wherein the output image is within a threshold visual distance of the original image, and wherein query vectors are generated from intermediate representations of the diffusion model during iterative denoising, cross-attention vectors are computed individually for the translated text and the style encoding, and resulting attention outputs are incorporated through residual connections during the iterative denoising.

A non-transitory computer-readable medium containing instructions comprising:

• obtaining an original image that includes text in a source language;
• recognizing content of the text in the original image and generating translated text comprising a translation of the content of the text, wherein the translated text is in a target language;
• determining a text region of the text in the original image, wherein the text region includes a subset of pixels of the original image;
• determining a style encoding for the text in the original image, wherein the style encoding is a mathematical representation of a visual style of the text in the original image;
• generating a masked version of the original image by setting the subset of pixels corresponding to the text region to a fixed value;
• generating a noisy version of the original image;
• providing the noisy version of the original image, the masked version of the original image, and the text region as direct inputs to a pre-trained diffusion model as part of an encoding and decoding process;
• providing the translated text and the style encoding as conditioning inputs to the pre-trained diffusion model, wherein each conditioning input is projected via a respective parameterized linear transformation that is trained during a training phase of the diffusion model;
• and
• obtaining, as output of the diffusion model, an output image that includes the translated text, wherein a visual style of the translated text in the output image is same as the visual style of the text in the original image, wherein the output image is within a threshold visual distance of the original image, and wherein cross-attention vectors are computed individually using query vectors derived from intermediate representations of the diffusion model during iterative denoising and are incorporated through residual layers.