# US 2025/0378604 A1 · Image text translation with style matching

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0378604 A1
- Document ID: US20250378604A1
- Status: Published application
- Kind: A1
- Application number: 18/740,478
- Filed: 2024-06-11
- Published: 2025-12-11
- Earliest claimed: 2024-06-11
- Assignees: Roblox Corporation
- Applicants: Roblox Corporation
- Inventors: Nicolas THIEBAUT; Dao LE; Xiao YU; Kyle Joseph SPENCE; Nameer HIRSCHKIND
- Web page: https://patentgrep.com/patent/US20250378604A1

## CPC

- G06T 11/60: Creating or editing images; Combining images with text
- G06F 40/51: Translation evaluation
- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G06T 11/23: using straight lines or curves
- G06V 10/82: using neural networks
- G06V 30/153: {using recognition of characters or words}
- G06V 30/19147: {Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting}
- G06V 30/1916: {Validation; Performance evaluation}
- G06T 2210/12: Bounding box
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Some implementations relate to translating text within images in a virtual environment while preserving the original style and visual characteristics of the text. In some implementations, the method includes obtaining an original image that includes text in a source language; recognizing content of the text and generating translated text in a target language; determining a text region of the text in the original image; determining a style encoding for the text; generating a masked version and noisy version of the original image; providing the noisy version, masked version, and text region as direct inputs to a pre-trained diffusion model; providing the translated text and the style encoding as conditioning inputs to the diffusion model; and obtaining an output image including the translated text, where a visual style of the translated text in the output image is the same as the visual style of the text in the original image.

## Claims

### Claim 1

A computer-implemented method comprising:
obtaining an original image that includes text in a source language;
recognizing content of the text in the original image and generating translated text comprising a translation of the content of the text, wherein the translated text is in a target language;
determining a text region of the text in the original image, wherein the text region includes a subset of pixels of the original image;
determining a style encoding for the text in the original image, wherein the style encoding is a mathematical representation of a visual style of the text in the original image;
generating a masked version of the original image by setting the subset of pixels corresponding to the text region to a fixed value;
generating a noisy version of the original image;
providing the noisy version of the original image, the masked version of the original image, and the text region as direct inputs to a pre-trained diffusion model;
providing the translated text and the style encoding as conditioning inputs to the pre-trained diffusion model; and
obtaining, as output of the diffusion model, an output image that includes the translated text, wherein a visual style of the translated text in the output image is same as the visual style of the text in the original image and wherein the output image is within a threshold visual distance of the original image.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the original image comprises an image asset from a virtual experience.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising rendering a virtual experience that includes the output image.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, further comprising determining the target language based on one of: a user profile of a user that participates in a virtual experience, or a user location of the user.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein determining the text region in the original image comprises:
identifying text pixels of the original image that correspond to the content of the text; and
generating a bounding box that includes all of the text pixels, wherein providing the text region to the pre-trained diffusion model comprises providing the bounding box.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein generating the masked version of the original image comprises replacing pixel values of pixels within the bounding box with a fixed value.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein generating the translated text comprises:
applying a translation algorithm to the content of the text to generate the translated text; and
rendering the translated text in a standard font, wherein providing the translated text to the pre-trained diffusion model comprises providing the rendered translated text in the standard font.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein the content of the text includes one or more alphanumerical symbols, and further comprising:
generating, using a text encoder, a set of vectors, wherein each vector of the set of vectors encodes a respective symbol of the one or more alphanumerical symbols, wherein providing the translated text to the pre-trained diffusion model further comprises providing the set of vectors.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein determining the style encoding for the text region of the original image comprises providing the text region of the original image as input to a style encoder, wherein the style encoder outputs the style encoding.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein the style encoder has a bottleneck architecture, wherein the text region of the original image is distilled into a small vector that prevents memorization of the input text while retaining visual attributes of the text region, wherein the visual attributes include color of the text, shape of the text, and combinations thereof.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, further comprising providing a prompt as an additional conditioning input to the pre-trained diffusion model, wherein the prompt comprises a command to write the translated text in the output image.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, wherein the conditioning inputs are provided as respective conditioning vectors to the pre-trained diffusion model, and further comprising:
computing cross-attention vectors individually; and
summing the contribution of the cross-attention vectors through residual layers of the pre-trained diffusion model.

### Claim 13 (depends on 1)

The computer-implemented method of claim 1, wherein the direct inputs are part of an encoding and decoding process of the pre-trained diffusion model, and wherein the conditioning inputs control the output image generation process of the pre-trained diffusion model.

### Claim 14 (depends on 1)

The computer-implemented method of claim 1, wherein the diffusion model is trained by:
obtaining a training set, wherein each element of the training set comprises:
a training image, wherein the training image comprises text within a text region;
a noisy version of the training image;
a masked version of the training image, wherein a subset of pixels corresponding to the text region of the training image are set to a fixed value to mask the text region;
the text region; and
a style encoding for the text in the training image, wherein the style encoding is a mathematical representation of a visual style of the text in the training image; and
training the diffusion model via self-supervised learning, wherein the training comprises, for each element in the training set:
providing the noisy version of the training image, the masked version of the training image, and the text region as direct inputs to the diffusion model;
providing original text from the text region and the style encoding as conditioning inputs to the diffusion model;
generating, by the diffusion model, an output image, by iteratively denoising the noisy version of the training image;
determining a loss value based on a comparison of the output image and the training image; and
modifying one or more parameters of the diffusion model based on the loss value.

### Claim 15

A system comprising:
one or more processors; and
memory coupled to the one or more processors storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
obtaining an original image that includes text in a source language;
recognizing content of the text in the original image and generating translated text comprising a translation of the content of the text, wherein the translated text is in a target language;
determining a text region of the text in the original image, wherein the text region includes a subset of pixels of the original image;
determining a style encoding for the text in the original image, wherein the style encoding is a mathematical representation of a visual style of the text in the original image;
generating a masked version of the original image by setting the subset of pixels corresponding to the text region to a fixed value;
generating a noisy version of the original image;
providing the noisy version of the original image, the masked version of the original image, and the text region as direct inputs to a pre-trained diffusion model;
providing the translated text and the style encoding as conditioning inputs to the pre-trained diffusion model; and
obtaining, as output of the diffusion model, an output image that includes the translated text, wherein a visual style of the translated text in the output image is same as the visual style of the text in the original image and wherein the output image is within a threshold visual distance of the original image.

### Claim 16 (depends on 15)

The system of claim 15, wherein the original image comprises an image asset from a virtual experience.

### Claim 17 (depends on 15)

The system of claim 15, wherein the instructions cause the system to perform an operation comprising rendering a virtual experience that includes the output image.

### Claim 18 (depends on 15)

The system of claim 15, wherein the instructions cause the system to perform an operation comprising determining the target language based on one of: a user profile of a user that participates in a virtual experience, or a user location of the user.

### Claim 19 (depends on 15)

The system of claim 15, wherein determining the text region in the original image comprises:
identifying text pixels of the original image that correspond to the content of the text; and
generating a bounding box that includes all of the text pixels, wherein providing the text region to the pre-trained diffusion model comprises providing the bounding box.

### Claim 20

A non-transitory computer-readable medium containing instructions comprising:
obtaining an original image that includes text in a source language;
recognizing content of the text in the original image and generating translated text comprising a translation of the content of the text, wherein the translated text is in a target language;
determining a text region of the text in the original image, wherein the text region includes a subset of pixels of the original image;
determining a style encoding for the text in the original image, wherein the style encoding is a mathematical representation of a visual style of the text in the original image;
generating a masked version of the original image by setting the subset of pixels corresponding to the text region to a fixed value;
generating a noisy version of the original image;
providing the noisy version of the original image, the masked version of the original image, and the text region as direct inputs to a pre-trained diffusion model;
providing the translated text and the style encoding as conditioning inputs to the pre-trained diffusion model; and
obtaining, as output of the diffusion model, an output image that includes the translated text, wherein a visual style of the translated text in the output image is same as the visual style of the text in the original image and wherein the output image is within a threshold visual distance of the original image.
