# US 2023/0068103 A1 · Image generation system and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0068103 A1
- Document ID: US20230068103A1
- Status: Published application
- Kind: A1
- Application number: 17/894,090
- Filed: 2022-08-23
- Published: 2023-03-02
- Earliest claimed: 2021-08-31
- Assignees: Alibaba Damo (Hangzhou) Technology Co., Ltd.
- Applicants: Alibaba Damo (Hangzhou) Technology Co., Ltd.
- Inventors: Huiling ZHOU; Jinbao XUE; Zhikang LI; Jie LIU; Shuai BAI; Chang ZHOU; Hongxia YANG; Jingren ZHOU
- Web page: https://patentgrep.com/patent/US20230068103A1

## CPC

- G06F 16/583: using metadata automatically derived from the content
- G06F 16/5866: {using information manually generated, e.g. tags, keywords, comments, manually generated location and time information}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06V 10/7788: {the supervisor being a human, e.g. interactive learning with a human teacher}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Embodiments of this application provide an image generation system and method. In an exemplary manufacturing industry scenario, a style requirement of a product category in a manufacturing industry is automatically captured according to user behavior data and product description information associated with the product category. Based on these data, a style description text may be generated and converted to product images by using a text prediction-based image generation model. The product images are further screened by using an image-text matching model, to obtain a product image with high quality. This process covers from style description text mining to text-to-image prediction to image quality evaluation. It provides an automation product image generation capability for the manufacturing industry, shorten a cycle of designing and producing the product image in the manufacturing industry, and improve production efficiency of the product image.

## Claims

### Claim 1

An image generation system, comprising:
a text mining component, configured to determine a specified object category of which a sample image needs to be generated in a manufacturing industry, and generate a style description text for the specified object category according to user behavior data associated with the specified object category and object description information of the specified object category, wherein the style description text reflects a style requirement of the specified object category; and
an image generation component, configured to input the style description text into a text prediction-based first image generation model for image generation, to obtain a plurality of initial object images; and input the plurality of initial object images and the style description text into a second image-text matching model for matching, to obtain at least one candidate object image of which a matching degree meets a threshold requirement,
wherein the first image generation model comprises an encoder-decoder structure implemented based on a vector quantization generative adversarial network (VQGAN) and a sparse attention mechanism.

### Claim 2 (depends on 1)

The system according to claim 1, wherein, to input the style description text into a text prediction-based first image generation model for image generation, the image generation component is further configured to:
input a text sequence corresponding to the style description text into the first image generation model, and generate a plurality of image sequences according to the text sequence based on a pre-trained codebook, wherein the codebook represents a quantization text representation of the image sequences; and
respectively perform image reconstruction on the plurality of image sequences, to obtain the plurality of initial object images.

### Claim 3 (depends on 2)

The system according to claim 2, wherein, to generate a plurality of image sequences according to the text sequence based on a pre-trained codebook, the image generation component is further configured to:
input the text sequence into an encoder of the first image generation model, and encode the text sequence, to obtain a first image feature; and
input the first image feature into a decoder of the first image generation model, and respectively decode the first image feature based on the codebook and by using the sparse attention mechanism in the decoder, to obtain the plurality of image sequences.

### Claim 4 (depends on 1)

The system according to claim 1, wherein, to input the plurality of initial object images and the style description text into a second image-text matching model for matching, the image generation component is further configured to:
input the plurality of initial object images and the style description text into the second image-text matching model, wherein the second image-text matching model is configured to respectively perform feature encoding on the plurality of initial object images and the style description text and map the plurality of initial object images and the style description text to a same semantic space, to obtain a plurality of second image features and a text feature; and
select, according to matching degrees between the plurality of second image features and the text feature, at least one initial object image of which a matching degree is greater than a threshold from the plurality of initial object images as the candidate object image.

### Claim 5 (depends on 1)

The system according to claim 1, wherein to generate a style description text for the specified object category, the text mining component is further configured to:
perform text mining on the user behavior data associated with the specified object category, to obtain an object property and a category description in which a user is interested;
perform text mining on description information of a new object that appears within a latest time period in the specified object category, to obtain an object property and a category description of the new object;
obtain category property data from a category-property-value knowledge system of the manufacturing industry according to the object property and the category description in which the user is interested and the object property and the category description of the new object, wherein the category property data comprises at least a commodity style property; and
generate the style description text according to the category property data.

### Claim 6 (depends on 1)

The system according to claim 1, wherein the manufacturing industry is a clothing industry, a printing industry, an articles-for-daily-use industry, a furniture industry, an appliance industry, or a passenger car industry.

### Claim 7 (depends on 1)

The system according to claim 1, further comprising:
a quality evaluation component, configured to display the at least one candidate object image to an evaluation system, obtain a selected target object image in response to selection of the evaluation system, and use the target object image for a subsequent manufacturing link.

### Claim 8

An image generation method, comprising:
generating, according to user behavior data associated with a specified object category and object description information of the specified object category in a manufacturing industry, a style description text for the specified object category, wherein the style description text reflects a style requirement of the specified object category;
inputting the style description text into a text prediction-based first image generation model for image generation, to obtain a plurality of initial object images, wherein the first image generation model comprises an encoder-decoder structure implemented based on a vector quantization generative adversarial network (VQGAN) and a sparse attention mechanism; and
inputting the plurality of initial object images and the style description text into a second image-text matching model for matching, to obtain at least one candidate object image of which a matching degree meets a threshold requirement.

### Claim 9 (depends on 8)

The method according to claim 8, wherein the inputting the style description text into a text prediction-based first image generation model for image generation, to obtain a plurality of initial object images comprises:
inputting a text sequence corresponding to the style description text into the first image generation model, and generating a plurality of image sequences according to the text sequence based on a pre-trained codebook, wherein the codebook represents a quantization text representation of the image sequences; and
respectively performing image reconstruction on the plurality of image sequences, to obtain the plurality of initial object images.

### Claim 10 (depends on 9)

The method according to claim 9, wherein the generating a plurality of image sequences according to the text sequence based on a pre-trained codebook comprises:
inputting the text sequence into an encoder of the first image generation model, and encoding the text sequence, to obtain a first image feature; and
inputting the first image feature into a decoder of the first image generation model, and respectively decoding the first image feature based on the codebook, to obtain the plurality of image sequences.

### Claim 11 (depends on 10)

The method according to claim 10, wherein the respectively decoding the first image feature based on the codebook, to obtain the plurality of image sequences comprises:
decoding the first image feature based on the codebook and by using a sparse attention mechanism in the decoder of the first image generation model, to obtain the plurality of image sequences.

### Claim 12 (depends on 11)

The method according to claim 11, wherein a length of each of the image sequences is greater than or equal to 4096.

### Claim 13 (depends on 10)

The method according to claim 10, wherein the respectively performing image reconstruction on the plurality of image sequences, to obtain the plurality of initial object images comprises:
inputting the plurality of image sequences into an image sequence model, and respectively performing, by a decoder of the image sequence model, image reconstruction on the plurality of image sequences, to obtain the plurality of initial object images, wherein
the image sequence model adopts an encoder-decoder structure, and the image sequence model is a neural network model obtained by training the codebook.

### Claim 14 (depends on 13)

The method according to claim 13, further comprising:
acquiring a plurality of cross-field original sample images; and
performing model training by using a vector quantization generative adversarial network and using the plurality of original sample images, to obtain the image sequence model and the codebook.

### Claim 15 (depends on 14)

The method according to claim 14, wherein the plurality of original sample images comprise a first sample group and a second sample group, and the performing model training by using a vector quantization generative adversarial network and using the plurality of original sample images, to obtain the image sequence model and the codebook comprises:
performing non-adversarial training on an initial model by using the original sample images in the first sample group, to obtain an image sequence model in an intermediate state; and
performing adversarial training on the image sequence model in the intermediate state by using the vector quantization generative adversarial network and using the original sample images in the second sample group, to obtain the image sequence model and the codebook.

### Claim 16 (depends on 8)

The method according to claim 8, wherein the inputting the plurality of initial object images and the style description text into a second image-text matching model for matching, to obtain at least one candidate object image of which a matching degree meets a threshold requirement comprises:
inputting the plurality of initial object images and the style description text into the second image-text matching model, wherein the second image-text matching model is configured to respectively perform feature encoding on the plurality of initial object images and the style description text and map the plurality of initial object images and the style description text to a same semantic space, to obtain a plurality of second image features and a text feature; and
selecting, according to matching degrees between the plurality of second image features and the text feature, at least one initial object image of which a matching degree is greater than a threshold from the plurality of initial object images as the candidate object image.

### Claim 17 (depends on 8)

The method according to claim 8, wherein the generating, according to user behavior data associated with a specified object category and object description information of the specified object category in a first manufacturing industry, a style description text for the specified object category comprises:
performing text mining on the user behavior data associated with the specified object category, to obtain an object property and a category description in which a user is interested;
performing text mining on description information of a new object that appears within a latest time period in the specified object category, to obtain an object property and a category description of the new object;
obtaining category property data from a category-property-value knowledge system of the first manufacturing industry according to the object property and the category description in which the user is interested and the object property and the category description of the new object, wherein the category property data comprises at least a commodity style property; and
generating the style description text according to the category property data.

### Claim 18 (depends on 8)

The method according to claim 8, further comprising:
displaying the at least one candidate object image to an evaluation system, obtaining a selected target object image in response to selection of the evaluation system, and using the target object image for a subsequent manufacturing link.
