# US 2022/0277136 A1 · Template-based redesign of a document based on document content

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0277136 A1
- Document ID: US20220277136A1
- Status: Published application
- Kind: A1
- Application number: 17/188,302
- Filed: 2021-03-01
- Published: 2022-09-01
- Earliest claimed: 2021-03-01
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Sumit Shekhar; Vedant Raval; Tripti Shukla; Simarpreet singh Saluja; Paridhi Maheshwari; Divyam Gupta
- Web page: https://patentgrep.com/patent/US20220277136A1

## CPC

- G06F 40/186: Templates
- G06F 18/211: Selection of the most significant subset of features
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 40/106: Display of layout of documents; Previewing
- G06N 3/0475: Generative networks
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06V 10/82: using neural networks
- G06N 3/045: Combinations of networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/08: Learning methods
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Certain embodiments involve a template-based redesign of documents based on the contents of documents. For instance, a computing system selects a template for modifying an input document. To do so, the computing system uses a generative adversarial network to generate an interpolated layout image from an input layout image, which represents the input document, and a template layout image, which represents the selected template. The computing system matches the input element to an interpolated element from the interpolated layout image. The computing system generates an output document by, for example, modifying a layout of the input document to match the interpolated layout image, such as by fitting the input element into a shape of the interpolated element.

## Claims

### Claim 1

A method in which one or more devices perform operations comprising:
generating an input layout image representing an input document, wherein the input layout image comprises a first shape representing an input element of the input document, and wherein the first shape is associated with a first tag describing the input element;
selecting a template for modifying the input document;
accessing a template layout image representing the template, wherein the template layout image comprises a second shape representing a template element of the template, and wherein the second shape is associated with a second tag describing the template element;
applying a generative adversarial network (GAN) to predict an interpolated layout image from the input layout image and the template layout image, wherein the interpolated layout image comprises a third shape associated with a third tag describing an interpolated element;
matching the input element to the third shape based on a similarity between the third shape and the first shape and between the third tag and the first tag associated with the input element; and
generating an output document by modifying a layout of the input document to match the interpolated layout image, wherein modifying the layout comprises fitting the input element into the third shape of the interpolated layout image, based on matching the input element to the third shape.

### Claim 2 (depends on 1)

The method of claim 1, wherein the third tag is a color corresponding to an element type, and wherein matching the input element to the third shape based on a similarity between the third shape and the first shape and between the third tag and the first tag associated with the input element comprises matching an input element type of the input element to the element type corresponding to the color.

### Claim 3 (depends on 1)

The method of claim 1, the operations further comprising training the GAN to predict a layout image that is interpolated from a first layout image and a second layout image, wherein training the GAN comprises:
accessing training tuples, wherein each training tuple of the training tuples comprises a respective first layout image representing a respective first document, a respective second layout image representing a respective second document, and a respective direction of interpolation, and wherein each training tuple is associated with a respective interpolated layout image; and
training the GAN using the training tuples,
wherein, for each training tuple, the respective first layout image, the respective second layout image, and the respective direction of interpolation are used as training inputs, and the respective interpolated layout image is used as an expected output.

### Claim 4 (depends on 1)

The method of claim 1, the operations further comprising:
predicting a direction of interpolation from the input layout image and the template layout image,
wherein the GAN utilizes the direction of interpolation in computing the interpolated layout image from the input layout image and the template layout image.

### Claim 5 (depends on 1)

The method of claim 1, wherein selecting the template comprises:
generating a similarity graph comprising (a) input nodes representing input elements of the input document, (b) template nodes representing template elements of a template, and (b) edges connecting the input nodes to the templates nodes, wherein an edge between an input node representing the input element and a template node representing the template element has a similarity weight indicating a similarity between the input element and the template element;
computing a score for the template based on the similarity graph; and
selecting the template based on the score.

### Claim 6 (depends on 5)

The method of claim 5, wherein the similarity weight of the edge between the input node and the template node is based on a similarity between the first shape and the second shape.

### Claim 7 (depends on 6)

The method of claim 6, wherein the similarity weight of the edge between the input node and the template node is further based on a similarity between the first tag associated with the input element and the second tag associated with the template element.

### Claim 8 (depends on 7)

The method of claim 7, wherein generating the input layout image comprises:
adjusting the first shape for the input element to avoid overlapping of the first shape with another shape corresponding to another input element of the input document;
adjusting a font size of the input element to fit within the first shape as adjusted; and
updating the font size in the first tag.

### Claim 9 (depends on 5)

The method of claim 5, wherein computing the score for the template comprises:
determining a first count of input elements in the input document and a second count of template elements in the template;
computing a first content area size comprising area occupied by the input elements and a second content area size comprising area occupied by the template elements;
determining respective types of the input elements and the template elements; and
computing the score as a function of the first count, the second count, the first content area size, the second content area size, and the respective types of the input elements and the template elements.

### Claim 10

A system for redesigning documents, the system comprising processing hardware configured to perform operations comprising:
a step for generating an input layout image representing an input document, wherein the input layout image comprises a first shape representing an input element of the input document, and wherein the first shape is associated with a first tag describing the input element;
selecting a template for modifying the input document;
accessing a template layout image representing the template, wherein the template layout image comprises a second shape representing a template element of the template, and wherein the second shape is associated with a second tag describing the template element;
a step for applying a generative adversarial network (GAN) to predict an interpolated layout image from the input layout image and the template layout image, wherein the interpolated layout image comprises a third shape associated with a third tag describing an interpolated element;
matching the input element to the third shape based on a similarity between the third shape and the first shape and between the third tag and the first tag associated with the input element; and
a step for generating an output document by modifying a layout of the input document to match the interpolated layout image.

### Claim 11 (depends on 10)

The system of claim 10, wherein selecting the template comprises:
generating a similarity graph comprising (a) input nodes representing input elements of the input document, (b) template nodes representing template elements of a template, and (b) edges connecting the input nodes to the templates nodes, wherein an edge between an input node representing the input element and a template node representing the template element has a similarity weight indicating a similarity between the input element and the template element;
computing a score for the template based on the similarity graph; and
selecting the template based on the score.

### Claim 12 (depends on 11)

The system of claim 11, wherein the similarity weight of the edge between the input node and the template node is based on a similarity between the first shape and the second shape and on a similarity between the first tag associated with the input element and the second tag associated with the template element.

### Claim 13 (depends on 11)

The system of claim 11, wherein computing the score for the template comprises:
determining a first count of input elements in the input document and a second count of template elements in the template;
computing a first content area size comprising area occupied by the input elements and a second content area size comprising area occupied by the template elements;
determining respective types of the input elements and the template elements; and
computing the score as a function of the first count, the second count, the first content area size, the second content area size, and the respective types of the input elements and the template elements.

### Claim 14

A non-transitory computer-readable medium embodying program code for redesigning documents, the program code comprising instructions that, when executed by a processor, cause the processor to perform operations comprising:
selecting a template for modifying an input document, wherein selecting the template comprises:
generating a similarity graph comprising (a) input nodes representing input elements of the input document, (b) template nodes representing template elements of a template, and (b) edges connecting the input nodes to the templates nodes, wherein an edge between an input node representing an input element and a template node representing the template element has a similarity weight indicating a similarity between the input element and the template element;
computing a score for the template based on the similarity graph; and
selecting the template based on the score;
determining an interpolated layout representing an interpolation between the input document and the template; and
generating an output document by modifying an input element of the input document to match an interpolated element represented in the interpolated layout.

### Claim 15 (depends on 14)

The non-transitory computer-readable medium of claim 14, the operations further comprising computing the similarity weight of the edge between the input node and the template node, wherein computing the similarity weight comprises:
accessing a first similarity category associated with a first score and a second similarity category associated with a second score, wherein the first similarity category and the second similarity category are associated with element pairs;
classifying, as belonging to the first similarity category, a similarity type applicable to the input element and the template element; and
assigning, to the edge, the first score associated with the first similarity category based on the similarity type applicable to the input element and the template element belonging to the first similarity category.

### Claim 16 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein computing the score for the template comprises:
determining a first count of input elements in the input document and a second count of template elements in the template;
computing a first content area size comprising area occupied by the input elements and a second content area size comprising area occupied by the template elements;
determining respective types of the input elements and the template elements; and
computing the score as a function of the first count, the second count, the first content area size, the second content area size, and the respective types of the input elements and the template elements.

### Claim 17 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein determining the interpolated layout representing the interpolation between the input document and the template comprises:
generating an input layout image representing the input document, wherein the input layout image comprises a first shape representing the input element of the input document, and wherein the first shape is associated with a first tag describing the input element;
accessing a template layout image representing the template, wherein the template layout image comprises a second shape representing a template element of the template, and wherein the second shape is associated with a second tag describing the template element; and
applying a generative adversarial network (GAN) to predict an interpolated layout image from the input layout image and the template layout image.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein the similarity weight of the edge between the input node and the template node is based on a similarity between the first shape and the second shape.

### Claim 19 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein the similarity weight of the edge between the input node and the template node is further based on a similarity between the first tag associated with the input element and the second tag associated with the template element.

### Claim 20 (depends on 17)

The non-transitory computer-readable medium of claim 17, the operations further comprising training the GAN to predict a layout image that is interpolated from a first layout image and a second layout image, wherein training the GAN comprises:
accessing training tuples, wherein each training tuple of the training tuples comprises a respective first layout image representing a respective first document, a respective second layout image representing a respective second document, and a respective direction of interpolation, and wherein each training tuple is associated with a respective interpolated layout image; and
training the GAN using the training tuples,
wherein, for each training tuple, the respective first layout image, the respective second layout image, and the respective direction of interpolation are used as training inputs, and the respective interpolated layout image is used as an expected output.
