# US 2019/0317739 A1 · Methods and apparatus to automatically generate code for graphical user interfaces

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0317739 A1
- Document ID: US20190317739A1
- Status: Published application
- Kind: A1
- Application number: 16/455,216
- Filed: 2019-06-27
- Published: 2019-10-17
- Earliest claimed: 2019-06-27
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Javier Sebastian Turek; Javier Felip Leon; Luis Carlos Maria Remis; David Israel Gonzalez Aguirre; Ignacio Javier Alvarez; Justin Gottschlich
- Web page: https://patentgrep.com/patent/US20190317739A1

## CPC

- G06F 40/143: Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]
- G06F 8/10: Requirements analysis; Specification techniques
- G06F 8/20: Software design
- G06F 8/38: for implementing user interfaces
- G06F 40/00: Handling natural language data (speech analysis or synthesis, speech recognition G10L)
- G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30)
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06T 7/168: involving transform domain methods
- G06F 9/451: Execution arrangements for user interfaces
- G06N 5/04: Inference or reasoning models
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Methods and apparatus to automatically generate code for graphical user interfaces are disclosed. An example apparatus includes a textual description analyzer to encode a user-provided textual description of a GUI design using a first neural network. The example apparatus further includes a DSL statement generator to generate a DSL statement with a second neural network. The DSL statement is to define a visual element of the GUI design. The DSL statement is generated based on at least one of the encoded textual description or a user-provided image representative of the GUI design. The example apparatus further includes a rendering tool to render a mockup of the GUI design based on the DSL statement.

## Claims

### Claim 1

An apparatus comprising:
a textual description analyzer to encode a user-provided textual description of a GUI design with a first neural network;
a DSL statement generator to generate a DSL statement with a second neural network, the DSL statement to define a visual element of the GUI design, the DSL statement generated based on at least one of the encoded textual description or a user-provided image representative of the GUI design; and
a rendering tool to render a mockup of the GUI design based on the DSL statement.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the user-provided image is a hand drawn sketch.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein the first neural network is a recurrent neural network.

### Claim 4 (depends on 1)

The apparatus of claim 1, further including an image analyzer to determine a difference between the user-provided image and a rendered image with a third neural network, the rendered image corresponding to an approximation of the GUI design, the DSL statement generator to generate the DSL statement based on the difference.

### Claim 5 (depends on 4)

The apparatus of claim 4, wherein the third neural network is a convolutional neural network.

### Claim 6 (depends on 4)

The apparatus of claim 4, wherein the image analyzer is to generate an encoded representation of the difference with a fourth neural network, the DSL statement generator to generate the DSL statement based on the encoded representation of the difference.

### Claim 7 (depends on 6)

The apparatus of claim 6, wherein the fourth neural network is a multilayer perceptron neural network.

### Claim 8 (depends on 4)

The apparatus of claim 4, wherein the difference are first difference is a first difference, the DSL statement is a first DSL statement, and the rendered image is a first rendered image, the rendering tool to generate a second rendered image based on the DSL statement, the image analyzer to determine a second difference between the user-provided image and the second rendered image with the third neural network, the DSL statement generator to generate a second DSL statement based on the second difference.

### Claim 9 (depends on 1)

The apparatus of claim 1, wherein the DSL statement generator is to generate a DSL command with the second neural network, the DSL command to define the visual element, the apparatus further including a positional inference generator to determine a value of a positional property for the visual element with a spatial transformer network, the DSL statement generator to generate the DSL statement by associating the value of the positional property with the DSL command.

### Claim 10 (depends on 9)

The apparatus of claim 9, wherein the second neural network is a recurrent neural network, the DSL statement generator to generate different DSL commands at each iteration of the recurrent neural network, a first DSL command generated in a first iteration to be provided as an input to the recurrent neural network to generate a second DSL command at a second iteration following the first iteration.

### Claim 11 (depends on 1)

The apparatus of claim 1, further including a style generator to automatically generate a style property for the visual element of the GUI design with a neural network, the rendering tool to render the mockup based on the style property.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein the neural network is associated with a generative adversarial network.

### Claim 13 (depends on 11)

The apparatus of claim 11, wherein the neural network is a global style generator network and the style property is a global style property, the global style property to define a property applicable to multiple visual elements of the GUI design.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein the style generator includes a local style generator to automatically generate a local style property for at least one of the visual elements, the local style property based on at least one of (1) the DSL statement corresponding to the at least one of the visual elements, (2) a value of the global style property, or (3) a user-provided value.

### Claim 15

A non-transitory computer readable medium comprising instructions that, when executed, cause a machine to at least:
encode a user-provided textual description of a GUI design with a first neural network;
generate a DSL statement with a second neural network, the DSL statement to define a visual element of the GUI design, the DSL statement generated based on at least one of the encoded textual description or a user-provided image representative of the GUI design; and
render a mockup of the GUI design based on the DSL statement.

### Claim 16

(canceled)

### Claim 17

(canceled)

### Claim 18 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the instructions further cause the machine to:
determine a difference between the user-provided image and a rendered image with a third neural network, the rendered image corresponding to an approximation of the GUI design; and
generate the DSL statement based on the difference.

### Claim 19

(canceled)

### Claim 20

(canceled)

### Claim 21

(canceled)

### Claim 22 (depends on 18)

The non-transitory computer readable medium of claim 18, wherein the difference is a first difference, the DSL statement is a first DSL statement, and the rendered image is a first rendered image, the instructions to further cause the machine to:
generate a second rendered image based on the DSL statement;
determine a second difference between the user-provided image and the second rendered image with the third neural network; and
generate a second DSL statement based on the second difference.

### Claim 23 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the instructions further cause the machine to:
generate a DSL command with the second neural network, the DSL command to define the visual element;
determine a value of positional property for the visual element with a spatial transformer network; and
associate the value of the positional property with the DSL command.

### Claim 24

(canceled)

### Claim 25 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the instructions further cause the machine to:
automatically generate a style property for the visual element of the GUI design with a neural network; and
render the mockup based on the style property.

### Claim 26

(canceled)

### Claim 27 (depends on 25)

The non-transitory computer readable medium of claim 25, wherein the neural network is a global style generator network and the style property is a global style property, the global style property to define a property applicable to multiple visual elements of the GUI design.

### Claim 28 (depends on 27)

The non-transitory computer readable medium of claim 27, wherein the instructions further cause the machine to automatically generate a local style property for at least one of the visual elements with a local style generator network, the local style property based on at least one of (1) the DSL statement corresponding to the at least one of the visual elements, (2) a value of the global style property, or (3) a user-provided value.

### Claim 29

A method comprising:
encoding a user-provided textual description of a GUI design with a first neural network;
generating a DSL statement with a second neural network, the DSL statement to define a visual element of the GUI design, the DSL statement generated based on at least one of the encoded textual description or a user-provided image representative of the GUI design; and
rendering a mockup of the GUI design based on the DSL statement.

### Claim 30

(canceled)

### Claim 31

(canceled)

### Claim 32 (depends on 29)

The method of claim 29, further including:
determining a difference between the user-provided image and a rendered image with a third neural network, the rendered image corresponding to an approximation of the GUI design; and
generating the DSL statement based on the difference.

### Claim 33

(canceled)

### Claim 34

(canceled)

### Claim 35

(canceled)

### Claim 36 (depends on 32)

The method of claim 32, wherein the difference are first difference is a first difference, the DSL statement is a first DSL statement, and the rendered image is a first rendered image, the method further including:
generating a second rendered image based on the DSL statement;
determining a second difference between the user-provided image and the second rendered image with the third neural network; and
generating a second DSL statement based on the second difference.

### Claim 37

(canceled)

### Claim 38

(canceled)

### Claim 39 (depends on 29)

The method of claim 29, further including:
automatically generating a style property for the visual element of the GUI design with a neural network; and
rendering the mockup based on the style property.

### Claim 40

(canceled)

### Claim 41

(canceled)

### Claim 42

(canceled)
