# US 2021/0174801 A1 · Device, method, and program for enhancing output content through iterative generation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174801 A1
- Document ID: US20210174801A1
- Status: Published application
- Kind: A1
- Application number: 17/111,734
- Filed: 2020-12-04
- Published: 2021-06-10
- Earliest claimed: 2019-12-04
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Seohyun BACK; Yonghyun RYU; Wonho RYU; Haejun LEE; Cheolseung JUNG; Sai CHETAN; Jiyeon HONG
- Web page: https://patentgrep.com/patent/US20210174801A1

## CPC

- G06F 40/30: Semantic analysis
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 13/00: Animation
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/82: using neural networks
- G06V 20/20: in augmented reality scenes
- G06V 40/161: {Detection; Localisation; Normalisation}
- G10L 15/16: using artificial neural networks
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 2015/223: {Execution procedure of a spoken command}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method of improving output content through iterative generation is provided. The method includes receiving a natural language input, obtaining user intention information based on the natural language input by using a natural language understanding (NLU) model, setting a target area in base content based on a first user input, determining input content based on the user intention information or a second user input, generating output content related to the base content based on the input content, the target area, and the user intention information by using a neural network (NN) model, generating a caption for the output content by using an image captioning model, calculating similarity between text of the natural language input and the generated output content, and iterating generation of the output content based on the similarity.

## Claims

### Claim 1

A device for improving output content through iterative generation, the device comprising:
a memory storing instructions; and
at least one processor configured to execute the instructions to:
receive a natural language input,
obtain user intention information based on the natural language input by using a natural language understanding (NLU) model,
set a target area in base content based on a first user input,
determine input content based on the user intention information or a second user input,
generate output content related to the base content based on the input content, the target area, and the user intention information by using a neural network (NN) model,
generate a caption for the output content by using an image captioning model,
calculate a similarity between text of the natural language input and the generated output content, and
iterate generation of the output content based on the similarity.

### Claim 2 (depends on 1)

The device of claim 1,
wherein the base content, the input content, and the output content are images, and
wherein the output content is generated by compositing the input content into the target area of the base content.

### Claim 3 (depends on 1)

The device of claim 1,
wherein the base content comprises a plurality of areas, and
wherein the target area comprises an area selected from among the plurality of areas by the first user input.

### Claim 4 (depends on 1)

The device of claim 1,
wherein the natural language input comprises a voice input, and
wherein the voice input is converted into the text of the natural language input by using an automatic speech recognition (ASR) model.

### Claim 5 (depends on 1)

The device of claim 1, wherein the input content is determined based on content information included in the user intention information.

### Claim 6 (depends on 5)

The device of claim 5, wherein the input content is further determined from a plurality of pieces of content corresponding to the content information, and the plurality of pieces of content have different attributes from each other.

### Claim 7 (depends on 1)

The device of claim 1,
wherein an attribute of the input content comprises at least one of a pose, a facial expression, make-up, hair, apparel, or an accessory, and
wherein the attribute of the input content is determined based on content attribute information included in the user intention information.

### Claim 8 (depends on 1)

The device of claim 1,
wherein the NN model is related to a generated adversarial network (GAN) model, and
wherein the output content is generated by a generator of the GAN model.

### Claim 9 (depends on 1)

The device of claim 1, wherein probability distribution of the output content or the base content including the output content corresponds to probability distribution of real content.

### Claim 10 (depends on 1)

The device of claim 1,
wherein the NN model is related to a generated adversarial network (GAN) model, and
wherein a discriminator of the GAN model identifies the output content as fake content when the similarity does not satisfy a predetermined condition.

### Claim 11 (depends on 1)

The device of claim 1,
wherein the output content is first output content, and
wherein the at least one processor is further configured to, when the similarity does not satisfy a predetermined condition, execute the instructions to:
generate second output content different from the first output content based on the input content, the target area, and the user intention information by using the NN model.

### Claim 12 (depends on 1)

The device of claim 1,
wherein the input content is first input content, and the output content is first output content, and
wherein the at least one processor is further configured to, when the similarity does not satisfy a predetermined condition, execute the instructions to:
determine second input content different from the first input content, and
generate second output content different from the first output content based on the second input content and the target area by using the NN model, when the similarity does not satisfy the predetermined condition.

### Claim 13 (depends on 1)

The device of claim 1, wherein the at least one processor is further configured to execute the instructions to:
receive user feedback regarding a part of the output content, and
modify the part of the output content by using the NN model.

### Claim 14 (depends on 1)

The device of claim 1,
wherein the base content comprises a workspace of an application, and
wherein the input content comprises a work object located in the workspace.

### Claim 15 (depends on 14)

The device of claim 14,
wherein the output content comprises an animation related to the work object,
wherein the animation is generated based on the work object, the user intention information, and an application programming interface (API) of the application, and
wherein the caption for the output content comprises a caption for the animation.

### Claim 16 (depends on 1)

The device of claim 1, wherein the NLU model, the NN model, and the image captioning model are stored in the memory.

### Claim 17 (depends on 1)

The device of claim 1, wherein the iterative generation of the output content continues until the output content is generated having the caption similar to the text of the natural language input.

### Claim 18 (depends on 1)

The device of claim 1,
wherein the similarity is a vector similarity, and
wherein the text of the natural language input and the caption is encoded using a semantic vector to generate corresponding vectors, and a similarity between the vectors is calculated.

### Claim 19

A method of improving output content through iterative generation, the method comprising:
receiving a natural language input;
obtaining user intention information based on the natural language input by using a natural language understanding (NLU) model;
setting a target area in base content based on a first user input;
determining input content based on the user intention information or a second user input;
generating output content related to the base content based on the input content, the target area, and the user intention information by using a neural network (NN) model;
generating a caption for the output content by using an image captioning model;
calculating similarity between text of the natural language input and the generated output content; and
iterating generation of the output content based on the similarity.

### Claim 20

A non-transitory computer-readable storage medium comprising instructions which, when executed by at least one processor, causes the at least one processor to:
receive a natural language input;
obtain user intention information based on the natural language input by using a natural language understanding (NLU) model;
set a target area in base content based on a first user input;
determine input content based on the user intention information or a second user input;
generate output content related to the base content based on the input content, the target area, and the user intention information by using a neural network (NN) model;
generate a caption for the output content by using an image captioning model;
calculate similarity between text of the natural language input and the generated output content; and
iterate generation of the output content based on the similarity.
