# US 12,118,662 B2 · Optimizing computer-based generation of three-dimensional virtual objects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,662 B2
- Document ID: US12118662B2
- Status: Granted
- Kind: B2
- Application number: 17/933,378
- Filed: 2022-09-19
- Published: 2024-03-21
- Granted: 2024-10-15
- Earliest claimed: 2022-09-19
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Jeremy R. Fox; Martin G. Keen; Alexander Reznicek; Bahman Hekmatshoartabari
- Web page: https://patentgrep.com/patent/US12118662B2

## CPC

- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06V 20/20: in augmented reality scenes
- G06F 40/00: Handling natural language data (speech analysis or synthesis, speech recognition G10L)
- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06T 2219/2012: Colour editing, changing, or manipulating; Use of colour codes
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

In an approach to improve the generation of a virtual object in a three-dimensional virtual environment, embodiments of the present invention identify a virtual object to be generated in a three-dimensional virtual environment based on a natural language utterance. Additionally, embodiments generate the virtual object based on a CLIP-guided Generative Latent Space (CLIP-GLS) analysis, and monitor usage of the generated virtual object in the three-dimensional virtual space. Moreover, embodiments infer human perception data from the monitoring, and generate a utility score for the virtual object based on the human perception data.

## Claims

### Claim 1

A computer-implemented method comprising:
identifying, by one or more processors, a virtual object to be generated in a three-dimensional virtual environment based on a natural language utterance;
generating the virtual object based on a CLIP-guided Generative Latent Space (CLIP-GLS) analysis;
monitoring usage of the generated virtual object in the three-dimensional virtual space;
inferring human perception data from the monitoring; and
generating a utility score for the virtual object based on the human perception data.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
extracting features of the generated virtual object.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising:
analyzing a Hierarchical Text-Conditional Image Generation utterance to classify the virtual object to be constructed, and to classify one or more contextual cues in the three-dimensional virtual environment.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, further comprising:
analyzing a CLIP-Guided Generative Latent Space of the generated object to determine features of the classified virtual object of the classified contextual cues in the three-dimensional virtual environment.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, further comprising:
actively modifying one or more determined features of the classified virtual object or classified contextual cues in the three-dimensional environment by modifying CLIP-Guided Generative Latent Space based on feedback associated with the virtual object to produce a new object with increased utility in a similar context.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, further comprising:
inpainting a first virtual object in a predetermined virtual environment; and
modifying the first inpainted virtual object by matching one or more latent vectors of a second inpainted virtual object and blending the first inpainted virtual object and the second inpainted virtual object together.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprising:
modifying the latent space, based on user feedback, by sampling from the latent space to achieve a predetermined utility of the generated virtual object, wherein an agent may be trained to generate an updated version of the virtual object that maximizes a utility of an inpainted virtual object.

### Claim 8

A computer system, the computer system comprising:
one or more computer processors;
one or more computer readable storage devices; and
program instructions stored on the one or more computer readable storage devices for execution by at least one of the one or more computer processors, the stored program instructions comprising:
program instructions to identify a virtual object to be generated in a three-dimensional virtual environment based on a natural language utterance;
program instructions to generate the virtual object based on a CLIP-guided Generative Latent Space (CLIP-GLS) analysis;
program instructions to monitor usage of the generated virtual object in the three-dimensional virtual space;
program instructions to infer human perception data from the monitoring; and
program instructions to generate a utility score for the virtual object based on the human perception data.

### Claim 9 (depends on 8)

The computer system of claim 8, further comprising:
program instructions to extract features of the generated virtual object.

### Claim 10 (depends on 8)

The computer system of claim 8, further comprising:
program instructions to analyze a Hierarchical Text-Conditional Image Generation utterance to classify the virtual object to be constructed, and to classify one or more contextual cues in the three-dimensional virtual environment.

### Claim 11 (depends on 10)

The computer system of claim 10, further comprising:
program instructions to analyze a CLIP-Guided Generative Latent Space of the generated object to determine features of the classified virtual object of the classified contextual cues in the three-dimensional virtual environment.

### Claim 12 (depends on 8)

The computer system of claim 8, further comprising:
program instructions to actively modify one or more determined features of the classified virtual object or classified contextual cues in the three-dimensional environment by modifying CLIP-Guided Generative Latent Space based on feedback associated with the virtual object to produce a new object with increased utility in a similar context.

### Claim 13 (depends on 8)

The computer system of claim 8, further comprising:
program instructions to inpainting a first virtual object in a predetermined virtual environment; and
program instructions to modify the first inpainted virtual object by matching one or more latent vectors of a second inpainted virtual object and blending the first inpainted virtual object and the second inpainted virtual object together.

### Claim 14 (depends on 8)

The computer system of claim 8, further comprising:
program instructions to modify the latent space, based on user feedback, by sampling from the latent space to achieve a predetermined utility of the generated virtual object, wherein an agent may be trained to generate an updated version of the virtual object that maximizes a utility of an inpainted virtual object.

### Claim 15

A computer program product, the computer program product comprising:
one or more computer readable storage devices and program instructions stored on the one or more computer readable storage devices, the stored program instructions comprising:
program instructions to identify a virtual object to be generated in a three-dimensional virtual environment based on a natural language utterance;
program instructions to generate the virtual object based on a CLIP-guided Generative Latent Space (CLIP-GLS) analysis;
program instructions to monitor usage of the generated virtual object in the three-dimensional virtual space;
program instructions to infer human perception data from the monitoring; and
program instructions to generate a utility score for the virtual object based on the human perception data.

### Claim 16 (depends on 15)

The computer program product of claim 15, further comprising:
program instructions to extract features of the generated virtual object.

### Claim 17 (depends on 15)

The computer program product of claim 15, further comprising:
program instructions to analyze a Hierarchical Text-Conditional Image Generation utterance to classify the virtual object to be constructed, and to classify one or more contextual cues in the three-dimensional virtual environment; and
program instructions to analyze a CLIP-Guided Generative Latent Space of the generated object to determine features of the classified virtual object of the classified contextual cues in the three-dimensional virtual environment.

### Claim 18 (depends on 15)

The computer program product of claim 15, further comprising:
program instructions to actively modify one or more determined features of the classified virtual object or classified contextual cues in the three-dimensional environment by modifying CLIP-Guided Generative Latent Space based on feedback associated with the virtual object to produce a new object with increased utility in a similar context.

### Claim 19 (depends on 15)

The computer program product of claim 15, further comprising:
program instructions to inpainting a first virtual object in a predetermined virtual environment; and
program instructions to modify the first inpainted virtual object by matching one or more latent vectors of a second inpainted virtual object and blending the first inpainted virtual object and the second inpainted virtual object together.

### Claim 20 (depends on 15)

The computer program product of claim 15, further comprising:
program instructions to modify the latent space, based on user feedback, by sampling from the latent space to achieve a predetermined utility of the generated virtual object, wherein an agent may be trained to generate an updated version of the virtual object that maximizes a utility of an inpainted virtual object.