# US 2022/0036657 A1 · Systems and methods for automated insertion of supplemental content into a virtual environment using a machine learning model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036657 A1
- Document ID: US20220036657A1
- Status: Published application
- Kind: A1
- Application number: 16/940,982
- Filed: 2020-07-28
- Published: 2022-02-03
- Earliest claimed: 2020-07-28
- Assignees: Rovi Guides, Inc.
- Applicants: Rovi Guides, Inc.
- Inventors: Aashish Goyal; Ajay Kumar Mishra; Jeffry Copps Robert Jose
- Web page: https://patentgrep.com/patent/US20220036657A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/24: Classification techniques
- G06N 3/09: Supervised learning
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 20/00: Machine learning
- G06N 20/20: Ensemble learning
- G06Q 30/0271: {Personalized advertisement}
- G06Q 30/0277: {Online advertisement}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06N 3/08: Learning methods
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06T 2219/2021: Shape modification
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Insertion of supplemental content into a virtual environment is automated using a machine learning model. The machine learning model is trained to calculate a confidence value that a candidate virtual object fits into a virtual environment based on an input that includes a candidate virtual object, a list of persistent virtual objects, and a list of temporary virtual objects. The machine learning model is trained using the persistent and temporary objects displayed in the current virtual environment until it predicts that a selected virtual object fits into the current virtual environment. The trained machine learning model is then used to select a virtual object comprising supplemental content to be inserted as a new virtual object in the virtual environment.

## Claims

### Claim 1

A method for inserting supplemental content into a three-dimensional virtual environment, the method comprising:
identifying a first plurality of persistent virtual objects displayed in a plurality of consecutive virtual environment frames;
identifying a second plurality of temporary virtual objects displayed in the plurality of consecutive virtual environment frames;
selecting a first virtual object from a first virtual environment frame of the plurality of consecutive virtual environment frames;
training a machine learning model to calculate a confidence value that a candidate virtual object fits into a given virtual environment based on an input that includes (a) a candidate virtual object, (b) a list of persistent virtual objects in the virtual environment, and (c) a list of temporary virtual objects in the virtual environment, wherein the machine learning model is trained using a training example that predicts that the first virtual object fits into a virtual environment that comprises the first plurality of persistent virtual objects and the second plurality of temporary virtual objects;
retrieving a candidate object comprising supplemental content for insertion into the virtual environment;
determining, using the machine learning model, whether the candidate object fits into the virtual environment; and
in response to determining that the candidate object fits in the virtual environment, inserting the candidate object into the virtual environment.

### Claim 2 (depends on 1)

The method of claim 1, wherein identifying the first plurality of persistent virtual objects displayed in a plurality of consecutive virtual environment frames and identifying a second plurality of temporary virtual objects displayed in the plurality of consecutive virtual environment frames comprises:
identifying as the first plurality of persistent virtual objects all virtual objects displayed in a first virtual environment frame of the plurality of consecutive virtual environment frames; and
for each other virtual environment frame of the plurality of consecutive virtual environment frames:
comparing the first plurality of persistent virtual objects with a third plurality of virtual objects displayed in the respective virtual environment frame;
in response to determining, based on the comparing, that the first plurality of persistent virtual objects does not include a virtual object of the third plurality of virtual objects, identifying the respective virtual object of the third plurality of virtual objects as a temporary virtual object; and
in response to determining that the third plurality of virtual objects does not include a persistent virtual object of the plurality of persistent virtual objects, reidentifying the respective persistent virtual object as a temporary virtual object.

### Claim 3 (depends on 1)

The method of claim 1, wherein a virtual environment frame comprises all virtual objects in a field of view on a display on which the virtual environment in generated.

### Claim 4 (depends on 1)

The method of claim 1, wherein each persistent virtual object of the first plurality of persistent virtual objects comprises object attributes that describe the respective persistent virtual object, and wherein each temporary virtual object of the second plurality of temporary virtual objects comprises object attributes that describe the respective temporary virtual object.

### Claim 5 (depends on 4)

The method of claim 4, wherein retrieving a candidate object comprising supplemental content for insertion into the virtual environment comprises:
accessing a database of supplemental content items, each supplemental content item having at least one content attribute;
identifying a plurality of object attributes of each persistent virtual object of the first plurality of persistent virtual objections and each temporary virtual object of the second plurality of temporary virtual objects;
determining whether the at least one content attribute of a supplemental content item matches at least one object attribute of the plurality of object attributes; and
in response to determining that the at least one content attribute of the supplemental content item matches at least one object attribute of the plurality of object attributes, selecting the supplemental content item as a candidate for insertion.

### Claim 6 (depends on 4)

The method of claim 4, further comprising:
maintaining a list of virtual objects displayed from a first time to a current time;
wherein retrieving a candidate object comprising supplemental content for insertion into the virtual environment comprises retrieving a candidate object having a content attribute that matches at least one object attribute of a virtual object in the list.

### Claim 7 (depends on 1)

The method of claim 1, wherein retrieving a candidate object comprising supplemental content for insertion into the virtual environment comprises:
identifying a plurality of available virtual objects;
determining, for each available virtual object of the plurality of virtual objects, a time at which the respective available virtual object was previously displayed in the virtual environment;
determining, for each available virtual object of the plurality of available virtual objects, a weight for the respective available virtual object based on the time at which the respective available virtual object was previously displayed; and
selecting a candidate object having at least one content attribute matching the available virtual object having a highest weight.

### Claim 8 (depends on 1)

The method of claim 1, wherein the candidate object comprises a three-dimensional object.

### Claim 9

A system for inserting supplemental content into a three-dimensional virtual environment, the system comprising:
output circuitry configured to drive display of a virtual environment; and
control circuitry configured to:
identify a first plurality of persistent virtual objects displayed in a plurality of consecutive virtual environment frames;
identify a second plurality of temporary virtual objects displayed in the plurality of consecutive virtual environment frames;
select a first virtual object from a first virtual environment frame of the plurality of consecutive virtual environment frames;
train a machine learning model to calculate a confidence value that a candidate virtual object fits into a given virtual environment based on an input that includes (a) a candidate virtual object, (b) a list of persistent virtual objects in the virtual environment, and (c) a list of temporary virtual objects in the virtual environment, wherein the machine learning model is trained using a training example that predicts that the first virtual object fits into a virtual environment that comprises the first plurality of persistent virtual objects and the second plurality of temporary virtual objects;
retrieve a candidate object comprising supplemental content for insertion into the virtual environment;
determine, using the machine learning model, whether the candidate object fits into the virtual environment; and
in response to determining that the candidate object fits in the virtual environment, insert the candidate object into the virtual environment.

### Claim 10 (depends on 9)

The system of claim 9, wherein the control circuitry configured to identify the first plurality of persistent virtual objects displayed in a plurality of consecutive virtual environment frames and to identify a second plurality of temporary virtual objects displayed in the plurality of consecutive virtual environment frames is further configured to:
identify as the first plurality of persistent virtual objects all virtual objects displayed in a first virtual environment frame of the plurality of consecutive virtual environment frames; and
for each other virtual environment frame of the plurality of consecutive virtual environment frames:
compare the first plurality of persistent virtual objects with a third plurality of virtual objects displayed in the respective virtual environment frame;
in response to determining, based on the comparing, that the first plurality of persistent virtual objects does not include a virtual object of the third plurality of virtual objects, identify the respective virtual object of the third plurality of virtual objects as a temporary virtual object; and
in response to determining that the third plurality of virtual objects does not include a persistent virtual object of the plurality of persistent virtual objects, reidentify the respective persistent virtual object as a temporary virtual object.

### Claim 11 (depends on 9)

The system of claim 9, wherein a virtual environment frame comprises all virtual objects in a field of view on a display on which the virtual environment in generated.

### Claim 12 (depends on 9)

The system of claim 9, wherein each persistent virtual object of the first plurality of persistent virtual objects comprises object attributes that describe the respective persistent virtual object, and wherein each temporary virtual object of the second plurality of temporary virtual objects comprises object attributes that describe the respective temporary virtual object.

### Claim 13 (depends on 12)

The system of claim 12, wherein the control circuitry configured to retrieve a candidate object comprising supplemental content for insertion into the virtual environment is further configured to:
access a database of supplemental content items, each supplemental content item having at least one content attribute;
identify a plurality of object attributes of each persistent virtual object of the first plurality of persistent virtual objections and each temporary virtual object of the second plurality of temporary virtual objects;
determine whether the at least one content attribute of a supplemental content item matches at least one object attribute of the plurality of object attributes; and
in response to determining that the at least one content attribute of the supplemental content item matches at least one object attribute of the plurality of object attributes, select the supplemental content item as a candidate for insertion.

### Claim 14 (depends on 12)

The system of claim 12, wherein the control circuitry is further configured to:
maintain a list of virtual objects displayed from a first time to a current time;
wherein the control circuitry configured to retrieve a candidate object comprising supplemental content for insertion into the virtual environment is further configured to retrieve a candidate object having a content attribute that matches at least one object attribute of a virtual object in the list.

### Claim 15 (depends on 9)

The system of claim 9, wherein the control circuitry configured to retrieve a candidate object comprising supplemental content for insertion into the virtual environment is further configured to:
identify a plurality of available virtual objects;
determine, for each available virtual object of the plurality of virtual objects, a time at which the respective available virtual object was previously displayed in the virtual environment;
determine, for each available virtual object of the plurality of available virtual objects, a weight for the respective available virtual object based on the time at which the respective available virtual object was previously displayed; and
select a candidate object having at least one content attribute matching the available virtual object having a highest weight.

### Claim 16 (depends on 9)

The system of claim 9, wherein the candidate object comprises a three-dimensional object.

### Claim at position 17

17-40. (canceled)