US 2023/0144893 A1Published application

Automatic Artificial Reality World Creation

AssigneeMeta Platforms Technologies, LLC

InventorChun-Wei CHAN

Life of this patent

Estimating the term…

Loading patent…

Abstract

Methods and systems described herein are directed to creating an artificial reality environment having elements automatically created from source images. In response to a creation system receiving the source images, the system can employ a multi-layered comparative analysis to obtain virtual object representations of objects depicted in the source images. A first set of the virtual objects can be selected from a library by matching identifiers for the depicted objects with tags on virtual objects in the library. A second set of virtual objects can be objects for which no candidate first virtual objects was adequately matched in the library, prompting the creation of a virtual object by generating depth data and skinning a resulting 3D mesh based on the source images. Having determined the virtual objects, the system can compile them into the artificial reality environment according to relative locations determined from the source images.

Claims20

Dependent claims

A method of generating an artificial reality environment, the method comprising:

receiving one or more source images with depicted objects;
converting the one or more source images into machine learning model input;
generating object identifiers for one or more of the depicted objects by applying the machine learning model input to a first machine learning model trained to match image data to object identifiers;
selecting one or more first virtual objects, from a virtual object library, that match at least one of the object identifiers;
creating one or more second virtual objects, for one or more of the depicted objects that do not correspond to the selected one or more first virtual objects, by:
generating a 3D model, for each of the at least one second virtual objects, by applying at least portions of representations of the one or more source images to a second machine learning model trained to predict depth for the corresponding depicted objects and converting at least a part of depth data produced by the second machine learning model into a 3D mesh;
and
applying a texture for each of the of the at least one second virtual objects based on the corresponding portions of the one or more source images;
identifying relative locations for each of the one or more first virtual objects and the one or more second virtual objects;
and
compiling the one or more first virtual objects and the one or more second virtual objects, based on the identified relative locations, into the artificial reality environment.

8 more elements

A computing system for generating an artificial reality environment, the computing system comprising:

one or more processors;
and
one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:
receiving one or more source images with depicted objects;
selecting one or more first virtual objects, from a virtual object library, that match at least one of the depicted objects;
creating one or more second virtual objects, for one or more of the depicted objects that do not correspond to the one or more first virtual objects, by:
generating a 3D model, for each of the at least one second virtual objects;
and
applying a texture for each of the of the at least one second virtual objects based on the corresponding portions of the one or more source images;
identifying relative locations for each of the first and second virtual objects;
and
compiling the first and second virtual objects, based on the identified relative locations, into the artificial reality environment.

9 more elements

A machine-readable storage medium having machine-executable instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform a method for generating an artificial reality environment, the method comprising:

receiving one or more source images with depicted objects;
selecting one or more first virtual objects, from a virtual object library, that match at least one of the depicted objects;
creating one or more second virtual objects, for one or more of the depicted objects that do not correspond to the one or more first virtual objects, by:
generating a 3D model, for each of the at least one second virtual objects;
and
applying a texture for each of the of the at least one second virtual objects based on the corresponding portions of the one or more source images;
identifying relative locations for each of the first and second virtual objects;
and
compiling the first and second virtual objects, based on the identified relative locations, into the artificial reality environment.

6 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/689,164Filed 8 Mar 2022
Priority: 10 Nov 2021
Published: 11 May 2023
Term ends: Not estimatedThe source records don’t support an estimate.

Classification19 codes

G06T 17/00
Three-dimensional [3d] modelling for computer graphics
G06V 20/647
Three-dimensional [3D] objects: by matching two-dimensional images to three-dimensional objects
G06T 7/55
Depth or shape recovery: from multiple images
G06T 7/74
Using feature-based methods: involving reference images or patches