US 11,887,246 B2 · Generating ground truth datasets for virtual reality experiences

Abstract

Systems and methods of generating ground truth datasets for producing virtual reality (VR) experiences, for testing simulated sensor configurations, and for training machine-learning algorithms. In one example, a recording device with one or more cameras and one or more inertial measurement units captures images and motion data along a real path through a physical environment. A SLAM application uses the captured data to calculate the trajectory of the recording device. A polynomial interpolation module uses Chebyshev polynomials to generate a continuous time trajectory (CTT) function. The method includes identifying a virtual environment and assembling a simulated sensor configuration, such as a VR headset. Using the CTT function, the method includes generating a ground truth output dataset that represents the simulated sensor configuration in motion along a virtual path through the virtual environment. The virtual path is closely correlated with the motion along the real path as captured by the recording device. Accordingly, the output dataset produces a realistic and life-like VR experience. In addition, the methods described can be used to generate multiple output datasets, at various sample rates, which are useful for training the machine-learning algorithms which are part of many VR systems.

Claims20

Dependent claims

A method of generating a ground truth output dataset using an electronic eyewear device comprising a processor, a camera, and an inertial measurement unit, the method comprising steps of:

recording a primary reference set using the electronic eyewear device in motion traversing a path through a physical environment, wherein the primary reference set comprises a series of real images captured by the camera and a series of real motion data captured by the inertial measurement unit;
calculating, by the processor, a headset trajectory based on the primary reference set;
generating, by the processor, a continuous time trajectory function for interpolating the headset trajectory relative to the series of real motion data;
and
generating, by the processor, an output dataset associated with a simulated sensor configuration and in accordance with the continuous time trajectory function, wherein the simulated sensor configuration comprises one or more simulated cameras and one or more simulated inertial measurement unit sensors, such that the output dataset represents the simulated sensor configuration in motion along a virtual path through a first virtual environment.

2 more elements

A system comprising:

an electronic eyewear device comprising a processor, a camera, and an inertial measurement unit;
a trajectory recording application configured to record a primary reference set using the electronic eyewear device in motion traversing a path through a physical environment, wherein the primary reference set comprises a series of real images captured by the camera and a series of real motion data captured by the inertial measurement unit;
a headset trajectory based on the primary reference set;
a polynomial interpolation module configured to generate a continuous time trajectory function for interpolating the headset trajectory relative to the series of real motion data;
and
an output dataset associated with a simulated sensor configuration and generated in accordance with the continuous time trajectory function, wherein the simulated sensor configuration comprises one or more simulated cameras and one or more simulated inertial measurement unit sensors, such that the output dataset represents the simulated sensor configuration in motion along a virtual path through a first virtual environment.

3 more elements

A non-transitory computer-readable medium storing program code that, when executed, is operative to cause a processor of an eyewear device to perform steps of:

recording a primary reference set using the eyewear device in motion traversing a path through a physical environment, wherein the primary reference set comprises a series of real images captured by a camera coupled to the eyewear device and a series of real motion data captured by an inertial measurement unit coupled to the eyewear device;
calculating, by the processor, a headset trajectory based on the primary reference set;
generating, by the processor, a continuous time trajectory function for interpolating the headset trajectory relative to the series of real motion data;
and
generating, by the processor, an output dataset associated with a simulated sensor configuration and in accordance with the continuous time trajectory function, wherein the simulated sensor configuration comprises one or more simulated cameras and one or more simulated inertial measurement unit sensors, such that the output dataset represents the simulated sensor configuration in motion along a virtual path through a first virtual environment.

2 more elements

Dependencies are inferred from the claim text.

Generating ground truth datasets for virtual reality experiences

Life of this patent

Generating ground truth datasets for virtual reality experiences

Life of this patent

Abstract

Claims20