# US 2021/0334601 A1 · Spatial localization design service

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334601 A1
- Document ID: US20210334601A1
- Status: Published application
- Kind: A1
- Application number: 17/371,093
- Filed: 2021-07-08
- Published: 2021-10-28
- Earliest claimed: 2018-05-08
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Michael EBSTYNE; Pedro Urbina ESCOS; Emanuel SHALEV; Alex KIPMAN; Yuri PEKELNY; Jonathan Chi Hang CHAN
- Web page: https://patentgrep.com/patent/US20210334601A1

## CPC

- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 11/261: {by simulating additional hardware, e.g. fault simulation}
- G06F 11/3684: {for test design, e.g. generating new test cases}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 30/20: Design optimisation, verification or simulation (optimisation, verification or simulation of circuit designs G06F30/30)
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 30/27: using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
- G06F 2111/18: using virtual or augmented reality
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 19/003: {Navigation within 3D models or images}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A synthetic world interface may be used to model digital environments, sensors, and motions for the evaluation, development, and improvement of localization algorithms. A synthetic data cloud service with a library of sensor primitives, motion generators, and environments with procedural and game-like capabilities, facilitates engineering design for a manufactural solution that has localization capabilities. In some embodiments, a sensor platform simulator operates with a motion orchestrator, an environment orchestrator, an experiment generator, and an experiment runner to test various candidate hardware configurations and localization algorithms in a virtual environment, advantageously speeding development and reducing cost. Thus, examples disclosed herein may relate to virtual reality (VR) or mixed reality (MR) implementations.

## Claims

### Claim 1

A system, comprising:
memory embodied with executable instructions for simulating a first hardware configuration; and
at least one processor programmed for:
simulating one or more virtual environments;
varying lighting in different instances of the simulated one or more virtual environments for testing a localization algorithm;
simulating three-dimensional (3D) motion of the first hardware configuration within the different instances of the one or more virtual environments in which the lighting is varied;
generating synthetic experiment data for the first hardware configuration having
the simulated 3D motion within the one or more virtual environments in which the lighting is varied;
applying one or more localization algorithms to the first hardware configurations with the simulated 3D motion in the one or more virtual environments in which the lighting is varied;
determining disparity data of the one or more localization algorithms against ground truth data from the one or more localization algorithms applied to the simulated first hardware configuration with the simulated 3D motion in the one or more virtual environments in which the lighting is varied;
based on the disparity data of the one or more localization algorithms against ground truth data, selecting a first localization algorithm for the first hardware configuration.

### Claim 2 (depends on 1)

The system of claim 1, wherein the first hardware configuration comprises a first lens and a first sensor.

### Claim 3 (depends on 1)

The system of claim 1, wherein the first hardware configuration is a first camera.

### Claim 4 (depends on 1)

The system of claim 1, wherein the at least one processor is further programmed for:
determining the disparity data exceeds a variance threshold;
based on the disparity data exceeding the variance threshold, directing an artificial intelligence (AI) application to run subsequent testing of the one or more localization algorithms against a different synthetic scene, motion, or hardware configuration; and
modifying the one or more localization algorithms based on subsequent testing.

### Claim 5 (depends on 1)

The system of claim 1, wherein 3D motion data comprises inertial measurement unit (IMU) data.

### Claim 6 (depends on 5)

The system of claim 5, wherein the IMU data comprises accelerometer data.

### Claim 7 (depends on 5)

The system of claim 5, wherein the IMU data comprises gyroscope data.

### Claim 8

A method comprising:
simulating one or more virtual environments;
varying lighting in different instances of the simulated one or more virtual environments for testing a localization algorithm;
simulating three-dimensional (3D) motion of a first hardware configuration within the different instances of the one or more virtual environments in which the lighting is varied;
generating synthetic experiment data for the first hardware configuration having the simulated 3D motion within the one or more virtual environments in which the lighting is varied;
applying one or more localization algorithms to the first hardware configurations with the simulated 3D motion in the one or more virtual environments in which the lighting is varied;
determining disparity data of the one or more localization algorithms against ground truth data from the one or more localization algorithms applied to the simulated first hardware configuration with the simulated 3D motion in the one or more virtual environments in which the lighting is varied;
based on the disparity data of the one or more localization algorithms against ground truth data, selecting a first localization algorithm for the first hardware configuration.

### Claim 9 (depends on 8)

The method of claim 8, further comprising running subsequent testing of the first localization algorithm incident to the disparity data exceeding a variance threshold.

### Claim 10 (depends on 9)

The method of claim 9, further comprising modifying the first localization algorithm based on the subsequent testing run incident to the disparity data exceeding the variance threshold.

### Claim 11 (depends on 8)

The method of claim 8, wherein the synthetic experiment data comprises synthetic images.

### Claim 12 (depends on 8)

The method of claim 8, wherein the synthetic experiment data comprises inertial measurement unit (IMU) data.

### Claim 13 (depends on 12)

The method of claim 12, wherein the IMU data comprises accelerometer data.

### Claim 14 (depends on 12)

The method of claim 12, wherein the IMU data comprises gyroscope data.

### Claim 15

One or more computer storage devices having computer-executable instructions stored thereon for developing a localization solution, which, on execution by a computer, cause the computer to perform operations comprising:
simulating one or more hardware configurations comprising one or more virtual sensors;
simulating one or more virtual environments;
generating synthetic experiment data for a plurality of candidate localization solutions having differing hardware configurations or localization algorithm parameters;
simulating movement of the one or more hardware configurations with the simulated motion in the one or more virtual environments;
applying one or more localization algorithms to the simulated movement of the one or more hardware configurations with the simulated movement in the one or more virtual environments;
determining disparity data of the simulated hardware configuration compared with a ground truth for the simulated hardware configuration;
based on the disparity data, running subsequent testing of the one or more localization algorithms against a different synthetic scene, motion, or hardware configuration; and
modifying at least one of the localization algorithms based on the subsequent testing.

### Claim 16 (depends on 15)

The one or more computer storage devices of claim 15 wherein the operations further comprise:
comparing localization algorithm output data with ground truth data.

### Claim 17 (depends on 15)

The one or more computer storage devices of claim 15, wherein the synthetic experiment data comprises synthetic images.

### Claim 18 (depends on 15)

The one or more computer storage devices of claim 15, wherein the synthetic experiment data comprises inertial measurement unit (IMU) data.

### Claim 19 (depends on 18)

The one or more computer storage devices of claim 18, wherein the IMU data comprises accelerometer data.

### Claim 20 (depends on 18)

The one or more computer storage devices of claim 18, wherein the IMU data comprises gyroscope data.
