# US 2019/0378340 A1 · Placing and solving constraints on a 3d environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0378340 A1
- Document ID: US20190378340A1
- Status: Published application
- Kind: A1
- Application number: 16/003,985
- Filed: 2018-06-08
- Published: 2019-12-12
- Earliest claimed: 2018-06-08
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Alvin Wen Jie CHIA; Michael J. EBSTYNE; Larry Marvin WALL; Trebor L. CONNELL
- Web page: https://patentgrep.com/patent/US20190378340A1

## CPC

- G06F 9/45512: {Command shells}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 16/583: using metadata automatically derived from the content
- G06F 30/00: Computer-aided design [CAD]
- G06F 30/13: Architectural design, e.g. computer-aided architectural design [CAAD] related to design of buildings, bridges, landscapes, production plants or roads
- G06F 40/205: Parsing
- G06F 8/427: {Parsing}
- G06F 2111/20: Configuration CAD, e.g. designing by assembling or positioning modules selected from libraries of predesigned modules
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 2219/2004: Aligning objects, relative positioning of parts
- G06T 2219/2016: Rotation, translation, scaling
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems and methods are disclosed for permitting the use of a natural language expression to specify object (or asset) locations in a virtual three-dimensional (3D) environment. By rapidly identifying and solving constraints for 3D object placement and orientation, consumers of synthetics services may more efficiently generate experiments for use in development of artificial intelligence (AI) algorithms and sensor platforms. Parsing descriptive location specifications, sampling the volumetric space, and solving pose constraints for location and orientation, can produce large numbers of designated coordinates for object locations in virtual environments with reduced demands on user involvement. Converting from location designations that are natural to humans, such as “standing on the floor one meter from a wall, facing the center of the room” to a six-dimensional (6D) pose specification (including 3-D location and orientation) can alleviate the need for a manual drag/drop/reorient procedure for placement of objects in a synthetic environment.

## Claims

### Claim 1

A system for placing and solving constraints on a 3D environment, the system comprising:
a parser for isolating input placement constraints into translational constraints and rotational constraints;
a spatial environment interface;
a translational constraint interpreter (TCI), in communication with the spatial environment interface, for determining and sampling available volumetric space in a synthetic environment to determine a set of points, in accordance with the isolated translational constraints; and
a rotational constraint interpreter (RCI), in communication with the spatial environment interface, for determining a set of poses that satisfy the isolated rotational constraints, from the set of points, for assets to be inserted into the synthetic environment.

### Claim 2 (depends on 1)

The system of claim 1 wherein the translational constraints comprise a distance from a reference.

### Claim 3 (depends on 1)

The system of claim 1 wherein the translational constraints comprise a spatial relationship relative to a reference.

### Claim 4 (depends on 1)

The system of claim 1 wherein the translational constraints comprise an implicit constraint preventing two assets conflicting for a common volume.

### Claim 5 (depends on 1)

The system of claim 1 wherein the rotational constraints comprise at least one selected from the list consisting of:
direction and facing a reference.

### Claim 6 (depends on 1)

The system of claim 1 further comprising:
a converter for converting the input placement constraints into a spatial query to solve, wherein the parser isolates input placement constraints into translational constraints and rotational constraints from the spatial query.

### Claim 7 (depends on 6)

The system of claim 6 wherein the input is obtained through at least one selected from the list consisting of:
text, speech, and an application programming interface (API).

### Claim 8

A method for placing and solving constraints on a 3D environment, the method comprising:
obtaining placement constraints;
parsing the obtained constraints to isolate translational constraints and rotational constraints;
determining and sampling available volumetric space in a synthetic environment to determine a set of points, in accordance with the translational constraints; and
determining a set of poses that satisfy the rotational constraints, from the set of points, for assets to be inserted into the synthetic environment.

### Claim 9 (depends on 8)

The method of claim 8 wherein the translational constraints comprise a distance from a reference.

### Claim 10 (depends on 8)

The method of claim 8 wherein the translational constraints comprise a spatial relationship relative to a reference.

### Claim 11 (depends on 8)

The method of claim 8 wherein the translational constraints comprise an implicit constraint preventing two assets conflicting for a common volume.

### Claim 12 (depends on 8)

The method of claim 8 wherein the rotational constraints comprise at least one selected from the list consisting of:
direction and facing a reference.

### Claim 13 (depends on 8)

The method of claim 8 further comprising:
converting input comprising the obtained constraints, into a spatial query to solve, wherein the parser isolates translational constraints and rotational constraints from the spatial query.

### Claim 14 (depends on 13)

The method of claim 13 wherein the input is obtained through at least one selected from the list consisting of:
text, speech, and an application programming interface (API).

### Claim 15

One or more computer storage devices having computer-executable instructions stored thereon for placing and solving constraints on a 3D environment, which, on execution by a computer, cause the computer to perform operations, the instructions comprising:
a parser component for isolating input placement constraints into translational constraints and rotational constraints;
a spatial environment interface component;
a translational constraint interpreter (TCI) component, in communication with the spatial environment interface component, for determining and sampling available volumetric space in a synthetic environment to determine a set of points, in accordance with the translational constraints; and
a rotational constraint interpreter (RCI) component, in communication with the spatial environment interface component, for determining a set of poses that satisfy the rotational constraints, from the set of points, for assets to be inserted into the synthetic environment.

### Claim 16 (depends on 15)

The one or more computer storage devices of claim 15 wherein the translational constraints comprise a distance from a reference.

### Claim 17 (depends on 15)

The one or more computer storage devices of claim 15 wherein the translational constraints comprise a spatial relationship relative to a reference.

### Claim 18 (depends on 15)

The one or more computer storage devices of claim 15 wherein the translational constraints comprise an implicit constraint preventing two assets conflicting for a common volume.

### Claim 19 (depends on 15)

The one or more computer storage devices of claim 15 wherein the rotational constraints comprise at least one selected from the list consisting of:
direction and facing a reference.

### Claim 20 (depends on 15)

The one or more computer storage devices of claim 15 further comprising:
a converter component for converting the input placement constraints into a spatial query to solve, wherein the parser isolates input placement constraints into translational constraints and rotational constraints from the spatial query, and
wherein the input is obtained through at least one selected from the list consisting of:
text, speech, and an application programming interface (API).