# US 2019/0172266 A1 · Rendering virtual objects in 3d environments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0172266 A1
- Document ID: US20190172266A1
- Status: Published application
- Kind: A1
- Application number: 16/200,245
- Filed: 2018-11-26
- Published: 2019-06-06
- Earliest claimed: 2017-02-01
- Assignees: Accenture Global Solutions Limited
- Applicants: Accenture Global Solutions Limited
- Inventors: Matthew Thomas Short; Sunny Webb; Joshua Opel; Theo E. Christensen
- Web page: https://patentgrep.com/patent/US20190172266A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G02B 27/017: {Head mounted}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06T 7/73: using feature-based methods
- G06V 20/20: in augmented reality scenes
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0141: {characterised by the informative content of the display}
- G06T 2207/30204: Marker
- G06T 2210/62: Semi-transparency
- G06T 2219/004: Annotating, labelling
- G06T 2219/024: Multi-user, collaborative environment
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems, methods, devices, and other techniques for placing and rendering virtual objects in three-dimensional environments. The techniques include providing, by a device, a view of an environment of a first user. A first computing system associated with the first user receives an instruction to display, within the view of the environment of the first user, a virtual marker at a specified position of the environment of the first user, the specified position derived from a second user's interaction with a three-dimensional (3D) model of at least a portion of the environment of the first user. The device displays, within the view of the environment of the first user, the virtual marker at the specified position of the environment of the first user.

## Claims

### Claim 1

A computer-implemented method, comprising:
providing, by a device, a view of an environment of a first user;
receiving, by a first computing system associated with the first user, an instruction to display within the view of the environment of the first user a virtual marker at a specified position of the environment of the first user, the specified position derived from a second user's interaction with a three-dimensional (3D) model of at least a portion of the environment of the first user; and
displaying, by the device and within the view of the environment of the first user, the virtual marker at the specified position of the environment of the first user.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the instruction to display the virtual marker at the specified position of the environment of the first user comprises a coordinate value that identifies the position of the virtual marker in a 3D space.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the coordinate value identifies the position of the virtual marker in a 3D space of a second environment other than the environment of the first user; and
the method comprises mapping the coordinate from the 3D space of the second environment to the 3D space of the environment of the first user to determine a corresponding position of the virtual marker in the environment of the first user.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein:
the coordinate value identifies the position of the virtual marker in the 3D space of the second environment using a first 3D offset from one or more anchor points of the second environment; and
mapping the coordinate from the 3D space of the second environment to the 3D space of the environment of the first user comprises:
identifying one or more anchor points of the environment of the first user; and
applying, using the coordinate value, a second 3D offset from the one or more anchor points of the environment of the first user to identify the corresponding position of the virtual marker in the environment of the first user.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein displaying the virtual marker at the specified position of the environment of the first user comprises superimposing the virtual marker over the view of the environment of the first user so that the virtual marker visually appears as if it is located a distance from the first user at the specified position of the environment.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the second user's interaction with the 3D model of the at least the portion of the environment of the first user occurs while the second user is remotely located from the environment of the first user.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein:
the 3D model of the at least the portion of the environment of the first user comprises a 3D model of a target object located in the environment of the first user;
the second user's interaction with the 3D model of the at least the portion of the environment comprises placing the virtual marker at a particular location on a surface of the 3D model of the target object; and
displaying the virtual marker at the specified position of the environment of the first user comprises displaying the virtual marker at a corresponding location on a surface of the target object within the view of the environment of the first user.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, further comprising:
obtaining, from one or more spatial sensors coupled to the first computing system, spatial sensing data that indicates spatial characteristics of the environment of the first user; and
generating, based on the spatial sensing data, the 3D model of the at least the portion of the environment of the first user.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, further comprising:
analyzing the 3D model of the at least the portion of the environment of the first user to identify a virtual object in the 3D model of the at least the portion of the environment of the first user that represents a target object located in the environment of the first user; and
generating a 3D model of the target object by extracting the identified virtual object from the 3D model of the at least the portion of the environment of the first user,
wherein the second user specifies the position of the virtual marker by placing the virtual marker at a particular location in a virtual environment that includes the 3D model of the target object.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the device is a head-mounted display (HMD) device, wherein the HMD device displays the virtual marker at the specified position of the environment of the first user while the first user is wearing the HMD device.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, wherein providing the view of the environment of the first user comprises generating an augmented view of the environment that shows one or more virtual objects at fixed positions in a 3D space of the environment.

### Claim 12

A computer-implemented method, comprising:
receiving, by a computing system, data that represents a 3D model of at least a portion of an environment of a first user;
rendering, by the computing system for display to a second user, the 3D model of the at least the portion of the environment of the first user;
identifying, by the computing system, that a user input placed a virtual marker at a specified position of a virtual environment that includes the 3D model of the at least the portion of the environment of the first user; and
transmitting, by the computing system, an instruction for the virtual marker to be displayed within a view of the environment of the first user at a specified position of the environment of the first user that corresponds to the specified position of the virtual environment at which the user input placed the virtual marker.

### Claim 13 (depends on 12)

The computer-implemented method of claim 12, wherein rendering the 3D model of the at least the portion of the environment of the first user comprises augmenting a direct real-world view of a second environment of a second user with the 3D model of the at least the portion of the environment of the first user.

### Claim 14 (depends on 13)

The computer-implemented method of claim 13, comprising using a head-mounted display (HMD) device to augment the direct real-world view of the second environment of the second user with the 3D model of the at least the portion of the environment of the first user.

### Claim 15 (depends on 13)

The computer-implemented method of claim 13, comprising using a portable computing device or a 3D-display device to render the 3D model of the at least the portion of the environment of the first user.

### Claim 16 (depends on 1)

The computer-implemented method of claim 1, wherein:
the 3D model of the at least the portion of the environment of the first user includes a virtual object that represents a target object that is located in the environment of the first user;
generating a 3D model of the target object from the 3D model of the at least the portion of the environment of the first user; and
identifying that the user input placed the virtual marker at the specified position of the virtual environment includes identifying that the user input placed the virtual marker at a particular location on a surface of the 3D model of the target object.

### Claim 17 (depends on 16)

The computer-implemented method of claim 16, further comprising, after transmitting the instruction for the virtual marker to be displayed within the view of the environment of the first user:
receiving, by the computing system, data that identifies a position in the environment of the first user at which a physical implement has been brought into contact with or proximity of the target object; and
rendering, by the computing system and for display to the second user within the virtual environment, a virtual marker that represents the physical implement at a position relative to the 3D model of the target object that corresponds to the position in the environment of the first user at which the physical implement was brought into contact with or proximity of the target object.

### Claim 18 (depends on 17)

The computer-implemented method of claim 17, further comprising:
displaying the virtual marker that represents the physical implement concurrently with the placed virtual marker; and
receiving a second user input that indicates whether the physical implement was properly positioned with respect to the target object,
wherein in response to identifying that the second user input indicates that the physical implement was not properly positioned with respect to the target object, the computing system is configured to transmit a notification to a computing system associated with the first user.

### Claim 19 (depends on 12)

The computer-implemented method of claim 12, wherein the 3D model of the at least the portion of the environment of the first user comprises a 3D model of a target object located in the environment of the first user; and
the method further comprises:
receiving, by the computing system, sensor data that identifies values for one or more operational parameters of the target object; and
rendering, by the computing system and for display to the second user along with the 3D model of the target object, user interface elements representing the values for the one or more operational parameters of the target object.

### Claim 20

One or more non-transitory computer-readable media encoded with instructions that, when executed by one or more processors, cause performance of operations comprising:
providing, by a device, a view of an environment of a first user;
receiving, by a first computing system associated with the first user, an instruction to display within the view of the environment of the first user a virtual marker at a specified position of the environment of the first user, the specified position derived from a second user's interaction with a three-dimensional (3D) model of at least a portion of the environment of the first user; and
displaying, by the device and within the view of the environment of the first user, the virtual marker at the specified position of the environment of the first user.
