# US 12,761,480 B2 · Automatic placement of a virtual object in a three-dimensional space

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,761,480 B2
- Document ID: US12761480B2
- Status: Granted
- Kind: B2
- Application number: 18/473,017
- Filed: 2023-09-22
- Published: 2024-01-11
- Granted: 2026-10-13
- Earliest claimed: 2016-08-11
- Assignees: Magic Leap, Inc.
- Applicants: Magic Leap, Inc.
- Inventors: Paul Armistead Hoover; Jonathan Lawrence Mann
- Web page: https://patentgrep.com/patent/US12761480B2

## CPC

- G02B 27/0172: {characterised by optical features}
- G02B 27/017: {Head mounted}
- G02B 27/0189: {Sight systems}
- G02B 30/34: Stereoscopes providing a stereoscopic pair of separated images corresponding to parallactically displaced views of the same object, e.g. three-dimensional [3D] slide viewers
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- H04N 13/156: Mixing image signals
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 13/279: the virtual viewpoint locations being selected by the viewers or determined by tracking
- H04N 13/344: with head-mounted left-right displays
- H04N 13/395: with depth sampling, i.e. the volume being constructed from a stack or sequence of two-dimensional [2D] image planes
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0178: {Eyeglass type (eyeglass details G02C)}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- H04N 2213/001: Constructional or mechanical details
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Augmented reality systems and methods for automatically repositioning a virtual object with respect to a destination object in a three-dimensional (3D) environment of a user are disclosed. The systems and methods can automatically attach the target virtual object to the destination object and re-orient the target virtual object based on the affordances of the virtual object or the destination object. The systems and methods can also track the movement of a user and detach the virtual object from the destination object when the user's movement passes a threshold condition.

## Claims

### Claim 1

An augmented reality (AR) system comprising:
a hardware computer processor; and
a non-transitory computer readable medium having software instructions stored thereon, the software instructions executable by the hardware computer processor to cause the AR system to perform operations comprising:
render a virtual environment to a user of the AR system;
present a virtual menu of items to the user of the AR system;
generate a focus indicator comprising a graphical representation of a cursor, virtual cone, or virtual ray;
display the focus indicator at a location in the virtual environment of the user associated with a current pose of the user;
receive a selection of an item by the user from the virtual menu of items based on a positioning of the focus indicator by the user;
automatically place the selected item at a location in the virtual environment of the user based at least in part on:
affordances of the selected item and affordances of objects, associated with inanimate physical objects, in the virtual environment of the user,
wherein automatically placing the selected item at the location in the virtual environment of the user includes applying a virtual force to the selected item based on the affordances of the selected item and the affordances of the objects in the virtual environment of the user, and wherein the virtual force replicates at least one of a spring force, a gravitational force, an adhesive force, or an electromagnetic force and is at least partially dependent on a user preference associated with an orientation speed,
wherein the applying the virtual force to the selected item based on the affordances of the selected item and the affordances of the object in the virtual environment of the user includes applying a repulsive force when the affordances of the selected item and the affordances of the object in the virtual environment of the user indicate a functional incompatibility for use or attachment at the location, such that placing the selected item at the location would prevent an intended use of the selected item or the object and
wherein, upon determining that the placing of the selected item at the location is suitable, the automatically placing the selected item at the location in the virtual environment of the user further includes:
calculating a distance between an initial location of the selected item from the virtual menu of items and the location,
determining whether the distance between distance is less than a first threshold distance, and rendering, when the distance is less than the first threshold distance, an orientation of the selected item to align one or more objects at the location by automatically orienting the selected item to cause a surface normal of the selected item to face the AR system, the surface normal being defined by the affordances of the selected item;
and
automatically orienting the selected item to cause a surface normal of the selected item to be perpendicular to a surface of the location or parallel to a normal of the location.

### Claim 2 (depends on 1)

The AR system of claim 1, wherein the software instructions are further configured to, upon selection, copy the selected item from the virtual menu of items into a virtual clipboard.

### Claim 3 (depends on 1)

The AR system of claim 1, wherein the software instructions are further configured to determine the location based on at least one of: an actuation of a user input device or a pose of a user.

### Claim 4 (depends on 1)

The AR system of claim 1, wherein the software instructions are further configured to determine the location based on receiving an indication from the user to place the selected item at the location that includes: identifying, using a pose sensor, a change in a pose of the user; identifying the location based at least partly on the pose of the user; and
receiving a confirmation from the user to place the selected item at the location.

### Claim 5 (depends on 4)

The AR system of claim 4, wherein the confirmation comprises at least one of a change in the pose of the user or a hand gesture on a user input device.

### Claim 6 (depends on 1)

The AR system of claim 1, wherein the location is a physical object with an at least partially planar surface within the environment of the user, and wherein the applying the repulsive force occurs when the affordances of the selected item and the affordances of the object in the virtual environment of the user indicate that the placing of the selected item at the location is unsuitable due to the object in the virtual environment of the user already occupying the location.

### Claim 7 (depends on 6)

The AR system of claim 6, wherein the physical object comprises a wall or a table.

### Claim 8 (depends on 1)

The AR system of claim 1, wherein the location is a virtual object.

### Claim 9 (depends on 1)

The AR system of claim 1, wherein the affordances of the selected item are associated with one or more of the following: a function, an orientation, a type, a location, a shape, a size, or an environment of the selected item.

### Claim 10 (depends on 1)

The AR system of claim 1, wherein the affordances of the objects in the virtual environment are associated with one or more of the following: a function, an orientation, a type, a location, a shape, a size, or an environment of the selected item.

### Claim 11 (depends on 1)

The AR system of claim 1, wherein one or more physical attributes is assigned to the selected item, and the placement of the selected item is determined by simulating interactions of the selected item, the location, and the virtual environment based on the physical attributes of the selected item.

### Claim 12 (depends on 11)

The AR system of claim 11, wherein the one or more physical attributes assigned to the selected item comprises at least one of a mass, a size, a density, a phase, a hardness, an elasticity, or an electromagnetic attribute.

### Claim 13 (depends on 1)

The AR system of claim 1, wherein automatically placing the selected item at the location in the virtual environment of the user further includes applying a virtual force to the selected item further based at least in part on affordances of the virtual environment, wherein the virtual environment includes at least one of gravity or air resistance.

### Claim 14 (depends on 1)

The AR system of claim 1, wherein the virtual force on the selected item based on the affordances of the selected item defines at least one of an amount, speed, or acceleration of movement of the selected item to the location.

### Claim 15 (depends on 1)

The AR system of claim 1, wherein the repulsive force occurs independently of geometric overlap detection.

### Claim 16 (depends on 1)

The AR system of claim 1, wherein the applying the repulsive force comprises determining that the selected item and the object are associated with incompatible affordance classes, including at least one of: support affordances, attachment affordances, display affordances, containment affordances, or interaction affordances.

### Claim 17 (depends on 1)

The AR system of claim 1, wherein the virtual force is modulated in magnitude or rate based on a user preference associated with an orientation speed.

### Claim 18 (depends on 1)

The AR system of claim 1, wherein the automatically orienting the selected item is performed only after determining that the distance between the selected item and the location is less than a threshold distance.

### Claim 19 (depends on 1)

The AR system of claim 1, wherein the automatically orienting the selected item includes a first orientation adjustment prior to attachment to the location and a second orientation adjustment after attachment, each determined based on affordances.

### Claim 20 (depends on 1)

The AR system of claim 1, wherein automatically orienting the selected item includes detecting when a functional surface is facing away from the AR system and reversing an orientation axis in response.