# US 12,112,574 B2 · Systems and methods for virtual and augmented reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,574 B2
- Document ID: US12112574B2
- Status: Granted
- Kind: B2
- Application number: 17/211,502
- Filed: 2021-03-24
- Published: 2021-09-30
- Granted: 2024-10-08
- Earliest claimed: 2020-03-24
- Assignees: Magic Leap, Inc.
- Applicants: Magic Leap, Inc.
- Inventors: Robert David Berkebile
- Web page: https://patentgrep.com/patent/US12112574B2

## CPC

- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 27/0172: {characterised by optical features}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/03547: {Touch pads, in which fingers can move on a surface}
- G06F 3/04812: Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 10/235: {based on user input or interaction}
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06V 20/20: in augmented reality scenes
- G06V 40/18: Eye characteristics, e.g. of the iris
- G01C 21/206: {specially adapted for indoor navigation}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)

## Abstract

An apparatus for providing a virtual or augmented reality experience, includes: a screen, wherein the screen is at least partially transparent for allowing a user of the apparatus to view an object in an environment surrounding the user; a surface detector configured to detect a surface of the object; an object identifier configured to obtain an orientation and/or an elevation of the surface of the object, and to make an identification for the object based on the orientation and/or the elevation of the surface of the object; and a graphic generator configured to generate an identifier indicating the identification for the object for display by the screen, wherein the screen is configured to display the identifier.

## Claims

### Claim 1

An apparatus for providing a virtual or augmented reality experience, comprising:
a screen, wherein the screen is at least partially transparent for allowing a user of the apparatus to view an object in an environment surrounding the user;
a surface detector configured to detect a surface of the object;
an object identifier configured to (1) obtain an orientation of the surface of the object and/or an elevation of the surface of the object, and (2) after the orientation and/or the elevation is obtained, determine whether the object is a wall, a floor, or a furniture based on the orientation of the surface of the object and/or the elevation of the surface of the object, wherein when the object identifier obtains the orientation and/or the elevation, an identity of the object with which the orientation and/or the elevation is associated is unknown to the object identifier; and
a graphic generator configured to generate an identifier for the object for display by the screen, the identifier indicating that the object is the wall, the floor, or the furniture, and wherein a transparent portion of the screen is configured to display the identifier.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the object identifier is configured to determine the object as the wall if the orientation of the surface of the object is substantially vertical, and if the elevation of the surface of the object is above an elevation threshold.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein the object identifier is configured to determine the object as the floor if the orientation of the surface of the object is substantially horizontal, and if the elevation of the surface of the object is below an elevation threshold.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein the object is configured to determine the object as the furniture if the orientation of the surface of the object is substantially horizontal, and if the elevation of the surface of the object is above a first elevation threshold and is below a second elevation threshold.

### Claim 5 (depends on 1)

The apparatus of claim 1, wherein the screen is configured to display the identifier at a location in the screen, such that when the user views the object in the environment through the screen, the identifier will be in a spatial relationship with respect to the object.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein the object identifier is configured to obtain an input indicating a selection of the object for which the identification of the object is to be determined.

### Claim 7 (depends on 6)

The apparatus of claim 6, wherein the input comprises a user input generated via a controller component, the user input indicating the selection of the object.

### Claim 8 (depends on 7)

The apparatus of claim 7, wherein the user input indicates a cursor position in the screen, and wherein the object identifier is configured to determine the object in the environment being selected based on the cursor position.

### Claim 9 (depends on 7)

The apparatus of claim 7, wherein the user input indicates an orientation of the controller component, and wherein the object identifier is configured to determine the object in the environment being selected based on a direction of pointing by the controller component towards the object in the environment.

### Claim 10 (depends on 1)

The apparatus of claim 1, further comprising a camera, wherein the object identifier is configured to select the object for identification based on a presence of an image of the object in a camera image provided by the camera.

### Claim 11 (depends on 10)

The apparatus of claim 10, wherein the object identifier is configured to select the object automatically without requiring the user to select the object for identification.

### Claim 12 (depends on 10)

The apparatus of claim 10, wherein the object identifier is configured to select the object in response to the object being presence in a sequence of camera images that comprise the camera image within a duration exceeding a time threshold.

### Claim 13 (depends on 1)

The apparatus of claim 1, further comprising a space definer configured to define a virtual space.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein the space definer is configured to define a virtual wall for the virtual space.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein the virtual wall is offset from a real physical wall in the environment surrounding the user, the real physical wall being the wall or another wall.

### Claim 16 (depends on 14)

The apparatus of claim 14, wherein the virtual wall is aligned with, or intersects, a real physical wall in the environment surrounding the user, the real physical wall being the wall or another wall.

### Claim 17 (depends on 14)

The apparatus of claim 14, wherein the screen is configured to display a wall identifier at a location in the screen, such that when the user views the virtual wall, the wall identifier will be in a spatial relationship with respect to the virtual wall.

### Claim 18 (depends on 13)

The apparatus of claim 13, wherein the space definer is configured to define a plurality of virtual walls for the virtual space; and
wherein the screen is configured to display wall identifiers for the respective virtual walls.

### Claim 19 (depends on 13)

The apparatus of claim 13, wherein the space definer is configured to define a corner for the virtual space.

### Claim 20 (depends on 13)

The apparatus of claim 13, wherein the space definer is configured to define a wall edge for the virtual space.

### Claim 21 (depends on 13)

The apparatus of claim 13, wherein the space definer is configured to obtain a user input generated via a controller component, the user input indicating a selection of a feature in the environment for defining at least a part of the virtual space.

### Claim 22 (depends on 21)

The apparatus of claim 21, wherein the feature in the environment comprises a physical wall, a wall corner, an edge, or any combination of the foregoing, wherein the physical wall is the wall or another wall.

### Claim 23 (depends on 21)

The apparatus of claim 21, wherein the user input indicates a cursor position in the screen.

### Claim 24 (depends on 21)

The apparatus of claim 21, wherein the user input indicates an orientation of the controller component, and wherein the selection of the feature in the environment is based on a direction of pointing by the controller component towards the feature in the environment.

### Claim 25 (depends on 13)

The apparatus of claim 13, further comprising a camera, wherein the apparatus is configured to select a feature in the environment, for defining at least a part of the virtual space, based on a presence of an image of the feature in a camera image provided by the camera.

### Claim 26 (depends on 25)

The apparatus of claim 25, wherein the apparatus is configured to select the feature in the environment automatically.

### Claim 27 (depends on 25)

The apparatus of claim 25, wherein the apparatus is configured to select the feature in response to the feature being presence in a sequence of camera images that includes the camera image within a duration exceeding a time threshold.

### Claim 28 (depends on 1)

The apparatus of claim 1, wherein the screen is configured to display a pointer based on an orientation of a controller component, and wherein the apparatus further comprises a pointer configurator, wherein the pointer configurator is configured to adjust a configuration of the pointer displayed in the screen, and/or to present an interaction graphic in association with the pointer.

### Claim 29 (depends on 28)

The apparatus of claim 28, wherein the pointer configurator is configured to adjust the configuration of the pointer and/or to present the interaction graphic in response to an interaction between the pointer and a virtual object.

### Claim 30 (depends on 1)

The apparatus of claim 1, wherein the graphic generator is also configured to provide a virtual content for interaction by the user and an additional user, and wherein the screen is configured to display the virtual content.

### Claim 31 (depends on 30)

The apparatus of claim 30, wherein the apparatus is configured to connect the user and the additional user to a same virtual space so that the user and the additional user can interact with the virtual content at the same virtual space.

### Claim 32 (depends on 30)

The apparatus of claim 30, wherein the graphic generator is configured to provide the virtual content for interaction by the user and the additional user in different respective rooms.

### Claim 33 (depends on 30)

The apparatus of claim 30, wherein the graphic generator is configured to provide the virtual content for interaction by the user and the additional user in a same room.

### Claim 34 (depends on 30)

The apparatus of claim 30, wherein the apparatus is configured to register the user and the additional user to the environment so that the user and the additional user can interact with the virtual content while they are in the environment.

### Claim 35 (depends on 34)

The apparatus of claim 34, wherein the apparatus is configured to register the user and the additional user to the environment by determining one or more anchor points that are associated with a location of the user and a location of the additional user.

### Claim 36 (depends on 1)

The apparatus of claim 1, wherein the object identifier is configured to obtain the orientation and/or the elevation of the surface of the object, and to determine whether the object is the wall, the floor, or the furniture, based on the orientation and/or the elevation of the surface of the object, while the object in the environment is viewable by the user through the screen.

### Claim 37 (depends on 1)

The apparatus of claim 1, wherein the object identifier is configured to determine the object as the wall if the orientation of the surface of the object is substantially vertical, and if the elevation of the surface of the object is above a first elevation threshold;
wherein the object identifier is configured to determine the object as the floor if the orientation of the surface of the object is substantially horizontal, and if the elevation of the surface of the object is below a second elevation threshold; and
wherein the object is configured to determine the object as the furniture if the orientation of the surface of the object is substantially horizontal, and if the elevation of the surface of the object is above a third elevation threshold and is below a fourth elevation threshold.

### Claim 38

A method performed by an apparatus that is configured to provide a virtual or augmented reality experience, the method comprising:
detecting, by a surface detector, a surface of an object in an environment surrounding a user of the apparatus;
obtaining, by an object identifier, an orientation of the surface of the object and/or an elevation of the surface of the object;
after the orientation and/or the elevation is obtained, determining, by the object identifier, whether the object is a wall, a floor, or a furniture, based on the orientation of the surface of the object and/or the elevation of the surface of the object, wherein when the act of obtaining the orientation and/or the elevation is being performed, an identity of the object with which the orientation and/or the elevation is associated is unknown to the object identifier; and
generating, by a graphic generator, an identifier for the object for display by a transparent portion of a screen, the identifier indicating that the object is the wall, the floor, or the furniture, wherein the screen is at least partially transparent for allowing the user of the apparatus to view the object in the environment surrounding the user.

### Claim 39 (depends on 38)

The method of claim 38, wherein the act of determining whether the object is the wall, the floor, or the furniture comprises:
determining the object as the wall if the orientation of the surface of the object is substantially vertical, and if the elevation of the surface of the object is above a first elevation threshold;
determining the object as the floor if the orientation of the surface of the object is substantially horizontal, and if the elevation of the surface of the object is below a second elevation threshold; and
determining the object as the furniture if the orientation of the surface of the object is substantially horizontal, and if the elevation of the surface of the object is above a third elevation threshold and is below a fourth elevation threshold.