# US 2023/0409108 A1 · Converting a 2d positional input into a 3d point in space

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0409108 A1
- Document ID: US20230409108A1
- Status: Published application
- Kind: A1
- Application number: 18/459,811
- Filed: 2023-09-01
- Published: 2023-12-21
- Earliest claimed: 2020-01-24
- Assignees: Magic Leap, Inc.
- Applicants: Magic Leap, Inc.
- Inventors: Marc Alan McCall
- Web page: https://patentgrep.com/patent/US20230409108A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/014: {Hand-worn input/output arrangements, e.g. data gloves}
- G06F 3/02: Input arrangements using manually operated switches, e.g. using keyboards or dials
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/0414: {using force sensing means to determine a position}
- G06F 3/0416: {Control or interface arrangements specially adapted for digitisers}
- G06F 3/04812: Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 2203/04101: 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
- G06T 2219/004: Annotating, labelling
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A user may interact and select positions in three-dimensional space arbitrarily through conversion of a two-dimensional positional input into a three-dimensional point in space. The system may allow a user to use one or more user input devices for pointing, annotating, or drawing on virtual objects, real objects or empty space in reference to the location of the three-dimensional point in space within an augmented reality or mixed reality session.

## Claims

### Claim 1

A computerized method, performed by a computing system having one or more hardware computer processors and one or more non-transitory computer readable storage device storing software instructions executable by the computing system to perform the computerized method comprising:
determining a head pose of a user associated with a wearable display device;
determining a cursor screen position at a first distance from the wearable display device;
determining a cursor plane position at a second distance from the wearable display device;
determining a two-dimensional position on the cursor screen;
determining a one-dimensional position defined by at least one of the cursor plane or a mesh; and
displaying a virtual object to the user at a three-dimensional position comprised of the two-dimensional position and the one-dimensional position.

### Claim 2 (depends on 1)

The computerized method of claim 1, wherein the head pose of the user determines a view frustum.

### Claim 3 (depends on 2)

The computerized method of claim 2, wherein the cursor screen is positioned closer to the wearable display device than to the cursor plane.

### Claim 4 (depends on 3)

The computerized method of claim 3, wherein the cursor screen bisects the view frustum.

### Claim 5 (depends on 1)

The computerized method of claim 1, wherein the cursor plane is configured to have a maximum and minimum distance from the user.

### Claim 6 (depends on 1)

The computerized method of claim 1, wherein the two-dimensional position on the cursor screen is a default two-dimensional position.

### Claim 7 (depends on 6)

The computerized method of claim 6, wherein the default two-dimensional position is a center of the cursor screen.

### Claim 8 (depends on 1)

The computerized method of claim 1, wherein the one-dimensional position is further defined by a first intersection of a ray cast orthogonal from a head pose origin associated with a position and a rotation.

### Claim 9 (depends on 8)

The computerized method of claim 8, wherein the first intersection is with at least one of the cursor plane or the mesh.

### Claim 10 (depends on 9)

The computerized method of claim 9, wherein the mesh comprises one or more real world, virtual, or mixed objects.

### Claim 11 (depends on 1)

The computerized method of claim 1, wherein the virtual object is a cursor.

### Claim 12 (depends on 1)

The computerized method of claim 1, wherein the virtual object is a point or a line.

### Claim 13

A system for converting a two-dimensional positional input into a three-dimensional position for visualizing augmented reality, the system comprising:
a wearable headset;
one or more non-transitory computer readable storage mediums having program instructions embodied therewith; and
one or more processors configured to execute the program instructions to cause the system to:
determine a three-dimensional position within an augmented reality environment that includes one or more virtual objects and one or more real-world objects;
determine a two-dimensional position on a cursor screen based on the first two-dimensional offset and the second two-dimensional offset;
determine a one-dimensional position on a cursor plane or a mesh associated with one of the virtual objects or real-world objects;
identify an intersection of the determined one-dimensional position within the augmented reality environment where an axis orthogonal from a head pose origin associated with a position and a rotation, through the determined two-dimensional position on the cursor screen, and intersecting the cursor plane or the mesh associated with one of the virtual objects or real-world objects;
determine an updated three-dimensional position within the augmented reality environment; and
display a virtual object at the updated three-dimensional position within the augmented reality environment to a user via the wearable headset.

### Claim 14 (depends on 13)

The computerized method of claim 13, wherein at least one of the cursor plane or the mesh is fixed at the updated three-dimensional position.

### Claim 15 (depends on 14)

The computerized method of claim 14, further comprising displaying annotations at the updated three-dimensional position based on user interactions with the input device.

### Claim 16 (depends on 14)

The computerized method of claim 14, wherein the user draws on at least one of the cursor plane or the mesh.

### Claim 17 (depends on 13)

The computerized method of claim 13, further comprising:
determining a head movement compensation based on a change in at least one of an orientation and a position from a previous head pose and the head pose.

### Claim 18 (depends on 17)

The computerized method of claim 17, wherein the determined head movement compensation is a two-dimensional offset.

### Claim 19 (depends on 18)

The computerized method of claim 18, wherein the three-dimensional position is back projected on to the two-dimensional cursor screen by using the determined head movement compensation offset.

### Claim 20 (depends on 13)

The computerized method of claim 13, further comprising:
determining an input device movement based on a change in a previous two-dimensional position and the two-dimensional position, based at least on a detected interaction between a contactor operated by the user and a touch sensitive surface of an input device.