# US 2025/0155968 A1 · Tracking and drift correction

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0155968 A1
- Document ID: US20250155968A1
- Status: Published application
- Kind: A1
- Application number: 19/024,397
- Filed: 2025-01-16
- Published: 2025-05-15
- Earliest claimed: 2018-09-14
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Mohamed Selim Ben Himane
- Web page: https://patentgrep.com/patent/US20250155968A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/74: {involving reference images or patches}
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/30204: Marker
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Some implementations provide improved user interfaces for interacting with a virtual environment. The virtual environment is presented by a display of a first device having an image sensor. The first device uses the image sensor to determine a relative position and orientation of a second device based on a marker displayed on a display of the second device. The first device uses the determined relative position of the second device to display a representation of the second device including virtual content in place of the marker.

## Claims

### Claim 1

A method comprising:
at a first device comprising a processor and an image sensor:
obtaining image data comprising an image of a physical environment using the image sensor, the image comprising a marker displayed via the second device, wherein the marker is adapted over time based on a circumstance;
determining a relative position and orientation of the second device to the first device based on the marker in the image data; and
generating a control signal based on the relative position and orientation of the second device.

### Claim 2 (depends on 1)

The method of claim 1, wherein the circumstance comprises detecting an obstruction.

### Claim 3 (depends on 1)

The method of claim 1, wherein the circumstance comprises detecting an obstruction between the image sensor and the second device.

### Claim 4 (depends on 1)

The method of claim 1, wherein the circumstance comprises detecting a touch event.

### Claim 5 (depends on 1)

The method of claim 1, wherein the circumstance comprises a finger blocking a portion of a display of the second device.

### Claim 6 (depends on 1)

The method of claim 1, wherein the circumstance comprises a relative motion, position, or rotation of the first device and second device.

### Claim 7 (depends on 1)

The method of claim 1, wherein the circumstance comprises an attribute of the physical environment.

### Claim 8 (depends on 1)

The method of claim 1, wherein the circumstance comprises a lighting condition of the physical environment.

### Claim 9 (depends on 1)

The method of claim 1, wherein the marker comprises a pattern.

### Claim 10 (depends on 9)

The method of claim 9, wherein the relative position and orientation of the second device to the first device is determined based on the relative distances between elements of the pattern produced by the LEDs on the second device.

### Claim 11 (depends on 9)

The method of claim 9, wherein the pattern is a spatial pattern or a binary pattern that encodes data associated with motion of the second device.

### Claim 12 (depends on 11)

The method of claim 11, wherein the motion is determined via an accelerometer or inertial measurement unit on the second device.

### Claim 13 (depends on 1)

The method of claim 1, wherein the relative position and orientation of the second device to the first device is determined based on a single image of the image data.

### Claim 14 (depends on 1)

The method of claim 1, wherein the relative position and orientation of the second device to the first device is determined based on multiple images of the image data.

### Claim 15 (depends on 1)

The method of claim 1, wherein the first device uses the position and orientation of the second device to enable the second device to be used as a three-dimensional (3D) controller, a 3D pointer, or a user interface input device.

### Claim 16 (depends on 1)

The method of claim 1, wherein the generated control signal modifies a user interface element displayed by the first device.

### Claim 17 (depends on 1)

The method of claim 1, wherein the first device is a head-mounted device (HMD) and the second device comprises a hand-held device.

### Claim 18 (depends on 1)

The method of claim 1 further comprising adjusting the determined relative position and orientation of the second device to the first device over time based on motion tracking on the first device and the second device over time, wherein the motion tracking comprises obtaining additional images of the physical environment using the image sensor, the additional images comprising the marker.

### Claim 19

A first device comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the first device to perform operations comprising:
obtaining image data comprising an image of a physical environment using the image sensor, the image comprising a marker displayed via the second device, wherein the marker is adapted over time based on a circumstance;
determining a relative position and orientation of the second device to the first device based on the marker in the image data; and
generating a control signal based on the relative position and orientation of the second device.

### Claim 20

A non-transitory computer-readable storage medium storing program instructions executable via one or more processors to perform operations comprising:
obtaining image data comprising an image of a physical environment using the image sensor, the image comprising a marker displayed via the second device, wherein the marker is adapted over time based on a circumstance;
determining a relative position and orientation of the second device to the first device based on the marker in the image data; and
generating a control signal based on the relative position and orientation of the second device.
