# US 12,008,151 B2 · Tracking and drift correction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,151 B2 - Document ID: US12008151B2 - Status: Granted - Kind: B2 - Application number: 17/314,130 - Filed: 2021-05-07 - Published: 2021-08-26 - Granted: 2024-06-11 - Earliest claimed: 2018-09-14 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Mohamed Selim Ben Himane - Web page: https://patentgrep.com/patent/US12008151B2 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/74: {involving reference images or patches} - G06T 2200/24: involving graphical user interfaces [GUIs] - G06T 2207/30204: Marker - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Some implementations provide improved user interfaces for interacting with a virtual environment. The virtual environment is presented by a display of a first device having an image sensor. The first device uses the image sensor to determine a relative position and orientation of a second device based on a marker displayed on a display of the second device. The first device uses the determined relative position of the second device to display a representation of the second device including virtual content in place of the marker. ## Claims ### Claim 1 A method comprising: at a first device comprising a processor, a computer-readable storage medium, and a first display: receiving data corresponding to a tracked position or orientation of a second device from the second device, wherein the tracked position and orientation of the second device are determined based at least in part on image data from an image sensor of the second device, wherein the tracked position and orientation of the second device is based on identifying a stationary reference point of the physical environment, wherein the tracked position and orientation of the second device is tracked by: identifying portions of multiple images in the image data, the multiple images corresponding to multiple times during a time period, the portions of the multiple images corresponding to the stationary reference point of the physical environment; determining changes, during the time period, in positioning of the portions of the multiple images corresponding to the stationary reference point; and tracking the position and orientation of the second device during the time period based on the changes in the positioning of the portions of the multiple images corresponding to the stationary reference point; determining a relative position and orientation of the second device to the first device based on the received data; and generating a control signal based on the relative position and orientation of the second device. ### Claim 2 (depends on 1) The method of claim 1, wherein the generated control signal is based on input on the second device, wherein the first device uses the relative position and orientation of the second device to enable the second device to be used as a three-dimensional (3D) controller, a 3D pointer, or a user interface input device. ### Claim 3 (depends on 1) The method of claim 1, wherein the generated control signal modifies a user interface element displayed by the first device. ### Claim 4 (depends on 1) The method of claim 1, wherein the first device is a head-mounted-device and the second device comprises a touch screen. ### Claim 5 (depends on 1) The method of claim 1 further comprising displaying real-world content on the first display, wherein the real-world content includes a representation of the physical environment. ### Claim 6 (depends on 1) The method of claim 1 further comprising adjusting the determined relative position and orientation of the second device to the first device over time. ### Claim 7 (depends on 1) The method of claim 1, wherein the position and orientation of the second device is determined via simultaneous localization and mapping. ### Claim 8 (depends on 1) The method of claim 1 further comprising: detecting that an estimated error associated with the relative position and orientation of the second device to the first device is greater than a threshold; and in accordance with detecting that the estimated error is greater than the threshold: adjusting the determined relative position and orientation of the second device to the first device over time based on additional data. ### Claim 9 (depends on 8) The method of claim 8, wherein the virtual content comprises a control corresponding to an interaction. ### Claim 10 (depends on 1) The method of claim 1 further comprising: displaying a representation of the second device on the first display based on the relative position and orientation of the second device, the representation comprising virtual content. ### Claim 11 (depends on 10) The method of claim 10 further comprising: obtaining data indicative of a touch event on the second device corresponding to the control; and initiating the interaction in response to the touch event. ### Claim 12 (depends on 1) The method of claim 1, wherein at least some of the image data used to determine the position and orientation of the second device comprises multiple images depicting a fixed reference point in a physical environment. ### Claim 13 (depends on 12) The method of claim 12, wherein the position and orientation of the second device is determined based on determining a motion of the second device based on tracking the fixed reference point in the multiple images. ### Claim 14 (depends on 1) The method of claim 1, wherein the data used to determine the position and orientation of the second device is from a single sensor on the second device. ### Claim 15 A first device comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the first device to perform operations comprising: receiving data corresponding to a tracked position or orientation of a second device from the second device, wherein the tracked position and orientation of the second device are determined based at least in part on image data from an image sensor of the second device, wherein the tracked position and orientation of the second device is based on identifying a stationary reference point of the physical environment, wherein the tracked position and orientation of the second device is tracked by: identifying portions of multiple images in the image data, the multiple images corresponding to multiple times during a time period, the portions of the multiple images corresponding to the stationary reference point of the physical environment; determining changes, during the time period, in positioning of the portions of the multiple images corresponding to the stationary reference point; and tracking the position and orientation of the second device during the time period based on the changes in the positioning of the portions of the multiple images corresponding to the stationary reference point; determining a relative position and orientation of the second device to the first device based on the received data; and generating a control signal based on the relative position and orientation of the second device. ### Claim 16 (depends on 15) The first device of claim 15, wherein the generated control signal is based on input on the second device, wherein the operations further comprise using the relative position and orientation of the second device to enable the second device to be used as a three-dimensional (3D) controller, a 3D pointer, or a user interface input device. ### Claim 17 (depends on 15) The first device of claim 15, wherein the generated control signal modifies a user interface element displayed by the first device. ### Claim 18 (depends on 15) The first device of claim 15, wherein the first device is a head-mounted-device and the second device comprises a touch screen. ### Claim 19 (depends on 15) The first device of claim 15, wherein the position and orientation of the second device is determined via simultaneous localization and mapping. ### Claim 20 A non-transitory computer-readable storage medium, storing program instructions executable by one or more processors of a first device to perform operations comprising: receiving data corresponding to a tracked position or orientation of a second device from the second device, wherein the tracked position and orientation of the second device are determined based at least in part on image data from an image sensor of the second device, wherein the tracked position and orientation of the second device is based on identifying a stationary reference point of the physical environment, wherein the tracked position and orientation of the second device is tracked by: identifying portions of multiple images in the image data, the multiple images corresponding to multiple times during a time period, the portions of the multiple images corresponding to the stationary reference point of the physical environment; determining changes, during the time period, in positioning of the portions of the multiple images corresponding to the stationary reference point; and tracking the position and orientation of the second device during the time period based on the changes in the positioning of the portions of the multiple images corresponding to the stationary reference point; determining a relative position and orientation of the second device to the first device based on the received data; and generating a control signal based on the relative position and orientation of the second device. ### Claim 21 (depends on 20) The non-transitory computer-readable storage medium of claim 20, wherein the generated control signal is based on input on the second device, wherein the operations further comprise using the relative position and orientation of the second device to enable the second device to be used as a three-dimensional (3D) controller, a 3D pointer, or a user interface input device.