# US 2026/0093345 A1 · Controller Creation using Object Scanning and IR Retroreflectors > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2026/0093345 A1 - Document ID: US20260093345A1 - Status: Published application - Kind: A1 - Application number: 19/319,293 - Filed: 2025-09-04 - Published: 2026-04-02 - Earliest claimed: 2024-09-27 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Feng Zheng; Chunjia Hu; Timo Birnschein; Waleed Abdulla - Web page: https://patentgrep.com/patent/US20260093345A1 ## CPC - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/0308: {comprising a plurality of distinctive and separately oriented light emitters or reflectors associated to the pointing device, e.g. remote cursor controller with distinct and separately oriented LEDs at the tip whose radiations are captured by a photo-detector associated to the screen} - G06F 3/0325: {using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image} - G06T 7/60: Analysis of geometric attributes - G06T 7/74: {involving reference images or patches} - G06T 7/75: {involving models} - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T 2207/10048: Infrared image - G06T 2207/30204: Marker - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A technique for creating a control from a three-dimensional object involves registering the object as a controller. Markers are placed on the object in a constellation pattern. The object is captured in image data with the markers. A geometry of the three-dimensional is determined from the image data, and a location of the markers relative to the geometry is determined. The three-dimensional object is registered as a controller by associating the geometry with the marker locations. The constellation is tracked in additional image data and motion characteristics for the controller are determined based on a determined orientation of the constellation. The orientation of the constellation is used to determine an orientation of the controller. The determined orientation of the controller is then used for a user input action. ## Claims ### Claim 1 A method comprising: capturing image data of a three-dimensional object; identifying a constellation of markers on the three-dimensional object in the image data; defining a spatial relationship between the constellation and a geometry of the three-dimensional object; capturing image data of the constellation; determining an orientation of the constellation from the image data of the constellation; and determining an orientation of the three-dimensional object based on the orientation of the constellation and the spatial relationship. ### Claim 2 (depends on 1) The method of claim 1, further comprising: generating a three-dimensional digital reconstruction of the three-dimensional object based on the orientation of the constellation. ### Claim 3 (depends on 1) The method of claim 1, further comprising: registering an input portion of the three-dimensional object in accordance with the constellation. ### Claim 4 (depends on 3) The method of claim 3, wherein identifying the constellation of markers comprises: prompting a user to touch the markers; and in response to the prompt, detecting a touch between a finger and the three-dimensional object. ### Claim 5 (depends on 4) The method of claim 4, further comprising: determining a location characteristic of the input portion of the three-dimensional object based on the touch and the registration; and performing a user input action based on the location characteristic of the input portion of the three-dimensional object. ### Claim 6 (depends on 1) The method of claim 1, wherein the constellation of markers comprises at least one selected from a group consisting of an IR illuminator and an IR reflector. ### Claim 7 (depends on 1) The method of claim 1, wherein defining the spatial relationship between the constellation and the geometry of the three-dimensional object comprises: registering three-dimensional object as a controller by associating the constellation with the geometry of the three-dimensional object. ### Claim 8 A non-transitory computer readable medium comprising computer readable code executable by one or more processors to: capture image data of a three-dimensional object; identify a constellation of markers on the three-dimensional object in the image data; define a spatial relationship between the constellation and a geometry of the three-dimensional object; capture image data of the constellation; determine an orientation of the constellation from the image data of the constellation; and determine an orientation of the three-dimensional object based on the orientation of the constellation and the spatial relationship. ### Claim 9 (depends on 8) The non-transitory computer readable medium of claim 8, further comprising computer readable code to: generating a three-dimensional digital reconstruction of the three-dimensional object based on the orientation of the constellation. ### Claim 10 (depends on 9) The non-transitory computer readable medium of claim 9, further comprising computer readable code to: register an input portion of the three-dimensional object in accordance with the constellation. ### Claim 11 (depends on 10) The non-transitory computer readable medium of claim 10, wherein the computer readable code to identify the constellation of markers comprises computer readable code to: prompt a user to touch the markers; and in response to the prompt, detect a touch between a finger and the three-dimensional object. ### Claim 12 (depends on 11) The non-transitory computer readable medium of claim 11, further comprising computer readable code to: determine a location characteristic of an input portion of the three-dimensional object based on the touch and the registration; and perform a user input action based on the location characteristic of the input portion of the three-dimensional object. ### Claim 13 (depends on 8) The non-transitory computer readable medium of claim 8, wherein the constellation of markers comprises at least one selected from a group consisting of an IR illuminator and an IR reflector. ### Claim 14 (depends on 8) The non-transitory computer readable medium of claim 8, wherein the image data comprises a plurality of images capturing the three-dimensional object from different views. ### Claim 15 A system comprising: one or more processors; and one or more computer readable media comprising the computer readable code executable by one or more processors to: capture image data of a three-dimensional object; identify a constellation of markers on the three-dimensional object in the image data; define a spatial relationship between the constellation and a geometry of the three-dimensional object; capture image data of the constellation; determine an orientation of the constellation from the image data of the constellation; and determine an orientation of the three-dimensional object based on the orientation of the constellation and the spatial relationship. ### Claim 16 (depends on 15) The system of claim 15, further comprising computer readable code to: generate a three-dimensional digital reconstruction of the three-dimensional object based on the orientation of the constellation. ### Claim 17 (depends on 15) The system of claim 15, further comprising computer readable code to: register an input portion of the three-dimensional object in accordance with the constellation. ### Claim 18 (depends on 15) The system of claim 15, wherein the constellation of markers comprises at least one selected from a group consisting of an IR illuminator and an IR reflector. ### Claim 19 (depends on 15) The system of claim 15, wherein the image data comprises a plurality of images capturing the three-dimensional object from different views. ### Claim 20 (depends on 15) The system of claim 15, wherein the computer readable code to define the spatial relationship between the constellation and the geometry of the three-dimensional object comprises computer readable code to: register three-dimensional object as a controller by associating the constellation with the geometry of the three-dimensional object.