# US 12,602,118 B2 · Six-degree of freedom pose estimation of a stylus > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,118 B2 - Document ID: US12602118B2 - Status: Granted - Kind: B2 - Application number: 18/233,126 - Filed: 2023-08-11 - Published: 2025-02-13 - Granted: 2026-04-14 - Earliest claimed: 2023-08-11 - Assignees: zSpace, Inc. - Applicants: zSpace, Inc. - Inventors: Mark Wilson - Web page: https://patentgrep.com/patent/US12602118B2 ## CPC - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/03545: {Pens or stylus} - G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T 2207/20084: Artificial neural networks [ANN] - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Systems and methods for six-degree of freedom (6-DoF) pose estimation of a user input device, e.g., in a three-dimensional (3D) display system rendering interactive augmented reality (AR) and/or virtual reality (VR) experiences include the user input device capturing, via a camera disposed at a forward-facing tip of the user input device, images in a direction that the user input device is directed and determining, via an inertial measurement unit (IMU), motion of the user input device in three-dimensional (3D) space. The user input device may then determine pose information associated with the user input device based on the images and motion of the user input device. The determination of the pose information may be via usage of at least one of a neural network model, estimation model trained on a set of unique and identifiable patterns, and/or an estimation model trained on a dataset of images. ## Claims ### Claim 1 A method for determining a pose of a user input device, comprising: the user input device, capturing, via at least one camera disposed at a forward facing tip of the user input device, images in a direction that the user input device is directed; determining, via an inertial measurement unit (IMU) comprised in the user input device, motion of the user input device in three-dimensional (3D) space; and determining pose information associated with the user input device based, at least in part, on the images captured via the at least one camera of the user input device and motion determined via the IMU comprised in the user input device. ### Claim 2 (depends on 1) The method of claim 1, wherein the at least one camera comprises a wide-angle camera. ### Claim 3 (depends on 1) The method of claim 1, wherein the at least one camera is a monochromatic camera or a color camera. ### Claim 4 (depends on 1) The method of claim 1, wherein the at least one camera further comprises a lens configured to split a view of the at least one camera between the direction that user input device is directed and a second direction. ### Claim 5 (depends on 4) The method of claim 4, wherein, in a main orientation of the user input device, the second direction is downward-facing with respect to the main orientation of the user input device. ### Claim 6 (depends on 4) The method of claim 4, wherein capturing, via at least one camera, images in the direction that the user input device is directed further comprises capturing images in the second direction. ### Claim 7 (depends on 4) The method of claim 4, wherein the direction corresponds to a view of a display of a computer system, and wherein the second direction corresponds to a view of a keyboard of the computer system. ### Claim 8 (depends on 1) The method of claim 1, wherein the at least one camera further comprises a second camera, wherein the second camera is disposed on a bottom of the user input device, and wherein capturing the images further comprises capturing the images via the second camera. ### Claim 9 A user input device, comprising: a memory; at least one processor in communication with the memory; an inertial measurement unit (IMU) in communication with at least the memory; at least one camera, disposed at a forward-facing tip of the user input device, and in communication with at least the memory; and wherein the at least one processor is further configured to: capture, via the at least one camera disposed at the forward-facing tip of the user input device, images in a direction that the user input device is directed; determine, via the IMU, motion of the user input device in three-dimensional (3D) space; and determine pose information associated with the user input device based, at least in part, on the images captured via the at least one camera and motion of the user input device determined via the IMU. ### Claim 10 (depends on 9) The user input device of claim 9, wherein the at least one processor is further configured to: send, to a computer system, the pose information. ### Claim 11 (depends on 9) The user input device of claim 9, wherein the user input device further comprises: at least one antenna; and at least one radio in communication with the at least one antenna. ### Claim 12 (depends on 9) The user input device of claim 9, wherein the at least one processor is further configured to: send, to a computer system, the pose information via a wireless connection with the computer system. ### Claim 13 (depends on 12) The user input device of claim 12, wherein the wireless connection operates according to at least one short-range radio access technology (RAT). ### Claim 14 (depends on 13) The user input device of claim 13, wherein the at least one RAT comprises at least one of Bluetooth, Wi-Fi, or Near Field Communication (NCF). ### Claim 15 (depends on 9) The user input device of claim 9, wherein the at least one processor is further configured to: operate as one or more of an active stylus, two-dimensional (2D) scanner, a three-dimensional (3D) scanner, a barcode scanner, a quick response (QR) code scanner, a digital writing tool, a digital painting tool, or a digital camera. ### Claim 16 (depends on 9) The user input device of claim 9, wherein the at least one processor is further configured to: receive a first user input; and select a mode of operation based on the first user input, wherein the mode of operation comprises one or more of active stylus, digital camera, two-dimensional (2D) scanner, three-dimensional (3D) scanner, barcode scanner, quick response (QR) code scanner, digital writing tool, or digital painting tool. ### Claim 17 (depends on 16) The user input device of claim 16, wherein, to receive the first user input, the at least one processor is further configured to: receive the first user input via one or more buttons of the user input device. ### Claim 18 (depends on 9) The user input device of claim 9, wherein the at least one camera further comprises a second camera, wherein the second camera is disposed on a bottom of the user input device, and wherein capturing the images further comprises capturing the images via the second camera. ### Claim 19 (depends on 9) The user input device of claim 9, wherein the images comprise a sequence of images. ### Claim 20 An apparatus, comprising: a memory; and one or more processors in communication with the memory and configured to cause a user input device to: capture, via at least one camera in communication with the one or more processors and disposed at a forward-facing tip of the user input device, images in a direction that the user input device is directed; determine, via an inertial measurement unit (IMU) in communication with the one or more processors and comprised in the user input device, motion of the user input device in three-dimensional (3D) space; and determine pose information associated with the user input device based, at least in part, on the images captured via the at least one camera and motion of the user input device determined via the IMU comprised in the user input device. ### Claim 21 (depends on 20) The apparatus of claim 20, wherein the pose information comprises a six-degree of freedom position and orientation of the user input device. ### Claim 22 (depends on 20) The apparatus of claim 20, wherein the pose information is determined relative to a display of a computer system. ### Claim 23 (depends on 20) The apparatus of claim 20, wherein the user input device comprises an active stylus. ### Claim 24 (depends on 20) The apparatus of claim 20, wherein, to determine pose information associated with the user input device based, at least in part, on the images and motion of the user input device, the one or more processors are further configured to: provide the images and motion of the user input device as inputs to a neural network model; and receive, from the neural network model, an estimate of a pose of the user input device, wherein the pose information comprises the estimate of the pose of the user input device. ### Claim 25 (depends on 24) The apparatus of claim 24, wherein the neural network model comprises a convolutional neural network. ### Claim 26 (depends on 20) The apparatus of claim 20, wherein the at least one camera further comprises a lens configured to split a view of the at least one camera between the direction that user input device is directed and a second direction. ### Claim 27 (depends on 26) The apparatus of claim 26, wherein, in a main orientation of the user input device, the second direction is downward-facing with respect to the main orientation of the user input device. ### Claim 28 (depends on 26) The apparatus of claim 26, wherein capturing, via at least one camera, images in the direction that the user input device is directed further comprises capturing images in the second direction. ### Claim 29 (depends on 26) The apparatus of claim 26, wherein the direction corresponds to a view of a display of a computer system, and wherein the second direction corresponds to a view of a keyboard of the computer system. ### Claim 30 (depends on 1) The method of claim 1, wherein the user input device comprises an active stylus.