# US 2019/0318501 A1 · Tracking pose of handheld object > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0318501 A1 - Document ID: US20190318501A1 - Status: Published application - Kind: A1 - Application number: 15/954,362 - Filed: 2018-04-16 - Published: 2019-10-17 - Earliest claimed: 2018-04-16 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Alexandru Octavian BALAN; Yogeshwar Narayanan NAGARAJ; Constantin DULU; William GUYMAN; Ivan RAZUMENIC - Web page: https://patentgrep.com/patent/US20190318501A1 ## CPC - G06T 7/74: {involving reference images or patches} - A63F 13/211: using inertial sensors, e.g. accelerometers or gyroscopes - A63F 13/212: using sensors worn by the player, e.g. for measuring heart beat or leg activity - A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence) - A63F 13/235: using a wireless connection, e.g. infrared or piconet - A63F 13/26: having at least one additional display device, e.g. on the game controller or outside a game booth - A63F 13/92: Video game devices specially adapted to be hand-held while playing - G06F 3/012: {Head tracking input arrangements} - G06F 3/0308: {comprising a plurality of distinctive and separately oriented light emitters or reflectors associated to the pointing device, e.g. remote cursor controller with distinct and separately oriented LEDs at the tip whose radiations are captured by a photo-detector associated to the screen} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 7/248: {involving reference images or patches} - A63F 2300/405: being a wireless ad hoc network, e.g. Bluetooth®, Wi-Fi®, piconet - G06T 2207/30204: Marker - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Examples are disclosed that relate to tracking a pose of a handheld object used with a head-mounted display device. In one example, a method comprises: receiving image data from an image sensing system; detecting a plurality of feature points of the handheld object in a frame of the image data; receiving inertial measurement unit (IMU) data from an IMU of the handheld object; based on detecting the plurality of feature points and receiving the IMU data, determining a first pose of the handheld object; determining that at least a portion of the plurality of feature points is not detected in another frame of the image data; using the IMU data, updating the first pose of the handheld object to a second pose; and body-locking the second pose of the handheld object to a body location on a user wearing the head-mounted display device. ## Claims ### Claim 1 At a computing device, a method for tracking a pose of a handheld object used with a head-mounted display device, the method comprising: receiving image data from an image sensing system; detecting a plurality of feature points of the handheld object in a frame of the image data; receiving inertial measurement unit (IMU) data from an IMU of the handheld object; based on detecting the plurality of feature points and receiving the IMU data, determining a first pose of the handheld object; determining that at least a portion of the plurality of feature points is not detected in another frame of the image data; using the IMU data, updating the first pose of the handheld object to a second pose; and body-locking the second pose of the handheld object to a body location on a user wearing the head-mounted display device. ### Claim 2 (depends on 1) The method of claim 1, wherein body-locking the second pose of the handheld object further comprises: determining that a first threshold time has been reached; determining that the handheld object is not stationary; and based on determining that the first threshold time has been reached and that the handheld object is not stationary, body-locking the second pose of the handheld object to the body location on the user wearing the head-mounted display device. ### Claim 3 (depends on 2) The method of claim 2, further comprising: determining that a second threshold time has been reached after expiration of the first threshold time; and based on determining that the first threshold time has been reached, the handheld object is not stationary, and the second threshold time has been reached, body-locking the second pose of the handheld object to the body location on the user wearing the head-mounted display device. ### Claim 4 (depends on 2) The method of claim 2, further comprising: determining that the handheld object is stationary after determining that the handheld object is not stationary; and based on determining that the handheld object is stationary after determining that the handheld object is not stationary, using the IMU data to update an orientation of the second pose of the handheld object, and marking a position of the second pose as invalid. ### Claim 5 (depends on 1) The method of claim 1, further comprising: prior to body-locking the second pose of the handheld object, determining that a velocity of the handheld object is less than a threshold velocity; and based on determining that the velocity of the handheld object is less than the threshold velocity, body-locking the second pose of the handheld object to the body location on the user wearing the head-mounted display device. ### Claim 6 (depends on 1) The method of claim 1, further comprising: determining that the handheld object is stationary; and based on determining that the handheld object is stationary, world-locking the second pose of the handheld object. ### Claim 7 (depends on 1) The method of claim 1, wherein the body location on the user wearing the head-mounted display device comprises a neck or an elbow of the user. ### Claim 8 (depends on 1) The method of claim 1, further comprising detecting additional feature points in one or more additional frames of the image data; and based on detecting the additional feature points, ceasing to body-lock the second pose of the handheld object and updating the second pose to a third pose. ### Claim 9 (depends on 1) The method of claim 1, wherein the pose of the handheld object comprises a position and an orientation, and body-locking the second pose of the handheld object comprises locking the position and not locking the orientation. ### Claim 10 (depends on 1) The method of claim 1, wherein the image sensing system is mounted on the head-mounted display device. ### Claim 11 A computing device for tracking a pose of a handheld object used with a head-mounted display device, the computing device comprising: a logic device configured to execute instructions; and a storage device comprising instructions executable by the logic device to: receive image data from an image sensing system; detect a plurality of feature points of the handheld object in a frame of the image data; receive inertial measurement unit (IMU) data from an IMU of the handheld object; based on detecting the plurality of feature points and receiving the IMU data, determine a first pose of the handheld object; determine that at least a portion of the plurality of feature points is not detected in another frame of the image data; using the IMU data, update the first pose of the handheld object to a second pose; and body-lock the second pose of the handheld object to a body location on a user wearing the head-mounted display device. ### Claim 12 (depends on 11) The computing device of claim 11, wherein the instructions are executable to: determine that a first threshold time has been reached; determine that the handheld object is not stationary; and based on determining that the first threshold time has been reached and that the handheld object is not stationary, body-lock the second pose of the handheld object to the body location on the user wearing the head-mounted display device. ### Claim 13 (depends on 12) The computing device of claim 12, wherein the instructions are executable to: determine that a second threshold time has been reached after expiration of the first threshold time; and based on determining that the first threshold time has been reached, the handheld object is not stationary, and the second threshold time has been reached, body-lock the second pose of the handheld object to the body location on the user wearing the head-mounted display device. ### Claim 14 (depends on 12) The computing device of claim 12, wherein the instructions are executable to: determine that the handheld object is stationary after determining that the handheld object is not stationary; and based on determining that the handheld object is stationary after determining that the handheld object is not stationary, use the IMU data to update an orientation of the second pose of the handheld object, and marking a position of the second pose as invalid. ### Claim 15 (depends on 11) The computing device of claim 11, wherein the instructions are executable to: prior to body-locking the second pose of the handheld object, determine that a velocity of the handheld object is less than a threshold velocity; and based on determining that the velocity of the handheld object is less than the threshold velocity, body-lock the second pose of the handheld object to the body location on the user wearing the head-mounted display device. ### Claim 16 (depends on 11) The computing device of claim 11, wherein the instructions are executable to: determine that the handheld object is stationary; and based on determining that the handheld object is stationary, world-lock the second pose of the handheld object. ### Claim 17 (depends on 11) The computing device of claim 11, wherein the body location on the user wearing the head-mounted display device comprises a neck or an elbow of the user. ### Claim 18 (depends on 11) The computing device of claim 11, wherein the instructions are executable to: detect additional feature points in one or more additional frames of the image data; and based on detecting the additional feature points in the image data, cease body-locking the second pose of the handheld object and update the second pose to a third pose. ### Claim 19 (depends on 11) The computing device of claim 11, wherein the pose of the handheld object comprises a position and an orientation, and body-locking the second pose of the handheld object comprises locking the position and not locking the orientation. ### Claim 20 A head-mounted display device, comprising: an image sensing system; and a computing device comprising: a logic device configured to execute instructions; and a storage device comprising instructions executable by the logic device to: receive image data from the image sensing system; detect a plurality of feature points of the handheld object in a frame of the image data; receive inertial measurement unit (IMU) data from an IMU of the handheld object; based on detecting the plurality of feature points and receiving the IMU data, determine a first pose of the handheld object; determine that at least a portion of the plurality of feature points is not detected in another frame of the image data; using the IMU data, update the first pose of the handheld object to a second pose; determine that the handheld object is not stationary; determine that a first threshold time has been reached; and based on determining that the first threshold time has been reached and that the handheld object is not stationary, body-lock the second pose of the handheld object to a body location on a user wearing the head-mounted display device.