# US 2022/0035441 A1 · Systems and methods for object tracking using fused data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0035441 A1
- Document ID: US20220035441A1
- Status: Published application
- Kind: A1
- Application number: 16/983,958
- Filed: 2020-08-03
- Published: 2022-02-03
- Earliest claimed: 2020-08-03
- Assignees: Facebook Technologies, LLC
- Applicants: Facebook Technologies, LLC
- Inventors: Tsz Ho Yu; Chengyuan Yan; Christian Forster
- Web page: https://patentgrep.com/patent/US20220035441A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 18/251: {of input or preprocessed data}
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

In one embodiment, a method includes capturing, using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a part of a hand of the user holding a controller in an environment, identifying one or more features from the first image to estimate a pose of the hand of the user, estimating a first pose of the controller based on the pose of the hand of the user and an estimated grip that defines a relative pose between the hand of the user and the controller, receiving IMU data of the controller, and estimating a second pose of the controller by updating the first pose of the controller using the IMU data of the controller. The method utilizes multiple data sources to track the controller under various conditions of the environment to provide an accurate controller tracking consistently.

## Claims

### Claim 1

A method comprising, by a computing system:
capturing, using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a part of a hand of the user holding a controller in an environment;
identifying one or more features from the first image to estimate a pose of the hand of the user;
estimating a first pose of the controller based on the pose of the hand of the user and an estimated grip that defines a relative pose between the hand of the user and the controller;
receiving inertial measurement unit (IMU) data of the controller; and
estimating a second pose of the controller by updating the first pose of the controller using the IMU data of the controller.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first image is captured at a first frequency and the IMU data of the controller is received at a second frequency, wherein the second frequency is higher than the first frequency.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
receiving IMU data of the wearable device to estimate a pose of the wearable device; and
updating the pose of the hand of the user based on the pose of the wearable device.

### Claim 4 (depends on 3)

The method of claim 3, wherein the pose of the wearable device is estimated based on the IMU data of the wearable device and the first image of the user.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
estimating an IMU-predicted pose of the hand based on the updated first pose of the controller and the IMU data of the controller; and
estimating a second pose of the hand based on the IMU-predicted pose of the hand.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
capturing a second image of the user using the one or more cameras, the second image of the user depicting at least a part of the hand of the user holding the controller in the environment;
identifying the one or more features from the second image of the user; and
estimating a third pose of the hand based on the one or more features identified from the second image of the user.

### Claim 7 (depends on 6)

The method of claim 6, wherein a frequency of estimating the second pose of the hand is higher than a frequency of estimating the third pose of the hand.

### Claim 8 (depends on 1)

The method of claim 1, wherein the wearable device comprises:
the one or more cameras configured to capture images of the user;
a hand-tracking unit configured to estimate the pose of the hand of the user; and
a controller-tracking unit configured to estimate the second pose of the controller.

### Claim 9 (depends on 1)

The method of claim 1, wherein estimating the second pose of the controller comprises:
estimating a pose of the controller relative to the environment based on the estimated grip and the estimated pose of the hand of the user relative to the environment;
adjusting the pose of the controller relative to the environment based on the IMU data of the controller;
estimating a pose of the controller relative to the hand based on the adjusted pose of the controller relative to the environment and the IMU of the controller; and
estimating the second pose of the controller based on the adjusted pose of the controller relative to the environment and the estimated pose of the controller relative to the hand.

### Claim 10

One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
capture, using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a part of a hand of the user holding a controller in an environment;
identify one or more features from the first image to estimate a pose of the hand of the user;
estimate a first pose of the controller based on the pose of the hand of the user and an estimated grip that defines a relative pose between the hand of the user and the controller;
receive inertial measurement unit (IMU) data of the controller; and
estimate a second pose of the controller by updating the first pose of the controller using the IMU data of the controller.

### Claim 11 (depends on 10)

The media of claim 10, wherein the first image is captured at a first frequency and the IMU data of the controller is received at a second frequency, wherein the second frequency is higher than the first frequency.

### Claim 12 (depends on 10)

The media of claim 10, wherein the software is further operable when executed to:
receive IMU data of the wearable device to estimate a pose of the wearable device; and
update the pose of the hand of the user based on the pose of the wearable device.

### Claim 13 (depends on 12)

The media of claim 12, wherein the pose of the wearable device is estimated based on the IMU data of the wearable device and the first image of the user.

### Claim 14 (depends on 10)

The media of claim 10, wherein the software is further operable when executed to:
estimate an IMU-predicted pose of the hand based on the updated first pose of the controller and the IMU data of the controller; and
estimate a second pose of the hand based on the IMU-predicted pose of the hand.

### Claim 15 (depends on 14)

The media of claim 14, wherein the software is further operable when executed to:
capture a second image of the user using the one or more cameras, the second image of the user depicting at least a part of the hand of the user holding the controller in the environment;
identify the one or more features from the second image of the user; and
estimate a third pose of the hand based on the one or more features identified from the second image of the user.

### Claim 16 (depends on 15)

The media of claim 15, wherein a frequency of estimating the second pose of the hand is higher than a frequency of estimating the third pose of the hand.

### Claim 17 (depends on 10)

The media of claim 10, wherein the wearable device comprises:
the one or more cameras configured to capture images of the user;
a hand-tracking unit configured to estimate the pose of the hand of the user; and
a controller-tracking unit configured to estimate the second pose of the controller.

### Claim 18 (depends on 10)

The media of claim 10, wherein estimating the second pose of the controller comprises:
estimating a pose of the controller relative to the environment based on the estimated grip and the estimated pose of the hand of the user relative to the environment;
adjusting the pose of the controller relative to the environment based on the IMU data of the controller;
estimating a pose of the controller relative to the hand based on the adjusted pose of the controller relative to the environment and the IMU of the controller; and
estimating the second pose of the controller based on the adjusted pose of the controller relative to the environment and the estimated pose of the controller relative to the hand.

### Claim 19

A system comprising: one or more processors; and one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by the one or more of the processors to cause the system to:
capture, using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a part of a hand of the user holding a controller in an environment;
identify one or more features from the first image to estimate a pose of the hand of the user;
estimate a first pose of the controller based on the pose of the hand of the user and an estimated grip that defines a relative pose between the hand of the user and the controller;
receive inertial measurement unit (IMU) data of the controller; and
estimate a second pose of the controller by updating the first pose of the controller using the IMU data of the controller.

### Claim 20 (depends on 19)

The system of claim 19, wherein the instructions are further operable when executed to:
receive IMU data of the wearable device to estimate a pose of the wearable device; and
update the pose of the hand of the user based on the pose of the wearable device.
