Systems and methods for object tracking using fused data

One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

• capture, using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a part of a hand of the user holding a controller in an environment;
• identify one or more features from the first image to estimate a pose of the hand of the user;
• estimate a first pose of the controller based on the pose of the hand of the user and an estimated grip that defines a relative pose between the hand of the user and the controller;
• receive inertial measurement unit (IMU) data of the controller;
• and
• estimate a second pose of the controller by updating the first pose of the controller using the IMU data of the controller.

A system comprising:

• one or more processors;
• and one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by the one or more of the processors to cause the system to:
• capture, using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a part of a hand of the user holding a controller in an environment;
• identify one or more features from the first image to estimate a pose of the hand of the user;
• estimate a first pose of the controller based on the pose of the hand of the user and an estimated grip that defines a relative pose between the hand of the user and the controller;
• receive inertial measurement unit (IMU) data of the controller;
• and
• estimate a second pose of the controller by updating the first pose of the controller using the IMU data of the controller.