Multi-sensor handle controller hybrid tracking method and device

A multi-sensor handle controller hybrid tracking device, wherein, a surface of a handle controller is provided with an optical pattern mark point, and the device comprises:

• a plurality of cameras provided on a head mounted display and configured to track and shoot the optical pattern mark point;
• an optical sensor configured to acquire optical tracking data of the optical pattern mark point;
• an electromagnetic sensor configured to acquire electromagnetic tracking data of the handle controller;
• an inertial navigation sensor configured to acquire inertial navigation data of the handle controller;
• the head mounted display internally provided with a wireless transmission module and configured to receive the optical tracking data, the electromagnetic tracking data, and the inertial navigation data;
• a model construction module configured to construct a state transition model and an observation model of an extended Kalman filtering iteration strategy and perform an extended Kalman filtering fusion on the optical tracking data, the electromagnetic tracking data, and the inertial navigation data, wherein constructing a state transition model and an observation model of an extended Kalman filtering iteration strategy according to the tracking data further comprises:
• acquiring 6DoF data and velocity information of an optical pattern mark point on the handle controller relative to a first camera on a head mounted display in real time according to the optical tracking data,
• converting 6DoF data with respect to other cameras into 6DoF data under a coordinate system with the first camera as an origin, by using calibration parameters among the cameras, which is denoted as Poseoptic,
• converting electromagnetic 6DoF data of the handle controller into 6DoF data under the coordinate system with the first camera as the origin through calibration parameter of an electromagnetic signal receiver internally provided in the head mounted display and the first camera, which is denoted as PoseEM,
• fusing the data and the data Poseoptic and the data PoseEM into the new pose data, which is denoted as PoseOEM, and
• constructing the state transition model and the observation model of the extended Kalman filtering iteration strategy by using the new pose data;
• and
• a tracking module configured to determine position and orientation information of the handle controller in a space according to the extended Kalman filtering iteration strategy.