Multi-modal localization of streaming content providers in a virtual environment

A method, comprising:

• inputting depth image data, ultra-wideband (UWB) data, and inertial measurement unit (IMU) data into a machine learning engine at a refresh rate of a virtual environment,
• the depth image data including a depth image acquired at a first rate;
• the UWB data including samples of a UWB signal acquired at a second rate;
• the IMU data representing samples of a IMU signal acquired at a third rate, the first rate being different from at least one of the second rate or the third rate;
• the machine learning engine being configured to output a position value and an orientation value indicative of a position and an orientation of a mobile device within the virtual environment;
• and
• rendering a virtual object representing the mobile device in the virtual environment according to the position and orientation.

A computer program product comprising a nontransitory storage medium, the computer program product including code that, when executed by at least one processor, causes the at least one processor to perform a method, the method comprising:

• acquiring, at a first rate, depth image data in a real-world environment;
• acquiring, at a second rate, a sample of an ultra-wideband (UWB) signal;
• acquiring, at a third rate, a sample of an inertial measurement unit (IMU) signal, the first rate being different from at least one of the second rate or the third rate;
• inputting the depth image data, the sample of the UWB signal, and the sample of the IMU signal into a machine learning engine at a refresh rate of a virtual environment, the machine learning engine being configured to output a position value and an orientation value indicative of a position and an orientation of a mobile device within the virtual environment;
• and
• rendering a virtual object representing the mobile device in the virtual environment according to the position and orientation.

An apparatus, comprising:

• memory;
• and
• processing circuitry coupled to the memory, the processing circuitry being configured to:
• input depth image data, ultra-wideband (UWB) data, and inertial measurement unit (IMU) data into a machine learning engine at a refresh rate of a virtual environment,
• the depth image data including a depth image acquired at a first rate;
• the UWB data including samples of a UWB signal acquired at a second rate;
• the IMU data representing samples of a IMU signal acquired at a third rate, the first rate being different from at least one of the second rate or the third rate;
• the machine learning engine being configured to output a position value and an orientation value indicative of a position and an orientation of a mobile device within the virtual environment;
• and
• render a virtual object representing the mobile device in the virtual environment according to the position and orientation.