Technique for visualizing interactions with a technical device in an xr scene

A computer-implemented method for visualizing interactions in an extended reality (XR) scene, the computer-implemented method comprising:

• receiving a first dataset, the first dataset representing the XR scene including at least a technical device;
• displaying the XR scene on an XR headset or a head-mounted display (HMD);
• providing a room for a user, wherein the user wears the XR headset or HMD to interact with the XR scene, wherein the XR scene is displayed on the XR headset or HMD, wherein the room includes a set of optical sensors, and wherein the set of optical sensors includes at least one optical sensor at a fixed location relative to the room;
• detecting, via the set of optical sensors, optical sensor data of the user as a second dataset while the user interacts with the XR scene in the room;
• and
• fusing the first dataset and the second dataset to generate a third dataset, wherein
• the fusing includes applying a calibration algorithm, which utilizes at least one registration object deployed in the room,
• the calibration algorithm uses a set of registration objects, which are provided as real, physical objects in the room and which are provided as displayed virtual objects in the XR scene, and
• for registration purposes, the real, physical objects are moved to match the displayed virtual objects in the XR scene.

A computing device for visualizing interactions in an extended reality (XR) scene, the computing device comprising:

• a first input interface configured to receive a first dataset, the first dataset representing the XR scene including at least a technical device;
• a second input interface configured to receive, from a set of optical sensors, detected optical sensor data of a user as a second dataset while the user interacts with the XR scene in a room, wherein the XR scene is displayed to the user on a XR headset or head-mounted display (HMD), and wherein the set of optical sensors includes at least one optical sensor at a fixed location relative to the room;
• and
• at least one processor configured to
• pre-process the first dataset to obtain a pre-processed first dataset, wherein pre-processing the first dataset to obtain the pre-processed first dataset includes applying an image-to-image transfer learning algorithm to the first dataset, and
• fuse the pre-processed first dataset and the second dataset to generate a third dataset.

A computing device for visualizing interactions in an extended reality (XR) scene, the computing device comprising:

• a first input interface configured to receive a first dataset, the first dataset representing the XR scene including at least a technical device;
• a second input interface configured to receive, from a set of optical sensors, detected optical sensor data of a user as a second dataset while the user interacts with the XR scene in a room, wherein the XR scene is displayed to the user on a XR headset or head-mounted display (HMD), and wherein the set of optical sensors includes at least one optical sensor at a fixed location relative to the room;
• and
• at least one processor configured to fuse the first dataset and the second dataset to generate a third dataset, wherein
• fusing the first dataset and the second dataset to generate the third dataset includes applying a calibration algorithm, which utilizes at least one registration object deployed in the room,
• the calibration algorithm uses a set of registration objects, which are provided as real, physical objects in the room and which are provided as displayed virtual objects in the XR scene, and
• for registration purposes, the real, physical objects are moved to match the displayed virtual objects in the XR scene.