# US 2025/0147577 A1 · Technique for visualizing interactions with a technical device in an xr scene

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0147577 A1
- Document ID: US20250147577A1
- Status: Published application
- Kind: A1
- Application number: 19/016,554
- Filed: 2025-01-10
- Published: 2025-05-08
- Earliest claimed: 2022-11-02
- Assignees: Siemens Healthineers AG
- Applicants: Siemens Healthineers AG
- Inventors: Daniel RINCK; Aniol SERRA JUHE
- Web page: https://patentgrep.com/patent/US20250147577A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G09B 23/28: for medicine
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06V 10/803: {of input or preprocessed data}
- G06V 20/20: in augmented reality scenes
- G16H 40/63: for local operation
- G16H 40/67: for remote operation
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/502: {Headgear, e.g. helmet, spectacles}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)

## Abstract

A computer-implemented method for visualizing interactions in an extended reality (XR) scene, the computer-implemented method comprising: receiving a first dataset representing an XR scene including a technical device; displaying the XR scene on an XR headset or a head-mounted display (HMD); providing a room for a user wearing the XR headset or HMD for interacting with the XR scene, wherein the room includes a set of optical sensors including at least one optical sensor at a fixed location relative to the room; detecting optical sensor data of the user as a second dataset while the user is interacting with the XR scene in the room; and fusing the first dataset and the second dataset to generate a third dataset.

## Claims

### Claim 1

A computer-implemented method for visualizing interactions in an extended reality (XR) scene, the computer-implemented method comprising:
receiving a first dataset, the first dataset representing the XR scene including at least a technical device;
displaying the XR scene on an XR headset or a head-mounted display (HMD);
providing a room for a user, wherein the user wears the XR headset or HMD to interact with the XR scene, wherein the XR scene is displayed on the XR headset or HMD, wherein the room includes a set of optical sensors, and wherein the set of optical sensors includes at least one optical sensor at a fixed location relative to the room;
detecting, via the set of optical sensors, optical sensor data of the user as a second dataset while the user interacts with the XR scene in the room;
pre-processing the first dataset to obtain a pre-processed first dataset, wherein the pre-processing includes applying an image-to-image transfer learning algorithm to the first dataset; and
fusing the pre-processed first dataset and the second dataset to generate a third dataset.

### Claim 2 (depends on 1)

The computer-implemented method according to claim 1, wherein the set of optical sensors includes at least one depth camera to provide point cloud data.

### Claim 3 (depends on 2)

The computer-implemented method according to claim 2, wherein the at least one depth camera includes an RGBD camera.

### Claim 4 (depends on 2)

The computer-implemented method according to claim 2,
wherein the first dataset, the second dataset and the third dataset include a point cloud, and
wherein the third dataset includes a fusion of point clouds of the first dataset and the second dataset.

### Claim 5 (depends on 4)

The computer-implemented method according to claim 4, further comprising:
using a trained neural network to provide output data based on input data, wherein
the input data includes the third dataset and the output data represents a semantic context of the optical sensor data of the user.

### Claim 6 (depends on 1)

The computer-implemented method according to claim 1,
wherein the first dataset, the second dataset and the third dataset include a point cloud, and
wherein the third dataset includes a fusion of point clouds of the first dataset and the second dataset.

### Claim 7 (depends on 1)

The computer-implemented method according to claim 1, further comprising:
generating real-time instructions based on the third dataset; and
providing the real-time instructions to the user.

### Claim 8 (depends on 1)

The computer-implemented method according to claim 1, further comprising:
using a trained neural network to provide output data based on input data, wherein
the input data includes the third dataset and the output data represents a semantic context of the optical sensor data of the user.

### Claim 9 (depends on 8)

The computer-implemented method according to claim 8, wherein the trained neural network is trained by providing input data labeled with content data, and wherein the content data represents a semantic context of user interaction.

### Claim 10 (depends on 8)

The computer-implemented method according to claim 8, further comprising:
pre-processing the third dataset before being used as the input data for the trained neural network, wherein
the pre-processing the third dataset includes application of an image-to-image transfer learning algorithm to the third dataset.

### Claim 11 (depends on 8)

The computer-implemented method according to claim 8, further comprising:
receiving, by the trained neural network as input data, detected optical sensor data of a user interacting with a real-world scene, wherein
the real-world scene includes a real-world technical device, and
the real-world technical device corresponds to the technical device of the XR scene.

### Claim 12 (depends on 1)

The computer-implemented method according to claim 1, wherein the fusing includes applying a calibration algorithm, which utilizes at least one registration object deployed in the room.

### Claim 13 (depends on 1)

The computer-implemented method according to claim 1, wherein the computer-implemented method is used for at least one of product development of the technical device or for controlling the technical device.

### Claim 14 (depends on 1)

A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by a computing device, cause the computing device to perform the computer-implemented method according to claim 1.

### Claim 15

A computer-implemented method for visualizing interactions in an extended reality (XR) scene, the computer-implemented method comprising:
receiving a first dataset, the first dataset representing the XR scene including at least a technical device;
displaying the XR scene on an XR headset or a head-mounted display (HMD);
providing a room for a user, wherein the user wears the XR headset or HMD to interact with the XR scene, wherein the XR scene is displayed on the XR headset or HMD, wherein the room includes a set of optical sensors, and wherein the set of optical sensors includes at least one optical sensor at a fixed location relative to the room;
detecting, via the set of optical sensors, optical sensor data of the user as a second dataset while the user interacts with the XR scene in the room; and
fusing the first dataset and the second dataset to generate a third dataset, wherein
the fusing includes applying a calibration algorithm, which utilizes at least one registration object deployed in the room,
the calibration algorithm uses a set of registration objects, which are provided as real, physical objects in the room and which are provided as displayed virtual objects in the XR scene, and
for registration purposes, the real, physical objects are moved to match the displayed virtual objects in the XR scene.

### Claim 16 (depends on 15)

The computer-implemented method according to claim 15, wherein the real, physical objects include a first set of spheres, and wherein the displayed virtual objects include a second set of spheres.

### Claim 17 (depends on 15)

A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by a computing device, cause the computing device to perform the computer-implemented method according to claim 15.

### Claim 18

A computing device for visualizing interactions in an extended reality (XR) scene, the computing device comprising:
a first input interface configured to receive a first dataset, the first dataset representing the XR scene including at least a technical device;
a second input interface configured to receive, from a set of optical sensors, detected optical sensor data of a user as a second dataset while the user interacts with the XR scene in a room, wherein the XR scene is displayed to the user on a XR headset or head-mounted display (HMD), and wherein the set of optical sensors includes at least one optical sensor at a fixed location relative to the room; and
at least one processor configured to
pre-process the first dataset to obtain a pre-processed first dataset, wherein pre-processing the first dataset to obtain the pre-processed first dataset includes applying an image-to-image transfer learning algorithm to the first dataset, and
fuse the pre-processed first dataset and the second dataset to generate a third dataset.

### Claim 19

A computing device for visualizing interactions in an extended reality (XR) scene, the computing device comprising:
a first input interface configured to receive a first dataset, the first dataset representing the XR scene including at least a technical device;
a second input interface configured to receive, from a set of optical sensors, detected optical sensor data of a user as a second dataset while the user interacts with the XR scene in a room, wherein the XR scene is displayed to the user on a XR headset or head-mounted display (HMD), and wherein the set of optical sensors includes at least one optical sensor at a fixed location relative to the room; and
at least one processor configured to fuse the first dataset and the second dataset to generate a third dataset, wherein
fusing the first dataset and the second dataset to generate the third dataset includes applying a calibration algorithm, which utilizes at least one registration object deployed in the room,
the calibration algorithm uses a set of registration objects, which are provided as real, physical objects in the room and which are provided as displayed virtual objects in the XR scene, and
for registration purposes, the real, physical objects are moved to match the displayed virtual objects in the XR scene.