Extended reality system and extended reality method with two-way digital interactive digital twins

An extended reality (XR) system, comprising:

• a camera device for capturing at least one image of a real asset;
• at least one processor;
• and
• memory containing instructions which, when executed by the at least one processor, cause the at least one processor to:
• receive:
• an asset label of a real asset and a real state of the real asset, the real state including:
• i) at least one image of the real asset, ii) anchor points in real world coordinates of real 3-Dimensional (3D) space, and iii) a real location of the real asset in the real world coordinates of the real 3D space;
• generate, using i) the at least one image of the real asset, ii) the anchor points in the real world coordinates of the real 3D space, and iii) the real location of the real asset in the real world coordinates of the real 3D space:
• a 3D map in an XR application which includes a virtual asset of the real asset with a virtual state which matches the real state;
• receive navigation through the 3D map in the XR application to a virtual location of the virtual asset which matches the real location of the real asset in the real world coordinates of the real 3D space;
• receive an initiation of an event of the real asset;
• generate, using a physics engine:
• one or more predicted virtual states of the virtual asset which are predicted to achieve the event in the real world coordinates of a virtual 3D space, wherein the physics engine includes a rules based model or a convolutional neural network (CNN) model;
• receive one or more further real states of the real asset which achieve the event;
• generate the 3D map in an XR application with a concurrent display of the virtual asset in the one or more predicted virtual states and the real asset in the one or more further real states which achieve the event;
• and
• generate for display the 3D map on a display, wherein the at least one image of the real asset is taken from the camera device,
• wherein the receive the initiation of the event is from the real asset,
• wherein the receive the initiation of the event includes receive a communication from the real asset,
• wherein the receive the initiation of the event includes:
• receive one or more further real states of the real asset, and
• detect a change in the real state of the real asset,
• wherein the receive the one or more further real states of the real asset includes receive a respective bounding box for each of the at least one image of the real asset, and wherein the detect the change in the real state includes compare the respective bounding box and the at least one image of the real asset with a known attribute of the real asset.

A non-transitory memory containing instructions which, when executed by at least one processor, cause at least one processor to perform an extended reality (XR) method, the instructions comprising:

• instructions for receiving:
• an asset label of a real asset and a real state of the real asset, the real state including:
• i) at least one image of the real asset, ii) anchor points in real world coordinates of real 3-Dimensional (3D) space, and iii) a real location of the real asset in the real world coordinates of the real 3D space;
• instructions for generating, using i) the at least one image of the real asset, ii) the anchor points in the real world coordinates of the real 3D space, and iii) the real location of the real asset in the real world coordinates of the real 3D space:
• a 3D map in an XR application which includes a virtual asset of the real asset with a virtual state which matches the real state;
• instructions for receiving navigation through the 3D map in the XR application to a virtual location of the virtual asset which matches the real location of the real asset in the real world coordinates of the real 3D space;
• instructions for receiving an initiation of an event of the real asset;
• instructions for generating, using a physics engine:
• one or more predicted virtual states of the virtual asset which are predicted to achieve the event in the real world coordinates of a virtual 3D space, wherein the physics engine includes a rules based model or a convolutional neural network (CNN) model;
• instructions for receiving one or more further real states of the real asset which achieve the event;
• instructions for generating the 3D map in an XR application with a concurrent display of the virtual asset in the one or more predicted virtual states and the real asset in the one or more further real states which achieve the event;
• and
• instructions for generating for display the 3D map on a display, wherein the at least one image of the real asset is taken from a camera device,
• wherein the receiving the initiation of the event is from the real asset,
• wherein the receiving the initiation of the event includes receiving a communication from the real asset,
• wherein the receiving the initiation of the event includes:
• receiving one or more further real states of the real asset, and
• detecting a change in the real state of the real asset,
• wherein the receiving the one or more further real states of the real asset includes receiving a respective bounding box for each of the at least one image of the real asset, and wherein the detecting the change in the real state includes comparing the respective bounding box and the at least one image of the real asset with a known attribute of the real asset.