System and apparatus for remote interaction with an object

An apparatus for remote interaction with an object, the apparatus comprising:

• a local system comprising:
• a mixed reality headset configured to display a virtual representation of a physical element relative to the object;
• and
• a device configured to obtain information relating to or dependent on position and orientation of the physical element interacting with the object;
• a remote system located remote from the object and in communication with the local system, the remote system including:
• an electronic device including a display for displaying an image based on the information relating to or dependent on the position and orientation of the physical element interacting with the object;
• and
• an input device for remote user interaction and configured to control position and orientation of the virtual representation displayed on the mixed reality headset, wherein the input device is a 6 degree of freedom input device utilized to control the position and orientation of the virtual representation;
• wherein the remote system is configured to communicate with the local system with latency of 0.5 seconds or less, facilitating alignment of the physical element with the position and orientation of the virtual representation
• wherein the local system includes a force detection arrangement configured to detect a local applied force by the physical element interacting with the object,
• wherein the input device comprises a haptic input device configured to determine remote applied force, and wherein the local system is configured to compare the local applied force to the remote applied force and the local system is configured to provide comparison information.

An apparatus for remote interaction with an object, the apparatus comprising:

• a local system comprising:
• a mixed reality headset configured to display a virtual representation of a physical element relative to the object;
• and
• a device configured to obtain information relating to or dependent on position and orientation of the physical element interacting with the object;
• a remote system located remote from the object and in communication with the local system, the remote system including:
• an electronic device including a display for displaying an image based on the information relating to or dependent on the position and orientation of the physical element interacting with the object;
• and
• an input device for remote user interaction and configured to control position and orientation of the virtual representation displayed on the mixed reality headset, wherein the input device is a 6 degree of freedom input device utilized to control the position and orientation of the virtual representation;
• wherein the remote system is configured to communicate with the local system with latency of 0.5 seconds or less, facilitating alignment of the physical element with the position and orientation of the virtual representation,
• wherein the input device comprises a three dimensional haptic controller for controlling the position and orientation of the virtual representation and providing haptic feedback to the remote user.

A method of interaction with an object comprising:

• establishing communication between a local system and a remote system located remote from the object;
• displaying on a mixed reality headset of the local system, a virtual rendering of a physical element positioned and oriented relative to the object;
• obtaining, by the local system, information relating to or dependent on position and orientation of the physical element interacting with the object;
• sending the information to the remote system;
• displaying, on a display of an electronic device of the remote system, the information relating to or dependent on the position and orientation of the physical element interacting with the object;
• receiving input at a 6 degree of freedom input device of the remote system for controlling position and orientation of the virtual rendering of the physical element;
• controlling the position and orientation of the virtual rendering of the physical element, displayed on the mixed reality headset, relative to the object based on the input received at the input device of the remote system;
• wherein the remote system communicates with the local system with latency of 0.5 seconds or less, facilitating alignment of the physical element with the position and orientation of the virtual rendering;
• generating, at the local system, a map of the object, and providing haptic feedback at the input device of the remote system, based on a relationship between the physical element and the map of the object
• wherein generating the map comprises generating a mesh or generating a depth map of constraints that the object is subject to.

A method of interaction with an object comprising:

• establishing communication between a local system and a remote system located remote from the object;
• displaying on a mixed reality headset of the local system, a virtual rendering of a physical element positioned and oriented relative to the object;
• obtaining, by the local system, information relating to or dependent on position and orientation of the physical element interacting with the object;
• sending the information to the remote system;
• displaying, on a display of an electronic device of the remote system, the information relating to or dependent on the position and orientation of the physical element interacting with the object;
• receiving input at a 6 degree of freedom input device of the remote system for controlling position and orientation of the virtual rendering of the physical element;
• controlling the position and orientation of the virtual rendering of the physical element, displayed on the mixed reality headset, relative to the object based on the input received at the input device of the remote system;
• wherein the remote system communicates with the local system with latency of 0.5 seconds or less, facilitating alignment of the physical element with the position and orientation of the virtual rendering,
• detecting a local applied force by the physical element interacting with the object, determining a remote applied force based on the input received at the input device and comparing the local applied force to the remote applied force and displaying comparison information utilizing the mixed reality headset.