System and apparatus for remote interaction with an object

A method of interaction with a subject comprising:

• establishing communication between a local system and a remote system located remote from the subject;
• capturing video of a physical element interacting with the subject;
• displaying on a display of the local system, the video of the physical element interacting with the subject and a virtual rendering of the physical element positioned and oriented relative to the subject to facilitate alignment of the physical element with the virtual representation of the physical element;
• obtaining, by the local system, information dependent on position and orientation of the physical element interacting with the subject;
• sending the information to the remote system;
• receiving, at an electronic device of the remote system, the video of the physical element interacting with the subject in real time;
• displaying, on a display of the electronic device of the remote system, the video of the physical element interacting with the subject, and the information dependent on the position and orientation of the physical element interacting with the subject;
• receiving input at an input device of the remote system;
• determining the position and orientation of the input device of the remote system and transmitting the determined position and orientation to the local system;
• controlling position and orientation of the virtual rendering of the physical element, displayed on the display of the local system, relative to the subject based on the determined position and orientation of the input device of the remote system;
• wherein the remote system communicates with the local system with latency of 0.5 seconds or less, facilitating alignment of the physical element with the position and orientation of the virtual rendering.

An apparatus for remote interaction with an object, the apparatus comprising:

• a local system comprising:
• a local electronic device including a local display configured to display a virtual representation of a physical element relative to the object, the local electronic device configured to obtain information relating to or dependent on position and orientation of the physical element interacting with the object;
• a force detection arrangement configured to detect a local applied force by the physical element on the subject;
• a remote system located remote from the object and configured to communicate with the local system, the remote system including:
• a remote electronic device including a remote display for displaying an image based on the information relating to or dependent on the position and orientation of the physical element interacting with the object;
• and
• a haptic input device for remote user interaction, wherein the remote system is configured to determine the position and orientation of the input device and transmit the determined position and orientation to the local system to control the position and orientation of the virtual representation of the physical element relative to the object, wherein the haptic input device of the remote system is configured to provide haptic feedback based on the local applied force detected by the force detection arrangement of the local system;
• wherein the remote system is configured to communicate with the local system with latency of 0.5 seconds or less, facilitating alignment of the physical element with the position and orientation of the virtual representation.