Method and system for facilitating remote presentation or interaction

A method for operating a facilitation system for facilitating remote presentation of a physical world comprising a medical system with a robotic manipulator and an operating environment of the medical system, the method, performed by a processing system, comprising:

• obtaining an image frame, with a processing system of the facilitation system, the image frame depicting the physical world;
• identifying, with the processing system, a depiction of the medical system in the image frame;
• obtaining, with the processing system, a first spatial registration, the first spatial registration registering a system model of the medical system with the medical system in the physical world, wherein the system model comprises a plurality of 3D objects spatially arranged based on a plurality of 3D poses;
• obtaining, with the processing system, an updated system model corresponding to the system model updated with a current kinematic configuration of the robotic manipulator, wherein the current kinematic configuration comprises positions of joints of the robotic manipulator, and wherein the updating of the system model comprises updating the 3D poses from previous 3D poses based on the positions of the joints;
• generating, with the processing system, a hybrid frame using the image frame, the first spatial registration, and the updated system model, wherein the hybrid frame comprises a digital replica of the physical world, the digital replica comprising the updated system model;
• and
• transmitting, with the processing system, the digital replica without the image frame, to a remote visualization system that is physically separate from the processing system, for rendering.

A non-transitory machine-readable medium comprising a plurality of machine-readable instructions which when executed by one or more processors associated with a facilitation system are adapted to cause the one or more processors to perform a method for operating the facilitation system for facilitating remote presentation of a physical world, the physical world comprising a medical system and an operating environment of the medical system with a robotic manipulator, the method, performed by a processing system, comprising:

• obtaining an image frame, with a processing system of the facilitation system, the image frame depicting the physical world;
• identifying, with the processing system, a depiction of the medical system in the image frame;
• obtaining, with the processing system, a first spatial registration, the first spatial registration registering a system model with the medical system in the physical world, wherein the system model comprises a plurality of 3D objects spatially arranged based on a plurality of 3D poses;
• obtaining, with the processing system, an updated system model corresponding to the system model updated with a current kinematic configuration of the robotic manipulator, wherein the current kinematic configuration comprises positions of joints of the robotic manipulator, and wherein the updating of the system model comprises updating the 3D poses from previous 3D poses based on the positions of the joints;
• generating, with the processing system, a hybrid frame using the image frame, the first spatial registration, and the updated system model, wherein the hybrid frame comprises a digital replica of the physical world, the digital replica comprising system model;
• and
• transmitting, with the processing system, the digital replica without the image frame, to a remote visualization system that is physically separate from the processing system, for rendering.