System, controller and method using virtual reality device for robotic surgery

A surgical robot system comprising:

• a robot configured to operate at least one end-effector positioned at a surgical site within a patient;
• at least one image acquisition device configured to acquire live images at the surgical site;
• a virtual reality (VR) device configured to display the acquired live images and determine tracking data provided by actions of a user via the VR device;
• and
• a control unit comprising:
• input/output (I/O) circuitry configured to receive the acquired live images from the at least one image acquisition device, provide the acquired live images to the VR device for display, receive the determined tracking data from the VR device, and provide robot control signals to the robot;
• and
• a processor configured to:
• transmit the acquired live images, received from the at least one image acquisition device via the I/O circuitry, to the VR device;
• process the determined tracking data from the VR device to select a target at the surgical site within the patient;
• determine a path for the at least one end-effector to reach the selected target based upon the acquired live images and the processed determined tracking data;
• and
• transmit the robot control signals to the robot to move the at least one instrument to the selected target via the determined path.

A non-transitory computer-readable storage medium storing machine readable instructions executable by a processor to operate a surgical robot system, including at least one robot configured to operate at least one end-effector to be positioned at a surgical site within a patient, at least one image acquisition device configured to acquire live images at the surgical site, and a head-mounted display (HMD) device to be worn by a user and configured to display the acquired live images and to determine at least one of motion of the user's head or movement of the user's eyes, the non-transitory computer readable medium comprising:

• transmitting code for causing transmission of the acquired live images, received from the at least one image acquisition device, to the HMD device;
• processing code for processing determined at least one of head-tracking data and eye-tracking data from the HMD device to select a target at the surgical site within the patient;
• determining code for determining a path for a distal portion of the robot, including the at least one end-effector, to reach the selected target at the surgical site within the patient based upon the acquired live images and the processed determined head-tracking data and/or eye-tracking data;
• and
• robot control code for causing transmission of the robot control signals to the robot to move the at least one end-effector to the selected target at the surgical site within the patient via the determined path.