Methods and systems for touchless control of surgical environment

A system for touchless control of one of more medical equipment devices in an operating room, the system comprising:

• a 3D optical sensor connected for detecting one or more gestures made by a practitioner in a sensing volume of the sensor;
• a controller connected to receive 3D optical data from the 3D optical sensor and configured to provide a three-dimensional control menu, the three-dimensional control menu comprising a plurality of menu items, each menu item selectable by the practitioner by one or more gestures made by the practitioner in a volumetric spatial region corresponding to the menu item and detected by the controller based on the 3D optical data;
• an IDU display for displaying an IDU image corresponding to the three-dimensional control menu onto a non-planar projection surface, the IDU image providing indicia of any one or more selected menu items,
• wherein selection of any particular one of the menu items results in delivering a corresponding operational command to at least one of the one or more medical equipment devices to control operation of the at least one of the one or more medical equipment devices;
• wherein the controller is configured, based on input from one or more sensors, to estimate a profile of the non-planar projection surface;
• wherein the controller is configured, based on input from one or more sensors, to estimate a viewing vector of the practitioner to the non-planar projection surface;
• and
• wherein the controller is configured to pre-adjust the IDU image prior to projecting the IDU image, the pre-adjustment based at least in part on the estimated profile of the non-planar projection surface and the estimated viewing vector;
• wherein the controller is configured to perform a simulation to pre-adjust the IDU image prior to projecting the IDU image which comprises:
• placing a virtual projector at a location of the practitioner's head and oriented along the viewing vector;
• placing a virtual camera at a location of the real projector;
• projecting an undistorted IDU image from the virtual projector onto the projection surface, the undistorted IDU image appearing undistorted from the location of the practitioner's head and oriented along the viewing vector;
• taking a snapshot of the projected undistorted IDU image using the virtual camera, the snapshot of the projected undistorted IDU image appearing distorted from the perspective of the virtual camera;
• and
• pre-adjusting the IDU image prior to projecting the IDU image by replacing the IDU image with the snapshot taken from the virtual camera and projecting the captured snapshot using the real projector.