The colorization of medical devices in robotic surgery using ai and machine learning

A computer-implemented method for object recognition during a medical procedure, comprising:

• receiving, from an imaging device positioned inside a patient, a stream of intraoperative three-dimensional (3D) imaging data and an imaging scene of a target, the imaging scene including a medical instrument;
• applying a machine-learning model, including a tool state recognition algorithm to the medical instrument to determine, based on one or more of the stream or tool data received from the medical instrument, a tool state of the medical instrument;
• and
• generating a modified imaging scene that includes a visual characteristic applied to the medical instrument.

A system for an intraoperative instrument recognition, comprising:

• at least one medical instrument that includes at least one actuator sensor configured to generate tool data indicative of a tool state of the at least one medical instrument;
• at least one imaging device configured to capture a stream of intraoperative three-dimensional (3D) imaging data;
• and
• an imaging analysis device that includes:
• at least one memory storage:
• instructions for intraoperative medical instrument recognition, including:
• a dynamic tool state recognition algorithm configured to recognize tool states based on one or more of the stream or the tool data from the at least one actuator sensor;
• a material recognition algorithm;
• and
• a shape recognition algorithm;
• and
• at least one processor operatively connected to the at least one memory storage and the at least one medical instrument, and configured to execute the instructions to perform operations including:
• receiving, from the imaging device, the stream of intraoperative three-dimensional (3D) imaging data from inside of a patient, the stream including anatomy of the patient and the at least one medical instrument;
• applying the material recognition algorithm to identify one or more objects formed of a predetermined material present in the intraoperative 3D imaging data;
• applying the shape recognition algorithm to the one or more identified objects to identify the at least one medical instrument;
• applying the dynamic tool state algorithm to the at least one identified medical instrument to identify a tool state of the at least one medical instrument;
• and
• generating a modified intraoperative imaging data stream in real-time with the receiving of the stream of intraoperative 3D imaging data, wherein the modified intraoperative imaging data stream includes a visual characteristic applied to a region of the intraoperative 3D imaging data corresponding to the at least one identified medical instrument, wherein the visual characteristic is indicative of the identified tool state.