Apparatus and method for using augmented reality vision system in surgical procedures

A method for generating a safety zone onto a video signal, the method comprising:

• storing a pre-operative image of an anatomical section of a patient;
• analyzing the pre-operative image to determine a safety zone around an anatomical body within the patient and generate data about the anatomical body of the patient, the data including a label for the anatomical body and a peripheral margin of at least a portion of the anatomical body, the peripheral margin configured to guide a user to a cutting location relative to the anatomical body, wherein the anatomical body is located within the anatomical section;
• receiving the video signal from a camera located within the patient during a surgical procedure, wherein the video signal includes a signal associated with an image of at least a portion of a surgical tool operated by the user;
• augmenting the safety zone, the data, and an identity of the user operating the surgical tool onto the video signal;
• displaying the video signal with the safety zone and the identity of the user operating the surgical tool;
• determining a location of the surgical tool within the displayed video signal;
• generating a notification to the user operating the surgical tool based on the location of the surgical tool relative to the safety zone;
• sensing a loading condition on the surgical tool;
• generating a feedback signal based on the sensed loading condition;
• and
• updating, in real time, the data and a location of the augmented identity of the user operating the surgical tool on the displayed video signal in response to a change in the location of the surgical tool within the displayed video signal.

A system for augmenting data onto a real time video signal, comprising:

• a pre-operative image generated of an anatomical section of a patient;
• a camera configured to send the real time video signal from the patient to the controller, wherein the camera is placed within the patient prior to starting surgery and the real time video signal includes an image of at least a portion of a first surgical tool operated by a first user;
• a sensor disposed on the first surgical tool and configured to sense a loading condition on the first surgical tool and to generate a feedback signal based on the sensed loading condition;
• the controller configured to analyze the pre-operative image and the feedback signal from the sensor to gather data about the anatomical section, and to augment an identity of the first user and the data about the anatomical section onto the real time video signal, the data including a label for the anatomical section and a peripheral margin of at least a portion of the anatomical section, the peripheral margin configured to guide the first user to a cutting location relative to the anatomical section;
• and
• a user interface configured to display the real time video signal and the augmented data, wherein the controller is further configured to identify the first surgical tool as under control of the first user on the user interface, and to update, in real time, the data and a location of the identity of the first user on the user interface in response to a change in a location of the first surgical tool within the video signal.

A method for augmenting a video image of a patient, comprising:

• processing a pre-operative image of an anatomical section of a patient to generate data about the anatomical section of the patient, the data including a label for the anatomical section and a peripheral margin of at least a portion of the anatomical section, the peripheral margin configured to guide a user to a cutting location relative to the anatomical section;
• identifying at least one anatomical object in the processed pre-operative image;
• identifying at least one of the at least one anatomical objects in the processed pre-operative image as a delicate;
• receiving a video image from a camera disposed within the patient;
• displaying in the video image at least a portion of a surgical tool operated by the user;
• augmenting onto the video image a safety boundary around the at least one anatomical object identified as delicate to prevent the user from treating the at least one anatomical object identified as delicate with the surgical tool;
• embedding the data and an identity of the user within the video image to supply an augmented reality display to the user about the anatomical section of the patient;
• determining a location of the surgical tool relative to the safety boundary;
• generating a notification to the user based on the determined location of the surgical tool;
• sensing a loading condition on the surgical tool;
• generating a feedback signal based on the sensed loading condition;
• and
• updating, in real time, the data and a location of the identity of the user operating the surgical tool embedded within the video image in response to a change in a location of the surgical tool within the video image.

A method for augmenting a video image of a patient, the method comprising:

• processing a pre-operative image of an anatomical section of a patient;
• identifying at least one anatomical object in the processed pre-operative image;
• generating a label for at least one of the at least one anatomical objects identified in the processed pre-operative image;
• determining a location and a margin of the at least one anatomical object identified in the processed pre-operative image, the margin configured to guide a user to a cutting location relative to the at least one anatomical object;
• identifying at least one of the at least one anatomical objects in the processed pre-operative image as delicate;
• determining a safety boundary around the at least one anatomical object identified as delicate;
• sensing a loading condition on the surgical tool;
• generating a feedback signal based on the sensed loading condition;
• receiving a video image from a camera disposed within the patient;
• embedding the label, the margin, and an identity of the user within the video image to supply an augmented reality display to the user about the anatomical section of the patient;
• overlaying the generated label and the determined margin on the video image;
• overlaying at least a portion of the determined safety boundary on the video image to prevent the user from treating the at least one anatomical object identified as delicate with the surgical tool;
• and
• updating, in real time, the label, the margin, the identity of the user and a location of the identity of the user operating the surgical tool embedded within the video image in response to a change in a location of the surgical tool within the video image.