Augmented stereoscopic visualization for a surgical robot using a captured fluorescence image and captured stereoscopic visible images

A robotic surgical system comprising:

• an illumination channel, held and positioned by the robotic surgical system, wherein light from the illumination channel illuminates tissue;
• a stereoscopic optical channel, held and positioned by the robotic surgical system, configured to transport first light from the tissue;
• another optical channel, held and positioned by the robotic surgical system, configured to transport second light from the tissue wherein the stereoscopic optical channel is different from the another optical channel;
• a combination light source, coupled to the illumination channel, comprising:
• a first light source;
• and
• a second light source wherein illumination light from the second light source is different from illumination light from the first light source;
• an image capture system comprising:
• a first capture unit coupled to the stereoscopic optical channel wherein the first capture unit captures a stereoscopic visible image from the first light;
• and
• a second capture unit coupled to the another optical channel wherein the second capture unit captures a fluorescence image from the second light;
• an image processing system, coupled to the first capture unit and to the second capture unit to receive the stereoscopic visible image and to receive the fluorescence image, comprising:
• a single-step calibration wherein the single-step calibration determines a displacement necessary to bring images from the stereoscopic optical channel and from the another optical channel into alignment;
• a spatial image registration, coupled to the single-step calibration, for registering the fluorescence image to a visible left image in the stereoscopic visible image and to a visible right image in the stereoscopic visible image;
• an image warper coupled to the spatial image registration and coupled to receive the fluorescence image, wherein the image warper is configured to convert the fluorescence image into a first fluorescence image, and to convert the fluorescence image into a second fluorescence image, and wherein the first fluorescence image and the second fluorescence image comprise a stereoscopic pair of fluorescence images;
• and
• an augmented stereoscopic display system coupled to the image processing system and to the image capture system, wherein the augmented stereoscopic display system outputs a stereoscopic image comprising a blend of the stereoscopic visible image and the stereoscopic pair of fluorescence images and further wherein the augmented stereoscopic display system comprises:
• a first blend circuit coupled to receive one of the first and second fluorescence images in the stereoscopic pair of fluorescence images, and to receive a visible left image of the stereoscopic visible image;
• a second blend circuit coupled to receive an other of the first and second fluorescence images in the stereoscopic pair of fluorescence images, and to receive a visible right image of the stereoscopic visible image;
• a stereoscopic display coupled to the first blend circuit and to the second blend circuit;
• and
• a display mode select, coupled to the first and second blend circuits, having a plurality of selectable modes,
• wherein upon selection of a first mode, the stereoscopic display provides a blend of the stereoscopic visible image and the stereoscopic pair of fluorescence images, and
• wherein upon selection of a second mode, the stereoscopic display provides only the real time stereoscopic visible image.

A method comprising:

• capturing a stereoscopic visible image of tissue from a stereoscopic optical channel held and positioned by a robotic surgical system;
• capturing a fluorescence image of the tissue from another optical channel held and positioned by the robotic surgical system, wherein the stereoscopic optical channel is different from the another optical channel;
• processing information associated with the fluorescence image to generate a first fluorescence image, and to generate a second fluorescence image, wherein the first fluorescence image and the second fluorescence image comprise a stereoscopic pair of fluorescence images;
• and
• generating an augmented stereoscopic image of the tissue comprising a blend of the stereoscopic visible image and the stereoscopic pair of fluorescence images;
• wherein the processing the information associated with the fluorescence image further comprises:
• converting the information associated with the fluorescence image into the first and second fluorescence images using an image warper;
• and
• wherein the generating an augmented stereoscopic image further comprises:
• blending one of the first and second fluorescence images in the stereoscopic pair of fluorescence images and a visible left image of the stereoscopic visible image;
• and
• blending another one of the first and second fluorescence images in the stereoscopic pair of fluorescence images and a visible right image of the stereoscopic visible image.