Augmented stereoscopic visualization for a surgical robot using a captured visible image combined with a fluorescence image and a captured visible image

A robotic surgical system comprising:

• an endoscope, held and positioned by the robotic surgical system, comprising:
• a stereoscopic optical channel for transporting light from tissue;
• a capture unit coupled to the stereoscopic optical channel, wherein the capture unit captures (1) a visible first image and (2) a visible second image combined with a fluorescence second image from the light, where the first image is one of a left image and a right image, and the second image is an other of the left image and the right image;
• an intelligent image processing system coupled to the capture unit to receive (1) the visible first image and (2) the visible second image combined with a fluorescence second image, wherein the intelligent image processing system further comprises:
• spatial image registration for registering the visible first image to the visible second image combined with the fluorescence second image;
• a first image warper, coupled to the spatial image registration, for converting the visible first image into a visible second image;
• a first image subtractor coupled to the first image warper to receive the visible second image and coupled to the capture unit to receive the visible second image combined with the fluorescence second image, wherein the image subtractor subtracts the visible second image from the visible second image combined with the fluorescence second image to generate a fluorescence second image;
• a second image subtractor coupled (i) to the first image subtractor to receive the fluorescence second image, and (ii) to the capture unit to receive the visible second image combined with the fluorescence second image, wherein the second image subtractor subtracts the fluorescence second image from the visible second image combined with the fluorescence second image to generate another visible second image;
• a second image warper, coupled to the spatial image registration and to the fluorescence second image for converting the fluorescence second image into a fluorescence first image;
• and
• an augmented stereoscopic display system coupled to the intelligent image processing system and to the image capture system, wherein the augmented stereoscopic display system outputs a real-time stereoscopic image comprising a three-dimensional presentation comprising:
• in one eye, a blend of the at least one fluorescence image of the fluorescence first and second images and one of the visible first and second images;
• and in the other eye, an other of the visible first and second images.

A method comprising:

• capturing a visible first image of tissue from a stereoscopic optical path in an endoscope held and positioned by a robotic surgical system;
• capturing a visible second image combined with a fluorescence second image of the tissue from the stereoscopic optical path in the endoscope held and positioned by the robotic surgical system, where the first image is one of a left image and a right image, and the second image is an other of the left image and the right image;
• processing the visible first image and the visible second image combined with the fluorescence second image to generate at least one fluorescence image of a stereoscopic pair of fluorescence images and a visible second image;
• and
• generating a real-time stereoscopic image comprising a three-dimensional presentation comprising:
• in one eye, a blend of the at least one fluorescence image of a stereoscopic pair of fluorescence images and one of the visible first and second images;
• and in the other eye, an other of the visible first and second images.