Method and system for augmented imaging in open treatment using multispectral information

A system for generating and displaying one or more augmented images of tissue of a patient undergoing open treatment said system comprising:

• an apparatus configured to estimate a spectral composition of light illuminating a region of interest of the tissue by measuring or computing said spectral composition based on control parameters controlling said light illuminating the region of interest,
• a multispectral camera for obtaining one or more multispectral images of the region of interest,
• a computing device comprising a machine learning module for applying the one or more multispectral images or one or more images derived from said one or more multispectral images to a machine learning based regressor or classifier, to thereby derive one or more tissue parameters associated with image regions or pixels of the one or more multispectral images, wherein said machine learning based regressor or classifier has been trained to predict the one or more tissue parameters from the one or more multispectral images under a given spectral composition of illumination,
• a display device,
• wherein the computing device is configured to generate one or more augmented images and display said one or more augmented images on the display device, wherein said one or more augmented images correspond to the one or more multispectral images and include the one or more tissue parameters,
• wherein the computing device is configured to make the machine learning based regressor or classifier match the estimated spectral composition of light illuminating said region of interest of the tissue by one of
• selecting the machine learning based regressor or classifier from among a plurality of candidate machine learning based regressors or classifiers that each have been previously trained for a different spectral composition of illumination, such that the given spectral composition of illumination machine learning based regressor or classifier is the most similar to the estimated spectral composition of said light illuminating said region of interest, or
• transforming the obtained one or more multispectral images based on information derived from the estimated spectral composition of the light illuminating said region of interest and applying the transformed one or more multispectral images to a standard machine learning based regressor or classifier, which has been trained under a standard illumination, wherein the transformation is for compensating for a change in the one or more multispectral images due to a deviation in the spectral composition of the illumination from the standard illumination, or
• retraining an already trained machine learning based regressor or classifier using simulation data that is adapted to the estimated spectral composition of light illuminating said region of interest.