Medical optical system, data processing system, computer program, and non-volatile computer-readable storage medium

A data processing system comprising:

• a receiving interface for receiving histological images representing a corresponding morphology of each of a plurality of microscopic tissue sections of a macroscopic tissue region, the macroscopic tissue region comprising a tumor, wherein each histological image has associated navigation data representing a position and an orientation of an endomicroscope used to capture the histological image;
• a classification device that processes, using a neural network, the microscopic tissue sections represented in the histological images, to classify each of the microscopic tissue sections represented in the histological images, as a tissue section representing the tumor or a tissue section representing healthy tissue in each case, and for outputting a classification result for each classified microscopic tissue section, wherein the neural network is:
• trained using a plurality of training samples, each particular training sample of the plurality of training samples representing a training tissue section representing a tumor or a training tissue section representing healthy tissue, and associated with a corresponding assigned classification indicating whether the training tissue section represents healthy tissue or tumor tissue;
• and
• trained to predict, for each particular training sample of the plurality of training samples, the corresponding assigned classification based on the morphology of the training tissue section represented by the respective training sample;
• and
• a combination device which generates a macroscopic classification image by combining the classification results and the navigation data indicating a spatial position of each microscopic tissue section within the macroscopic tissue region, the macroscopic classification image composed of a plurality of sections, each section comprising an indication of whether a corresponding microscopic tissue section corresponds to the tumor or the healthy tissue, and wherein spatial positioning of the microscopic tissue sections in the macroscopic classification image corresponds to relative positions of the microscopic tissue sections in the macroscopic tissue region;
• optical observation equipment for producing an overview image of the macroscopic tissue region;
• and
• an overlay apparatus configured to overlay the macroscopic classification image on the overview image, wherein the overlay apparatus uses the navigation data to register the macroscopic classification image with the overview image to ensure spatial accuracy of the overlay of the microscopic classification image on the overview image.

A non-volatile computer-readable storage medium with instructions stored thereon, wherein said instructions, when executed on a computer, cause the computer to:

• receive histological images representing a corresponding morphology of each of a plurality of microscopic tissue sections of a macroscopic tissue region, the macroscopic tissue region comprising a tumor, wherein each histological image has associated navigation data representing a position and an orientation of an endomicroscope used to capture the histological image;
• process, using a neural network, using a neural network, the microscopic tissue sections represented in the histological images, to classify each of the microscopic tissue sections represented in the histological images, as a tissue section representing the tumor or a tissue section representing healthy tissue in each case, and for outputting a classification result for each classified microscopic tissue section, wherein the neural network is, wherein the neural network is:
• trained using a plurality of training samples, each particular training sample of the plurality of training samples representing a training tissue section representing a tumor or a training tissue section representing healthy tissue, and associated with a corresponding assigned classification indicating whether the morphology of the training tissue section represents healthy tissue or tumor tissue;
• and
• trained to predict, for each particular training sample of the plurality of training samples, the corresponding assigned classification based on the morphology of the training tissue section represented by the respective training sample;
• and
• generate a macroscopic classification image by combining the classification results and the navigation data indicating a spatial position of each microscopic tissue section within the macroscopic tissue region, the macroscopic classification image composed of a plurality of sections, each section comprising an indication of whether a corresponding microscopic tissue section corresponds to the tumor or the healthy tissue, and wherein spatial positioning of the microscopic tissue sections in the macroscopic classification image corresponds to relative positions of the microscopic tissue sections in the macroscopic tissue region;
• produce an overview image of the macroscopic tissue region;
• and
• overlay the macroscopic classification image on the overview image using the navigation data to register the macroscopic classification image with the overview image to ensure spatial accuracy of the overlay of the microscopic classification image on the overview image.

A medical optical system comprising:

• an endomicroscope for recording histological images representing a corresponding morphology of each of a plurality of microscopic tissue sections of a macroscopic tissue region, the macroscopic tissue region comprising a tumor;
• a classification device that processes, using a neural network, the microscopic tissue sections represented in the histological images, to classify each of the microscopic tissue sections as a tissue section representing the tumor or a tissue section representing healthy tissue, and for outputting a classification result for each classified microscopic tissue section, wherein the neural network is:
• trained using a plurality of training samples, each particular training sample of the plurality of training samples representing a training tissue section representing a tumor or a training tissue section representing healthy tissue, and associated with a corresponding assigned classification indicating whether the training tissue section represents healthy tissue or tumor tissue;
• and
• trained to predict, for each particular training sample of the plurality of training samples, the corresponding assigned classification based on the morphology of the training tissue section represented by the respective training sample;
• and
• a navigation system for gathering navigation data representing a site of the macroscopic tissue region at which the microscopic tissue section underlying respective classification data is situated, the navigation data representing a position and an orientation of the endomicroscope used to capture each histological image;
• and
• a combination device which generates a macroscopic classification image by combining the classification results and the navigation data indicating a spatial position of each microscopic tissue section within the macroscopic tissue region, the macroscopic classification image composed of a plurality of sections, each section comprising an indication of whether a corresponding microscopic tissue section corresponds to the tumor or the healthy tissue, and wherein spatial positioning of the microscopic tissue sections in the macroscopic classification image corresponds to relative positions of the microscopic tissue sections in the macroscopic tissue region;
• optical observation equipment for producing an overview image of the macroscopic tissue region;
• and
• an overlay apparatus configured to overlay the macroscopic classification image on the overview image using the navigation data to register the macroscopic classification image with the overview image to ensure spatial accuracy of the overlay of the microscopic classification image on the overview image.

A medical optical system comprising:

• an endomicroscope for recording histological images representing a corresponding decay of fluorescence radiation emitted by each of a plurality of microscopic tissue sections of a macroscopic tissue region, the macroscopic tissue region comprising a tumor, wherein each histological image has associated navigation data representing a position and an orientation of the endomicroscope used to capture the histological image;
• a classification device that processes, using a neural network, the microscopic tissue sections represented in the histological images to classify each of the microscopic tissue sections as a tissue section representing the tumor or a tissue section representing healthy tissue based on the corresponding decay of fluorescence radiation emitted by the respective microscopic tissue section, and for outputting a classification result for each classified microscopic tissue section, wherein the neural network is:
• trained using a plurality of training samples, each particular training sample of the plurality of training samples representing a training decay of fluorescence radiation emitted by a training tissue section and associated with a corresponding assigned classification indicating whether the training decay of fluorescence radiation of the training tissue section represents healthy tissue or tumor tissue;
• and
• trained to predict, for each particular training sample of the plurality of training samples, the corresponding assigned classification based on the training decay of fluorescence radiation;
• a combination device which generates a macroscopic classification image by combining the classification results and the navigation data indicating a spatial position of each microscopic tissue section within the macroscopic tissue region, the macroscopic classification image composed of a plurality of sections, each section comprising an indication of whether a corresponding microscopic tissue section corresponds to the tumor or the healthy tissue, and wherein spatial positioning of the microscopic tissue sections in the macroscopic classification image corresponds to relative positions of the microscopic tissue sections in the macroscopic tissue region;
• optical observation equipment for producing an overview image of the macroscopic tissue region;
• and
• an overlay apparatus being configured to overlay the macroscopic classification image on the overview image, wherein the overlay apparatus uses the navigation data to register the macroscopic classification image with the overview image to ensure spatial accuracy of the overlay of the microscopic classification image on the overview image.

A medical optical system comprising:

• an endomicroscope for recording histological images representing each of a plurality of microscopic tissue sections of a macroscopic tissue region, the macroscopic tissue region comprising a tumor, wherein each histological image has associated navigation data representing a position and an orientation of the endomicroscope used to capture the histological image;
• optical observation equipment for producing a fluorescence image and an overview image corresponding to the macroscopic tissue region;
• a classification device that processes, using a neural network, the microscopic tissue sections represented in the histological images and the fluorescence image, to classify each of the microscopic tissue sections as a tissue section representing the tumor or a tissue section representing healthy tissue and for outputting a classification result for each classified microscopic tissue section, wherein the neural network is:
• trained using a plurality of training samples, each particular training sample of the plurality of training samples representing a training tissue section representing a tumor or a training tissue section representing healthy tissue, and associated with a corresponding fluorescence image and a corresponding assigned classification indicating whether the training tissue section represents healthy tissue or tumor tissue;
• and
• trained to predict, for each particular training sample of the plurality of training samples, the corresponding assigned classification based on of the training tissue section represented by the respective training sample and the corresponding fluorescence image;
• and
• a combination device which generates a macroscopic classification image by combining the classification results each indicating whether the corresponding microscopic tissue section represents the tumor tissue or the healthy tissue, the macroscopic classification image composed of a plurality of sections, each section comprising an indication of whether a corresponding microscopic tissue section corresponds to the tumor or the healthy tissue;
• and
• an overlay apparatus configured to overlay the macroscopic classification image on the overview image, wherein the overlay apparatus uses the navigation data to register the macroscopic classification image with the overview image to ensure spatial accuracy of the overlay of the microscopic classification image on the overview image.