# US 11,445,159 B2 · Context-sensitive white balancing for surgical microscopes > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,445,159 B2 - Document ID: US11445159B2 - Status: Granted - Kind: B2 - Application number: 17/109,813 - Filed: 2020-12-02 - Published: 2021-06-10 - Granted: 2022-09-13 - Earliest claimed: 2019-12-05 - Assignees: Carl Zeiss Meditec AG - Applicants: Carl Zeiss Meditec AG - Inventors: Abouzar Eslami; Corinna Maier-Matic; Thorsten Tritschler - Web page: https://patentgrep.com/patent/US11445159B2 ## CPC - G06T 5/92: based on global image properties - A61B 3/0025: {characterised by electronic signal processing, e.g. eye models} - A61B 3/132: {in binocular arrangement} - G02B 21/0012: {Surgical microscopes (counterbalanced structures for surgical microscopes G02B7/001)} - G02B 21/22: Stereoscopic arrangements - G02B 21/365: {Control or image processing arrangements for digital or video microscopes (G02B21/361, G02B21/362 take precedence)} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 5/60: using machine learning, e.g. neural networks - G06T 7/90: Determination of colour characteristics - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/82: using neural networks - G06V 10/95: {structured as a network, e.g. client-server architectures} - G06V 20/693: {Acquisition} - H04N 13/133: Equalising the characteristics of different image components, e.g. their average brightness or colour balance - H04N 13/15: for colour aspects of image signals - H04N 13/189: Recording image signals; Reproducing recorded image signals - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - H04N 13/257: Colour aspects - H04N 13/296: Synchronisation thereof; Control thereof - H04N 23/88: for colour balance, e.g. white-balance circuits or colour temperature control - G06T 2207/10012: Stereo images - G06T 2207/10024: Color image - G06T 2207/10056: Microscopic image - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30041: Eye; Retina; Ophthalmic - G06V 2201/033: of skeletal patterns - G06V 2201/034: of medical instruments - G06V 2201/10: Recognition assisted with metadata - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A computer-implemented method and a corresponding system for context-sensitive white balancing for a stereomicroscope are presented. The method comprises recording a first digital image by way of a first camera in a first optical path of the stereomicroscope, and recording a second digital image by way of a second camera in a second optical path of the stereomicroscope. Furthermore, the method comprises determining, by means of a trained machine learning system, the context identified in the images, and determining, by means of the trained machine learning system, camera parameters suitable for controlling color channels of the first and second cameras for white balancing. ## Claims ### Claim 1 A computer-implemented method for context-sensitive white balancing for a stereomicroscope, the method comprising: recording a first digital image by way of a first camera in a first optical path of the stereomicroscope; recording a second digital image by way of a second camera in a second optical path of the stereomicroscope; determining, by means of a trained machine learning system, the context identified in the images; and determining, by means of the trained machine learning system, camera parameters suitable for controlling color channels of the first and second cameras for white balancing, wherein the machine learning system is represented by a network comprising a first pipeline and a second pipeline, wherein the first pipeline performs a processing of the first digital image, wherein the second pipeline performs a processing of the second digital image, wherein the first pipeline is trained to determine the identified context of the first digital image and wherein the second pipeline is trained to determine the identified context of the second digital image independently of the identified context of the first image. ### Claim 2 (depends on 1) The method of claim 1, wherein the camera parameters additionally comprise a color consistency value indicative of a color difference between the first digital image and the second digital image. ### Claim 3 (depends on 1) The method of claim 1, wherein a respective classifier layer is used at the end of the first pipeline and at the end of the second pipeline. ### Claim 4 (depends on 1) The method of claim 1, wherein each of the first and second pipelines consists of a sequence of a plurality of groups, wherein each of the groups comprises a plurality of successive convolutional layers, followed in each case by a pooling layer. ### Claim 5 (depends on 4) The method of claim 4, wherein skip connections are selectively present between individual convolutional layers from among the successive convolutional layers. ### Claim 6 (depends on 4) The method of claim 4, wherein a respective interim result in the first and second pipelines after a predetermined number of layers of the network is fed to a combination layer. ### Claim 7 (depends on 6) The method of claim 6, wherein the combination layer is a beginning of a third pipeline in the machine learning system, wherein the third pipeline comprises a further group of a plurality of successive convolutional layers and/or pooling layers after the combination layer and an interim result at the end of the third pipeline is fed to a regression network as input data, wherein the regression network comprises the determined camera parameters as output data. ### Claim 8 (depends on 7) The method of claim 7, wherein the interim result at the end of the third pipeline is also fed to a classifier system as input data, wherein the classifier system comprises the color consistency value as output data. ### Claim 9 (depends on 1) The method of claim 1, wherein the camera parameters for at least one color channel comprise a correction value in order to control the white balancing. ### Claim 10 (depends on 1) The method of claim 1, wherein the machine learning system comprises a deep neural network, a decision tree system or a random forest system. ### Claim 11 (depends on 1) The method of claim 1, further comprising training the machine learning system with a set of pairs of digital images, wherein each pair corresponds to a respective imaging from the first optical path and the second optical path, by means of machine learning for generating a learning model for determining the context from each image and for determining the camera parameters and/or a color consistency value indicative of a color deviation between the recorded images of the first and second cameras. ### Claim 12 (depends on 11) The method of claim 11, wherein the images of the set of pairs of digital images are generated at least partly by means of an image generator. ### Claim 13 (depends on 1) The method of claim 1, wherein the learning system comprises a system for supervised learning, and wherein the images of the set of digital images for supervised learning are annotated with metadata for generating the learning model. ### Claim 14 (depends on 1) The method of claim 1, wherein the identified context comprises a predefined type of tissue, a bone, an instrument, or a sensor. ### Claim 15 A white balancing system for context-sensitive white balancing for a stereomicroscope, the white balancing system comprising: a first camera in a first optical path of the stereomicroscope for recording a first digital image; a second camera in a second optical path of the stereomicroscope for recording a second digital image; a context determining unit adapted for determining, by means of a trained machine learning system, the context identified in the images; and a parameter determining unit for determining, by means of the trained machine learning system, camera parameters suitable for controlling color channels of the first and second cameras for white balancing, wherein the machine learning system is represented by a network comprising a first pipeline and a second pipeline, wherein the first pipeline performs a processing of the first digital image and wherein the second pipeline performs a processing of the second digital image, wherein the first pipeline is trained to determine the identified context of the first digital image, and wherein the second pipeline is trained to determine the identified context of the second digital image independently of the identified context of the first image. ### Claim 16 (depends on 15) A microscope system comprising the white balancing system of claim 15. ### Claim 17 (depends on 1) A computer program product for context-sensitive white balancing for a stereomicroscope, wherein the computer program product comprises a computer-readable storage medium comprising program instructions stored thereon, wherein the program instructions are executable by one or a plurality of computers or control units and cause the one or the plurality of computers or control units to carry out the method of claim 1.