# US 12,108,930 B2 · Image correction of a surgical endoscope video stream > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,108,930 B2 - Document ID: US12108930B2 - Status: Granted - Kind: B2 - Application number: 18/235,039 - Filed: 2023-08-17 - Published: 2023-12-07 - Granted: 2024-10-08 - Earliest claimed: 2018-08-24 - Assignees: Cmr Surgical Limited - Applicants: Cmr Surgical Limited - Inventors: Luke David Ronald Hares - Web page: https://patentgrep.com/patent/US12108930B2 ## CPC - A61B 1/000095: {for image enhancement} - A61B 34/30: Surgical robots - A61B 34/37: Leader-follower robots (A61B34/35 takes precedence) - A61B 90/361: {Image-producing devices, e.g. surgical cameras} - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 5/92: based on global image properties - G06T 5/94: based on local image properties, e.g. for local contrast enhancement - G06T 7/11: Region-based segmentation - G06T 7/174: involving the use of two or more images - G06T 7/579: from motion - A61B 1/00149: {using articulated arms} - A61B 1/00193: {adapted for stereoscopic vision} - A61B 2034/301: {for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes} - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/10068: Endoscopic image - G06T 2207/20208: High dynamic range [HDR] image processing - G06T 2207/30004: Biomedical image processing - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method of real-time image correction of a video stream from a surgical endoscope of a surgical robotic system. The video stream comprises a sequence of images. The method comprises: for an image in the sequence of images, identifying a plurality of regions, each region having a different range of values of at least one image property to another region; applying a mask to the image which applies a region-specific modification to each region, each region-specific modification modifying the at least one image property of that region; deriving data from the surgical robotic system; determining a relationship between features in the image from the derived data; modifying the mask for a predicted relationship between the features at the time of a subsequent image in the video stream; and applying the modified mask to the subsequent image in the video stream. ## Claims ### Claim 1 A method of real-time image correction of a video stream from a surgical endoscope of a surgical robotic system, the video stream comprising a sequence of images, the method comprising: for an image in the sequence of images, identifying a plurality of regions, each region having a different range of values of at least one image property to another region; applying a mask to the image which applies a region-specific modification to each region, each region-specific modification modifying at least one image property of that region; determining a relationship between features in the image; modifying the mask for a predicted relationship between the features at the time of a subsequent image in the video stream; and applying the modified mask to the subsequent image in the video stream. ### Claim 2 (depends on 1) The method of claim 1, further comprising deriving data from the surgical robotic system and determining the relationship between features in the image from the derived data. ### Claim 3 (depends on 1) The method of claim 1, further comprising storing a set of predetermined modifications, and generating each region-specific modification as a weighted combination of two or more predetermined modifications. ### Claim 4 (depends on 3) The method of claim 3, wherein the predetermined modifications modify one or more of the following image properties: brightness, contrast, gamma and colour. ### Claim 5 (depends on 1) The method of claim 1, wherein the predicted relationship between the features at the time of the subsequent image is such that the shape of one or more of the features in the image has changed. ### Claim 6 (depends on 1) The method of claim 1, wherein the predicted relationship between the features at the time of the subsequent image is such that the relative positions of the features have changed. ### Claim 7 (depends on 2) The method of claim 2, wherein the data comprises the position of the surgical endoscope relative to another feature in the image. ### Claim 8 (depends on 2) The method of claim 2, wherein the data comprises depth information of the features in the image. ### Claim 9 (depends on 8) The method of claim 8, comprising deriving the depth information from movement of the surgical endoscope. ### Claim 10 (depends on 8) The method of claim 8, wherein the video stream is a combination of two stereo video channels of the surgical endoscope, the method comprising deriving the depth information from the two stereo video channels of the surgical endoscope. ### Claim 11 (depends on 2) The method of claim 2, wherein the data comprises identification of features in the image. ### Claim 12 (depends on 11) The method of claim 11, comprising selecting regions of the image to correspond to identified features of the image, and applying feature-specific modifications to those selected regions. ### Claim 13 (depends on 1) The method of claim 1, comprising tracking the surgeon's focus, selecting a first region of the image to be centred on the surgeon's focus, selecting a second region of the image to exclude the surgeon's focus, and applying different region-specific modifications to the first and second regions. ### Claim 14 (depends on 1) The method of claim 1, comprising identifying a plurality of regions of the subsequent image based on the predicted relationship between the features at the time of the subsequent image in the video stream, wherein applying the modified mask to the subsequent image in the video stream comprises applying a region-specific modification to each region of the subsequent image. ### Claim 15 (depends on 1) The method of claim 1, further comprising applying an iteratively updated mask to further images of the video stream by: determining an updated relationship between features in a further image of the sequence of images; further modifying the mask to form an updated mask for the updated predicted relationship between the features at the time of a yet further image in the video stream; applying the updated mask to the yet further image in the video stream; and iteratively performing the above steps for further images of the video stream. ### Claim 16 (depends on 15) The method of claim 15, further comprising deriving further data from the surgical robotic system and determining an updated relationship between features in the further image from the derived further data. ### Claim 17 (depends on 15) The method of claim 15, comprising updating the mask at a slower rate than the image frame rate of the video stream. ### Claim 18 (depends on 15) The method of claim 15, comprising only updating the mask upon gross changes of the features in the image. ### Claim 19 (depends on 18) The method of claim 18, comprising only updating the mask upon gross movement of the surgical endoscope and/or the patient the surgical endoscope is inside.