# US 11,026,561 B2 · Method and system for automatically repositioning a viewable area within an endoscope video view > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,026,561 B2 - Document ID: US11026561B2 - Status: Granted - Kind: B2 - Application number: 16/361,075 - Filed: 2019-03-21 - Published: 2020-09-24 - Granted: 2021-06-08 - Earliest claimed: 2019-03-21 - Assignees: Verb Surgical Inc. - Applicants: Verb Surgical Inc. - Inventors: Jagadish Venkataraman; Dave Scott; Eric Johnson - Web page: https://patentgrep.com/patent/US11026561B2 ## CPC - H04N 7/183: {for receiving images from a single remote source} - A61B 1/00009: {of image signals during a use of endoscope} - A61B 1/000096: {using artificial intelligence} - A61B 1/0004: {for electronic operation} - A61B 1/00045: {Display arrangement} - A61B 1/00055: {for alerting the user} - A61B 1/045: Control thereof - A61B 1/3132: {for laparoscopy} - A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis - A61B 34/25: {User interfaces for surgical systems} - A61B 34/30: Surgical robots - A61B 34/37: Leader-follower robots (A61B34/35 takes precedence) - A61B 90/37: {Surgical systems with images on a monitor during operation} - G06N 20/00: Machine learning - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - H04N 7/0117: {involving conversion of the spatial resolution of the incoming video signal (for graphics images G09G2340/0407)} - H04N 7/185: {from a mobile camera, e.g. for remote control} - H04N 19/59: involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution - A61B 2017/00119: {alarm; indicating an abnormal situation} - A61B 2017/00216: {with eye tracking or head position tracking control} - A61B 2034/2055: {Optical tracking systems} - A61B 2034/2057: {Details of tracking cameras} - A61B 2034/2065: {Tracking using image or pattern recognition} - A61B 2090/373: {using light, e.g. by using optical scanners} - G06V 20/44: {Event detection} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Embodiments described herein provide various examples of displaying video images of a surgical video captured at a first resolution on a screen of a surgical system having a second resolution lower than the first resolution. In one aspect, a process begins by receiving the surgical video and selecting a first portion of the video images having the same or substantially the same resolution as the second resolution. The process subsequently displays the first portion of the video images on the screen. While displaying the first portion of the video images, the process monitors a second portion of the video images not being displayed on the screen for a set of predetermined events, wherein the second portion is not visible to the user. When a predetermined event in the set of predetermined events is detected in the second portion, the process generates an alert to notify the user. ## Claims ### Claim 1 A computer-implemented method for displaying video images of a surgical video captured at a first resolution on a screen of a surgical system having a second resolution lower than the first resolution, the method comprising: for each video image captured at the first resolution, selecting a first portion of the video image having the same resolution as the second resolution of the screen, wherein selecting the first portion of the video image includes keeping the first portion of the video image substantially centered on a tool-tip of a surgical tool displayed in the video image; displaying the first portion of the video image on the screen without changing the resolution of the first portion of the video image; while displaying the first portion of the video image, monitoring a second portion of the video image not being displayed on the screen for a set of predetermined events, wherein the second portion of the video image is not visible to the user; and when a predetermined event in the set of predetermined events is detected in the second portion of the video image, generating an alert to notify the user that an off-screen event has been detected, wherein when the off-screen event is detected, immediately changing the view on the screen from the first portion of the video image to the full video image of the surgical video displayed at a reduced resolution to make both the first portion of the video image and the detected off-screen event visible to the user on the screen. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the first resolution is greater than the second resolution in both the horizontal and the vertical dimensions; and wherein the first portion of the video image is substantially the center portion of the video image. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein while displaying the first portion of the video image, the first portion of the video image occupies the full display area of the screen without leaving any blank space on the screen. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein monitoring the second portion of the video image for the set of predetermined events includes using one or more machine-learning models to process the second portion of the video image in order to detect each of the set of predetermined events. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein generating the alert to notify the user that an off-screen event has been detected includes displaying a warning message on the screen to prompt the user to take appropriate action. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the alert on the screen includes a direction indicator to show the user a direction in which the off-screen event was detected. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein when the off-screen event is detected, the method further comprises immediately changing the view on the screen from the first portion of the video image to a third portion of the video image, which contains the detected off-screen event. ### Claim 8 (depends on 7) The computer-implemented method of claim 7, wherein the third portion of the video image overlaps with the first portion and the second portion of the video image and has the same size as the first portion of the video image. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein after generating the alert to notify the user that an off-screen event has been detected, the method further comprises: providing the user with a selection option to either change the view on the screen from the first portion of the video image to another portion of the video image containing the detected off-screen event or change the view on the screen from the first portion of the video image to the full video image displayed at a reduced resolution; receiving a user selection in response to the selection option; and changing the view on the screen based on the received user selection to make the detected off-screen event visible to the user. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein after generating the alert to notify the user that an off-screen event has been detected, the method further comprises allowing the user to manually change the view on the screen from the first portion of the video image to another portion of the video image containing the detected off-screen event in response to the alert. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein the set of predetermined events includes surgical smoke and bleeding. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein the set of predetermined events includes a set of surgical tool-related events, which further includes: (1) whether the jaws of a surgical tool detected off-screen are closed or open; and (2) whether a tip of a surgical tool detected off-screen is too close to a critical anatomy. ### Claim 13 (depends on 12) The computer-implemented method of claim 12, wherein if the detected predetermined event is one of the set of surgical tool-related events, the method further comprises immediately deactivating functionalities or locking motion of a surgical tool associated with the detected surgical tool-related event. ### Claim 14 (depends on 1) The computer-implemented method of claim 1, wherein the surgical video is one of: an endoscopic surgery video; a laparoscopic surgery video; a robotic surgery video; and an open surgery video. ### Claim 15 (depends on 1) The computer-implemented method of claim 1, wherein keeping the first portion of the video image substantially centered on the tool-tip of the surgical tool displayed in the video image includes: detecting, using a machine-learning-based technique, the surgical tool in the video image; determining a location of the tool-tip of the detected surgical tool; and positioning the first portion of the video image based on the location of the tool-tip. ### Claim 16 A system for displaying a surgical video, comprising: one or more processors; and a screen having a native resolution, wherein the one or more processors are configured to: receive a surgical video comprising a sequence of video images captured at a first resolution for display on the screen, wherein the first resolution is greater than the native resolution of the screen; for each video image in the sequence of video images, select a first portion of the video image having the same resolution as the native resolution of the screen to be displayed on the screen without changing the resolution of the first portion of the video image, wherein selecting the first portion of the video image includes keeping the first portion of the video image substantially centered on a tool-tip of a surgical tool displayed in the video image; while displaying the first portion of the video image, monitor a second portion of the video image not being displayed on the screen for a set of predetermined events, wherein the second portion of the video image is not visible to the user; and if a predetermined event in the set of predetermined events is detected in the second portion of the video image, generate an alert to notify the user that an off-screen event has been detected, wherein when the off-screen event is detected, immediately change the view on the screen from the first portion of the video image to the full video image of the surgical video displayed at a reduced resolution to make both the first portion of the video image and the detected off-screen event visible to the user on the screen. ### Claim 17 (depends on 16) The system of claim 16, wherein when the off-screen event is detected, the one or more processors are further configured to immediately change the view on the screen from the first portion of the video image to a third portion of the video image, which contains the detected off-screen event. ### Claim 18 (depends on 16) The system of claim 16, wherein after generating the alert to notify the user that an off-screen event has been detected, the one or more processors are further configured to: provide the user with a selection option to either change the view on the screen from the first portion of the video image to another portion of the video image containing the detected off-screen event or change the view on the screen from the first portion of the video image to the full video image displayed at a reduced resolution; receive a user selection in response to the selection option; and change the view on the screen based on the received user selection to make the detected off-screen event visible to the user.