# US 10,769,443 B2 · Dominant tool detection system for surgical videos > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,769,443 B2 - Document ID: US10769443B2 - Status: Granted - Kind: B2 - Application number: 16/008,826 - Filed: 2018-06-14 - Published: 2019-12-19 - Granted: 2020-09-08 - Earliest claimed: 2018-06-14 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: Xiaojing Chen; Ko-Kai Albert Huang; Ming-Chang Liu - Web page: https://patentgrep.com/patent/US10769443B2 ## CPC - G06V 20/20: in augmented reality scenes - A61B 90/37: {Surgical systems with images on a monitor during operation} - G06F 18/24: Classification techniques - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - H04N 23/69: Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming - A61B 17/29: Forceps for use in minimally invasive surgery - A61B 17/320016: {Endoscopic cutting instruments, e.g. arthroscopes, resectoscopes (A61B10/0233, A61B17/29, A61B17/3207 take precedence)} - A61B 2090/3612: {with images taken automatically} - A61B 2090/372: {Details of monitor hardware} - G06F 18/22: Matching criteria, e.g. proximity measures - G06T 2207/10016: Video; Image sequence - G06V 2201/034: of medical instruments - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Implementations generally relate to detecting dominant tools in surgical videos. In some implementations, a method includes receiving at least one image frame. The method further includes detecting one or more objects in the at least one image frame. The method further includes classifying the one or more objects into one or more tool classifications, where the one or more objects are tools. The method further includes determining a handedness of the one or more tools. The method further includes determining a dominant tool from the one or more tools based at least in part on the one or more classifications of the one or more tools and based at least in part on the handedness of the one or more tools. ## Claims ### Claim 1 A system comprising: one or more processors; and logic encoded in one or more non-transitory computer-readable storage media for execution by the one or more processors and when executed operable to perform operations comprising: receiving at least one image frame; detecting one or more objects in the at least one image frame; classifying the one or more objects into one or more tool classifications, wherein the one or more objects are one or more tools; determining if each tool of the one or more tools is non-assistive based on the one or more tool classifications; determining a handedness of the one or more tools; and determining a dominant tool from the one or more tools based at least in part on the dominant tool being determined as non-assistive and based at least in part on the handedness of the one or more tools. ### Claim 2 (depends on 1) The system of claim 1, wherein the one or more tool classifications indicate if a particular tool is assistive or non-assistive. ### Claim 3 (depends on 1) The system of claim 1, wherein the one or more tool classifications indicate tool functions. ### Claim 4 (depends on 1) The system of claim 1, wherein the logic when executed is further operable to perform operations comprising: generating a visual indicator for the one or more tools; and demarcating each of the one or more tools in the at least one image frame. ### Claim 5 (depends on 1) The system of claim 1, wherein the logic when executed is further operable to perform operations comprising labeling each tool in the at least one image frame with a type of tool. ### Claim 6 (depends on 1) The system of claim 1, wherein the logic when executed is further operable to perform operations comprising performing the determining of the dominant tool in real-time. ### Claim 7 (depends on 1) The system of claim 1, wherein the logic when executed is further operable to perform operations comprising controlling a camera based at least in part on the determining of the dominant tool. ### Claim 8 A non-transitory computer-readable storage medium with program instructions stored thereon, the program instructions when executed by one or more processors are operable to perform operations comprising: receiving at least one image frame; detecting one or more objects in the at least one image frame; classifying the one or more objects into one or more tool classifications, wherein the one or more objects are one or more tools; determining if each tool of the one or more tools is non-assistive based on the one or more tool classifications; determining a handedness of the one or more tools; and determining a dominant tool from the one or more tools based at least in part on the dominant tool being determined as non-assistive and based at least in part on the handedness of the one or more tools. ### Claim 9 (depends on 8) The computer-readable storage medium of claim 8, wherein the one or more tool classifications indicate types of tools. ### Claim 10 (depends on 8) The computer-readable storage medium of claim 8, wherein the one or more tool classifications indicate tool functions. ### Claim 11 (depends on 8) The computer-readable storage medium of claim 8, wherein the instructions when executed are further operable to perform operations comprising: generating a visual indicator for the one or more tools; and demarcating each of the one or more tools in the at least one image frame. ### Claim 12 (depends on 8) The computer-readable storage medium of claim 8, wherein the instructions when executed are further operable to perform operations comprising labeling each tool in the at least one image frame with a type of tool. ### Claim 13 (depends on 8) The computer-readable storage medium of claim 8, wherein the instructions when executed are further operable to perform operations comprising performing the determining of the dominant tool in real-time. ### Claim 14 (depends on 8) The computer-readable storage medium of claim 8, wherein the instructions when executed are further operable to perform operations comprising controlling a camera based at least in part on the determining of the dominant tool.