# US 11,304,763 B2 · Image capturing of the areas outside the abdomen to improve placement and control of a surgical device in use > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,304,763 B2 - Document ID: US11304763B2 - Status: Granted - Kind: B2 - Application number: 16/182,269 - Filed: 2018-11-06 - Published: 2019-07-04 - Granted: 2022-04-19 - Earliest claimed: 2017-12-28 - Assignees: Cilag GmbH International - Applicants: Ethicon LLC - Inventors: Frederick E. Shelton, IV; Jason L. Harris; Chester O. Baxter, III; Mark S. Zeiner - Web page: https://patentgrep.com/patent/US11304763B2 ## CPC - A61B 34/25: {User interfaces for surgical systems} - A61B 18/00: Surgical instruments, devices or methods for transferring non-mechanical forms of energy to or from the body (eye surgery A61F9/007; ear surgery A61F11/00) - A61B 18/1402: {Probes for open surgery} - A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis - A61B 34/35: for telesurgery - A61B 34/37: Leader-follower robots (A61B34/35 takes precedence) - A61B 90/30: Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure - A61B 90/361: {Image-producing devices, e.g. surgical cameras} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215) - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G16H 20/40: relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture - G16H 40/63: for local operation - A61B 17/068: Surgical staplers {, e.g. containing multiple staples or clamps}({staplers containing only one staple A61B17/10; magazines or containers for staples A61B17/105;} for performing anastomosis A61B17/115; {staplers in general B25C5/00}) - A61B 18/1445: {at the distal end of a shaft, e.g. forceps or scissors at the end of a rigid rod} - A61B 2017/00119: {alarm; indicating an abnormal situation} - A61B 2017/00203: {with speech control or speech recognition} - A61B 2017/00207: {with hand gesture control or hand gesture recognition} - A61B 2017/00221: {with wireless transmission of data, e.g. by infrared radiation or radiowaves} - A61B 2018/00994: {combining two or more different kinds of non-mechanical energy or combining one or more non-mechanical energies with ultrasound} - A61B 2018/1253: {monopolar} - A61B 2018/126: {bipolar} - A61B 2034/2051: {Electromagnetic tracking systems} - A61B 2034/2055: {Optical tracking systems} - A61B 2034/2063: {Acoustic tracking systems, e.g. using ultrasound} - A61B 2034/2065: {Tracking using image or pattern recognition} - A61B 2034/2074: {Interface software} - A61B 2034/252: {indicating steps of a surgical procedure} - A61B 2034/254: {being adapted depending on the stage of the surgical procedure} - A61B 2034/256: {having a database of accessory information, e.g. including context sensitive help or scientific articles} - A61B 2090/373: {using light, e.g. by using optical scanners} - A61B 2090/3954: {magnetic, e.g. NMR or MRI} - A61B 2218/002: Irrigation - A61B 2218/007: Aspiration - A61B 2218/008: for smoke evacuation - G06T 2207/20081: Training; Learning - G06V 2201/034: of medical instruments - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA ## Abstract Various systems and methods for controlling surgical devices are disclosed. A computer system, such as a surgical hub, can be configured to be communicably coupled to a surgical device and a camera configured to view an operating room. The computer system can be programmed to receive images of surgical staff members and surgical devices within the operating room during a surgical procedure. The computer system can further determine conditions or characteristics associated with the surgical staff members and surgical devices, such as whether a surgical staff member is performing a particular gesture or the pose of a surgical device, such as a surgical instrument, during a surgical procedure. The computer system can then control the paired surgical devices accordingly. ## Claims ### Claim 1 A situationally aware surgical system configured for use during a surgical procedure, the situationally aware surgical system comprising: a surgical device; a camera configured to image individuals performing activities in an operating room; a processor; and a memory coupled to the processor, the memory storing instructions that, when executed by the processor, cause the processor to: receive an image of an individual within the operating room from the camera; detect whether the individual is performing a predetermined activity based on an imaged physical property of the individual during the normal course of the surgical procedure; determine a next step of the surgical procedure based on the detected activity; correlate the predetermined activity to a procedural outcome stored in the memory; and preemptively control the surgical device according to the determined next step without the individual taking an affirmative action with respect to the surgical device, wherein a control parameter of the surgical device is based, at least in part, on the correlation. ### Claim 2 (depends on 1) The situationally aware surgical system of claim 1, wherein: the surgical device comprises a display; and the instructions stored in the memory, when executed by the processor, further cause the processor to control information displayed on the display according to the imaged physical property. ### Claim 3 (depends on 2) The situationally aware surgical system of claim 2, wherein the surgical device comprises a surgical instrument controlled by the individual, and wherein the information displayed on the display corresponds to the surgical instrument controlled by the individual. ### Claim 4 (depends on 1) The situationally aware surgical system of claim 1, wherein: the surgical device comprises a surgical instrument; and the instructions stored in the memory, when executed by the processor, cause the processor to change an operation of the surgical instrument according to the determined next step. ### Claim 5 (depends on 4) The situationally aware surgical system of claim 4, wherein the surgical instrument is selected from the group consisting of an electrosurgical instrument, an ultrasonic surgical instrument, and a surgical stapling instrument. ### Claim 6 (depends on 1) The situationally aware surgical system of claim 1, wherein the instructions stored in the memory, when executed by the processor, cause the processor to: extract features from the image received from the camera; and determine whether the individual is performing the predetermined activity according to whether the extracted features correspond to the predetermined activity. ### Claim 7 (depends on 1) The situationally aware surgical system of claim 1, wherein the predetermined activity is a passing of the surgical device to the individual. ### Claim 8 (depends on 1) The situationally aware surgical system of claim 1, wherein the procedural outcome is negative, and wherein the control parameter deactivates the surgical device. ### Claim 9 (depends on 1) The situationally aware surgical system of claim 1, wherein, when executed by the processor, the instructions further cause the processor to generate a recommendation for the individual to alter the physical property. ### Claim 10 (depends on 1) The situationally aware surgical system of claim 1, wherein the procedural outcome is positive, and wherein the control parameter activates the surgical device. ### Claim 11 A situationally aware surgical system configured for use during a surgical procedure, the situationally aware surgical system comprising: a surgical device; a camera configured to image individuals performing activities in an operating room; a processor; and a memory coupled to the processor, the memory storing instructions that, when executed by the processor, cause the processor to: receive an image of the surgical device within the operating room from the camera; detect an imaged pose of the surgical device during a normal course of the surgical procedure without requiring an affirmative action of the surgical device; determine a next step of the surgical procedure based on the detected pose; correlate the detected pose to a procedural outcome stored in the memory; and preemptively control the surgical device according to the determined next step without a user of the surgical device taking an affirmative action with respect to the surgical device, wherein a control parameter of the surgical device is based, at least in part, on the correlation. ### Claim 12 (depends on 11) The situationally aware surgical system of claim 11, wherein the instructions stored in the memory, when executed by the processor, further cause the processor to change an operation of the surgical device according to the determined next step. ### Claim 13 (depends on 12) The situationally aware surgical system of claim 12, wherein: the surgical device comprises an end effector; and the operation comprises an orientation of the end effector. ### Claim 14 (depends on 12) The situationally aware surgical system of claim 12, wherein: the surgical device comprises an end effector configured to staple or deliver energy to a tissue according to a control algorithm; and the operation comprises the control algorithm. ### Claim 15 (depends on 11) The situationally aware surgical system of claim 11, wherein the instructions stored in the memory, when executed by the processor, cause the processor to cause the surgical device to display information corresponding to the detected pose. ### Claim 16 (depends on 15) The situationally aware surgical system of claim 15, wherein the displayed information corresponds to a surgical context. ### Claim 17 (depends on 16) The situationally aware surgical system of claim 16, wherein the instructions stored in the memory, when executed by the processor, cause the processor to: receive perioperative data from one or more surgical devices, the one or more surgical devices comprising the surgical device; and determine the surgical context based at least in part on the perioperative data from the one or more surgical devices. ### Claim 18 (depends on 11) The situationally aware surgical system of claim 11, wherein the instructions stored in the memory, when executed by the processor, cause the processor to detect the pose of the surgical device according to a static reference frame associated with the operating room. ### Claim 19 A situationally aware surgical system configured for use during a surgical procedure, the situationally aware surgical system comprising: a surgical device; a camera configured to image individuals performing activities in an operating room; a processor; and a memory coupled to the processor, the memory storing instructions that, when executed by the processor, cause the processor to: receive an image of a surgical device and an individual within the operating room from the camera; detect an imaged pose of the surgical device based on the image according to whether the image is of the surgical device during a normal course of the surgical procedure; detect whether the individual is performing a predetermined activity based on an imaged physical property of the individual; determine a next step of the surgical procedure based on at least one of the detected pose of the surgical device or the detected activity of the individual; correlate the detected pose to a procedural outcome stored in the memory; and preemptively control the surgical device according to the determined next step without the individual taking an affirmative action with respect to the surgical device, wherein a control parameter of the surgical device is based, at least in part, on the correlation. ### Claim 20 (depends on 19) The situationally aware surgical system of claim 19, wherein: the surgical device comprises a display; and the instructions stored in the memory, when executed by the processor, cause the processor to control information displayed on the display according to the determined next step. ### Claim 21 (depends on 19) The situationally aware surgical system of claim 19, wherein: the surgical device comprises a surgical instrument; and the instructions stored in the memory, when executed by the processor, cause the processor to change an operation of the surgical instrument according to the determined next step. ### Claim 22 (depends on 19) The situationally aware surgical system of claim 19, wherein the instructions stored in the memory, when executed by the processor, cause the processor to change an operation of the surgical device according to the determined next step. ### Claim 23 (depends on 19) The situationally aware surgical system of claim 19, wherein the instructions stored in the memory, when executed by the processor, cause the processor to cause the surgical device to display information corresponding to the detected pose. ### Claim 24 (depends on 19) The situationally aware surgical system of claim 19, wherein the instructions stored in the memory, when executed by the processor, cause the processor to detect the pose of the surgical device according to a static reference frame associated with the operating room. ### Claim 25 (depends on 1) The situationally aware surgical system of claim 1, wherein the physical property includes at least one of a posture, an arm position, and a wrist angle, or combinations thereof.