# US 12,357,406 B2 · Robotic surgical collision detection systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,357,406 B2 - Document ID: US12357406B2 - Status: Granted - Kind: B2 - Application number: 17/432,644 - Filed: 2020-03-16 - Published: 2022-05-26 - Granted: 2025-07-15 - Earliest claimed: 2019-03-20 - Assignees: Covidien LP - Applicants: Covidien LP - Inventors: Dwight Meglan; Meir Rosenberg; William J. Peine - Web page: https://patentgrep.com/patent/US12357406B2 ## CPC - A61B 34/37: Leader-follower robots (A61B34/35 takes precedence) - B25J 9/1666: {Avoiding collision or forbidden zones} - A61B 34/25: {User interfaces for surgical systems} - A61B 50/13: Trolleys{, e.g. carts} - B25J 9/1676: {Avoiding collision or forbidden zones} - A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis - A61B 34/30: Surgical robots - A61B 34/74: {Manipulators with manual electric input means} - A61B 2034/2055: {Optical tracking systems} - A61B 2034/2057: {Details of tracking cameras} - A61B 2034/2059: {Mechanical position encoders} - A61B 2034/2065: {Tracking using image or pattern recognition} - A61B 2034/302: {specifically adapted for manipulations within body cavities, e.g. within abdominal or thoracic cavities} - A61B 2034/742: {Joysticks} - A61B 2034/743: {Keyboards} - A61B 2034/744: {Mouse} - G05B 2219/40195: Tele-operation, computer assisted manual operation - G05B 2219/40476: Collision, planning for collision free path - G05B 2219/49143: Obstacle, collision avoiding control, move so that no collision occurs - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract Systems and methods for surgical robotic collision detection in accordance with aspects of the present disclosure are disclosed. In various embodiments, a system for surgical robotic collision detection includes a robotic cart having a robotic arm, an imaging device supported by the robotic cart or the robotic arm, the imaging device captures images within a field of vision of the imaging device, and a controller in operable communication with the robotic arm and the imaging device. The controller includes a processor and a memory storing instructions which, when executed by the processor, causes the controller to: receive the images from the imaging device, generate a grid including a first plurality of spatial points from the images, and detect a potential collision within the field of vision based on the generated grid. ## Claims ### Claim 1 A surgical robotic collision detection system, comprising: a robotic cart having a robotic arm; a plurality of imaging devices supported by the robotic cart or the robotic arm, the plurality of imaging devices configured to capture images within a field of vision of the plurality of imaging devices; and a controller in operable communication with the robotic arm and the plurality of imaging devices, the controller having a processor and a memory storing instructions thereon which, when executed by the processor, causes the controller to: receive the images from the plurality of imaging devices; generate a three-dimensional grid including a first plurality of spatial points from the images; and detect a potential collision of objects located within the field of vision based on: the generated three-dimensional grid; the geometric orientation of the objects located within the field of vision; and a sweep volume of movement of each of the objects located within the field of vision, wherein any overlap in the sweep volumes of the objects located within the field of vision provides an indication of the potential collision. ### Claim 2 (depends on 1) The system according to claim 1, wherein at least one imaging device of the plurality of imaging devices is selected from the group consisting of a stereoscopic imaging device, an optical imaging device, a ranging laser device, and an infrared (IR) imaging device. ### Claim 3 (depends on 1) The system according to claim 1, wherein at least one imaging device of the plurality of imaging devices includes a sensor configured to capture a first image at a first time point, the first image including a first object of the objects located within the field of vision in positional relation to the sensor. ### Claim 4 (depends on 3) The system according to claim 3, wherein the memory stores instructions which, when executed by the processor, causes the controller to receive the first image and generate a first depth map based on the first image. ### Claim 5 (depends on 4) The system according to claim 4, wherein the controller generates a first point cloud based on the first depth map, the first point cloud including the first plurality of spatial points contained within the grid. ### Claim 6 (depends on 5) The system according to claim 5, wherein the controller segments the first plurality of spatial points to identify a first spatial point subset of the first point cloud, each spatial point in the first spatial point subset corresponds to a surface of the first object. ### Claim 7 (depends on 6) The system according to claim 6, where the memory includes instructions that, when executed by the processor, causes the controller to: compare the first spatial point subset to a pre-identified configuration of a structure of the first object to identify the first object within the field of vision of the plurality of imaging devices. ### Claim 8 (depends on 3) The system according to claim 3, wherein: the sensor of the at least one imaging device of the plurality of imaging devices captures a second image at a second time point, and the memory further includes instructions that, when executed by the processor, causes the controller to: receive the second image; and generate a second depth map. ### Claim 9 (depends on 8) The system according to claim 8, where the memory further includes instructions that, when executed by the at least one processor, causes the controller to: generate a second point cloud within the coordinate system comprising a second plurality of spatial points, and where the second point cloud is based on the second depth map. ### Claim 10 (depends on 9) The system according to claim 9, where the memory further includes instructions that, when executed by the at least one processor, causes the controller to: segment the second plurality of spatial points to identify a second spatial point subset of the second point cloud and compare the second spatial point subset to the pre-identified configuration of a structure of the objects; match the first spatial point subset in the first point cloud with the second spatial point subset in the second point cloud to orient the first point cloud with the second point cloud; and identify motion of the objects within the field of vision of the plurality of imaging devices based on the orientation of the first point cloud relative to the second point cloud. ### Claim 11 (depends on 10) The system according to claim 10, wherein the memory further includes instructions that, when executed by the one or more processors, causes the controller to determine a spatial trajectory of the objects based upon the identified motion of the objects from the position of the objects in the first point cloud to the position of the objects in the second point cloud. ### Claim 12 (depends on 1) The system according to claim 1, further comprising a display device in communication with the controller, wherein the memory further includes instructions stored thereon which, when executed by the processor, causes the controller to: cause the display device to output an indication of a possible collision based on determining that a possible collision exists. ### Claim 13 (depends on 12) The system according to claim 12, wherein the indication includes a three-dimensional image of a position diagram. ### Claim 14 (depends on 12) The system according to claim 12, wherein the three-dimensional images of the position diagram illustrate a rendering of the possible collision at a later point in time in a case where the objects remain on the spatial trajectory. ### Claim 15 (depends on 1) The system according to claim 1, wherein the memory further includes instructions stored thereon which, when executed by the processor, causes the controller to: transmit a control signal to the robotic cart or the robotic arm to cause the robotic arm to reposition to avoid the possible collision. ### Claim 16 A method for detecting potential collisions between objects of a surgical robotic system including a robotic cart supporting a robotic arm, a plurality of imaging devices supported by the robotic cart or the robotic arm, and a controller, the controller in communication with the plurality of imaging devices and the robotic arm, the method comprising: receiving image data of a field of vision captured by the plurality of imaging devices; generating a three-dimensional depth map including a first plurality of spatial points based on the image data; and detecting a potential collision between objects captured in the image data based on: the generated three-dimensional depth map; the geometric orientation of the objects located within the field of vision; and a sweep volume of movement of each of the objects located within the field of vision, wherein any overlap in the sweep volumes of the objects located within the field of vision provides an indication of the potential collision. ### Claim 17 (depends on 16) The method of claim 16, further comprising generating a first point cloud based on the three-dimensional depth map, the first point cloud comprising the plurality of spatial points. ### Claim 18 (depends on 17) The method of claim 17, further comprising segmenting the plurality of spatial points in the point cloud to identify a spatial point subset of the point cloud, each point in the spatial point subset corresponds to a surface portion of the objects. ### Claim 19 (depends on 18) The method of claim 18, wherein the identifying of the spatial point subset includes matching the spatial point subset to a predetermined configuration of the objects and, based on the matching, identifying the objects. ### Claim 20 (depends on 18) The method of claim 18, further comprising: receiving second image data from the plurality of imaging devices at a point in time later than when the image data was captured by the plurality of imaging devices; generating a second three-dimensional depth map including a second plurality of spatial points based on the second image data; generating a second point cloud comprising a second plurality of spatial points based on the second three-dimensional depth map; segmenting the second plurality of spatial points to identify the spatial point subset within the second point cloud; and matching the spatial point subset in the first point cloud with the spatial point subset in the second point cloud. ### Claim 21 (depends on 20) The method of claim 20, further comprising: determining a spatial trajectory of the objects based on the positional difference of the spatial point subset matched in the first point cloud and the second point cloud; and displaying an indication of a possible collision when a possible collision exists. ### Claim 22 (depends on 21) The method according to claim 21, further comprising: transmitting a control signal to cause the robotic arm to modify motion in a direction to avoid the possible collision.