# US 12,108,992 B2 · Systems and methods for tracking a position of a robotically-manipulated surgical instrument > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,108,992 B2 - Document ID: US12108992B2 - Status: Granted - Kind: B2 - Application number: 17/264,257 - Filed: 2019-07-27 - Published: 2021-09-23 - Granted: 2024-10-08 - Earliest claimed: 2018-07-31 - Assignees: Intuitive Surgical Operations, Inc. - Applicants: Intuitive Surgical Operations, Inc. - Inventors: Siddarth Sen; Brian D. Hoffman; Geoffrey A. Richmond - Web page: https://patentgrep.com/patent/US12108992B2 ## CPC - A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis - A61B 34/37: Leader-follower robots (A61B34/35 takes precedence) - A61B 34/70: {Manipulators specially adapted for use in surgery} - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/09: Supervised learning - A61B 2034/2065: {Tracking using image or pattern recognition} - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract An exemplary surgical instrument tracking system includes at least one physical computing device that determines, based on endoscopic imagery of a surgical area and using a trained neural network, an observation for an object of interest depicted in the endoscopic imagery, associates, based on a probabilistic framework and kinematics of a robotically-manipulated surgical instrument located at the surgical area, the observation for the object of interest to the robotically-manipulated surgical instrument, and determines a physical position of the robotically-manipulated surgical instrument at the surgical area based on the kinematics of the robotically-manipulated surgical instrument and the observation associated with the robotically-manipulated surgical instrument. ## Claims ### Claim 1 A surgical instrument tracking system comprising: a memory storing instructions; a processor communicatively coupled to the memory and configured to execute the instructions to: determine, based on imagery of a surgical area, an observation for an object of interest depicted in the imagery; determine a probability of an association of the observation to a robotically-manipulated surgical instrument located at the surgical area based on kinematics of the robotically-manipulated surgical instrument; associate the observation for the object of interest to the robotically-manipulated surgical instrument based on the probability; and determine a physical position of the robotically-manipulated surgical instrument at the surgical area based on the kinematics of the robotically-manipulated surgical instrument and the observation associated with the robotically-manipulated surgical instrument. ### Claim 2 (depends on 1) The system of claim 1, wherein: the imagery comprises a left endoscopic image and a right endoscopic image that is stereoscopic with the left endoscopic image; the observation for the object of interest comprises a left observation and a right observation for the object of interest; and the processor is configured to execute the instructions to: determine the observation for the object of interest by using a trained neural network to determine the left observation for the object of interest based on the left endoscopic image and to determine the right observation for the object of interest based on the right endoscopic image; associate the observation for the object of interest to the robotically-manipulated surgical instrument by associating the left observation to the robotically-manipulated surgical instrument and associating the right observation to the robotically-manipulated surgical instrument; and determine the physical position of the robotically-manipulated surgical instrument at the surgical area by using the left observation and the right observation to determine a three-dimensional (“3D”) position of the object of interest in a 3D space and adjusting a 3D kinematic-based position of the robotically-manipulated surgical instrument in the 3D space based on the 3D position of the object of interest in the 3D space. ### Claim 3 (depends on 1) The system of claim 1, wherein: the imagery comprises a left endoscopic image and a right endoscopic image that is stereoscopic with the left endoscopic image; and the processor is configured to execute the instructions to stereo match the left and right endoscopic images and determine the observation for the object of interest by using a trained neural network to determine the observation for the object of interest based on the stereo matched left and right endoscopic images. ### Claim 4 (depends on 1) The system of claim 1, wherein: the imagery comprises a combination endoscopic image; and the processor is configured to execute the instructions to: generate the combination endoscopic image by combining a first endoscopic image and a second endoscopic image that is stereoscopic to the first endoscopic image; determine the observation for the object of interest by using a trained neural network to determine, based on the combination endoscopic image, a three-dimensional (“3D”) image-based position of the object of interest in a 3D space; and determine the physical position of the robotically-manipulated surgical instrument at the surgical area by adjusting a 3D kinematic-based position of the robotically-manipulated surgical instrument in the 3D space based on the 3D image-based position of the object of interest in the 3D space. ### Claim 5 (depends on 1) The system of claim 1, wherein: the imagery comprises a combination endoscopic image; and the processor is configured to execute the instructions to: generate the combination endoscopic image by combining multiple endoscopic images included in a sequence of endoscopic video frames; determine the observation for the object of interest by using a trained neural network to determine, based on the combination endoscopic image, a three-dimensional (“3D”) image-based position of the object of interest in a 3D space; and determine the physical position of the of the robotically-manipulated surgical instrument at the surgical area by adjusting a 3D kinematic-based position of the robotically-manipulated surgical instrument in the 3D space based on the 3D image-based position of the object of interest in the 3D space. ### Claim 6 (depends on 5) The system of claim 5, wherein the processor is configured to execute the instructions to determine, based on the combination endoscopic image, the 3D image-based position of the object of interest in the 3D space based at least in part on a motion of the object of interest identified from the combination endoscopic image. ### Claim 7 (depends on 1) The system of claim 1, wherein the processor is configured to execute the instructions to determine the physical position of the robotically-manipulated surgical instrument at the surgical area by adjusting a kinematic-based position of the robotically-manipulated surgical instrument based on an image-based position of the object of interest indicated by the observation. ### Claim 8 (depends on 1) The system of claim 1, wherein the processor is configured to execute the instructions to: determine, based on the imagery of the surgical area, an additional observation for an additional object of interest depicted in the imagery; determine a probability of an association of the additional observation to the robotically-manipulated surgical instrument located at the surgical area based on the kinematics of the robotically-manipulated surgical instrument; associate the additional observation for the additional object of interest to a false positive designation based on the probability of the association of the additional observation to the robotically-manipulated surgical instrument; and refrain from using the additional observation for the additional object of interest to determine the physical position of the robotically-manipulated surgical instrument. ### Claim 9 (depends on 1) The system of claim 1, wherein the processor is configured to execute the instructions to associate the observation for the object of interest to the robotically-manipulated surgical instrument by: determining a first probability for an association of the observation for the object of interest to the robotically-manipulated surgical instrument; determining a second probability for an association of the observation for the object of interest to a false positive designation; determining that the first probability is higher than the second probability; and associating the observation for the object of interest to the robotically-manipulated surgical instrument in response to the determining that the first probability is higher than the second probability. ### Claim 10 (depends on 1) The system of claim 1, wherein the processor is configured to execute the instructions to: determine, based on the imagery of the surgical area, an additional observation for an additional object of interest depicted in the imagery; determine a probability of an association of the additional observation to the robotically-manipulated surgical instrument located at the surgical area based on the kinematics of the robotically-manipulated surgical instrument; associate the additional observation for the additional object of interest to a false positive designation based on the probability of the association of the additional observation to the robotically-manipulated surgical instrument; and wherein the processor is configured to execute the instructions to associate the observation for the object of interest to the robotically-manipulated surgical instrument and associate the additional observation for the additional object of interest to the false positive designation by: determining a first probability for a first set of associations that includes an association of the observation for the object of interest to the robotically-manipulated surgical instrument and an association of the additional observation for the additional object of interest to the false positive designation; determining a second probability for a second set of associations that includes an association of the observation for the object of interest to the false positive designation and an association of the additional observation for the additional object of interest to the robotically-manipulated surgical instrument; determining that the first probability is higher than the second probability; and associating the observation for the object of interest to the robotically-manipulated surgical instrument and associating the additional observation for the additional object of interest to the false positive designation in response to the determining that the first probability is higher than the second probability. ### Claim 11 (depends on 1) The system of claim 1, wherein the observation for the object of interest indicates an image-based position of the object of interest and an additional attribute associated with the object of interest. ### Claim 12 (depends on 11) The system of claim 11, wherein the additional attribute associated with the object of interest comprises one of: a type of surgical instrument associated with the object of interest; an orientation associated with the object of interest; and a motion cue associated with the object of interest. ### Claim 13 (depends on 1) The system of claim 1, wherein the processor is configured to execute the instructions to output data representative of the physical position for use by a robotic surgical system to display user interface content collocated with a visual representation of the robotically-manipulated surgical instrument displayed on a display screen. ### Claim 14 A surgical instrument tracking system comprising: a memory storing instructions; a processor communicatively coupled to the memory and configured to execute the instructions to: determine, based on endoscopic imagery of a surgical area and using a trained neural network, an observation for an object of interest depicted in the endoscopic imagery; determine a probability of an association of the observation to a robotically-manipulated surgical instrument located at the surgical area based on kinematics of the robotically-manipulated surgical instrument; associate, based on the probability, the observation for the object of interest to the robotically-manipulated surgical instrument or to a false positive designation; when the observation for the object of interest is associated with the robotically-manipulated surgical instrument, determine a physical position of the robotically-manipulated surgical instrument at the surgical area by adjusting a kinematic-based position of the robotically-manipulated surgical instrument based on an image-based position of the observation associated with the robotically-manipulated surgical instrument; and when the observation for the object of interest is associated with the false positive designation, refrain from using the observation to determine the physical position of the robotically-manipulated surgical instrument at the surgical area. ### Claim 15 A method comprising: determining, by a surgical instrument tracking system, based on imagery of a surgical area, an observation for an object of interest depicted in the imagery; determining, by the surgical instrument tracking system, a probability of an association of the observation to a robotically-manipulated surgical instrument located at the surgical area based on kinematics of the robotically-manipulated surgical instrument; associating, by the surgical instrument tracking system, the observation for the object of interest to the robotically-manipulated surgical instrument based on the probability; and determining, by the surgical instrument tracking system, a physical position of the robotically-manipulated surgical instrument at the surgical area based on the kinematics of the robotically-manipulated surgical instrument and the observation associated with the robotically-manipulated surgical instrument. ### Claim 16 (depends on 15) The method of claim 15, wherein: the determining of the observation for the object of interest comprises using a trained neural network to determine a left observation for the object of interest based on a left image included in the imagery and to determine a right observation for the object of interest based on a right image included in the imagery; the associating of the observation for the object of interest to the robotically-manipulated surgical instrument comprises associating the left observation to the robotically-manipulated surgical instrument and associating the right observation to the robotically-manipulated surgical instrument; and the determining of the physical position of the robotically-manipulated surgical instrument at the surgical area comprises using the left observation and the right observation to determine a three-dimensional (“3D”) position of the object of interest in a 3D space and adjusting a 3D kinematic-based position of the robotically-manipulated surgical instrument in the 3D space based on the 3D position of the object of interest in the 3D space. ### Claim 17 (depends on 15) The method of claim 15, wherein: the determining of the observation for the object of interest comprises using a trained neural network to determine, based on a combination image that is included in the imagery and that is formed by combining a first image and a second image that is stereoscopic to the first image, a three-dimensional (“3D”) image-based position of the object of interest in a 3D space; and the determining of the physical position of the robotically-manipulated surgical instrument at the surgical area comprises adjusting a 3D kinematic-based position of the robotically-manipulated surgical instrument in the 3D space based on the 3D image-based position of the object of interest in the 3D space. ### Claim 18 (depends on 15) The method of claim 15, wherein: the determining of the observation for the object of interest comprises using a trained neural network to determine, based on a combination image that is included in the imagery and that is formed by combining a sequence of video frames, a three-dimensional (“3D”) image-based position of the object of interest in a 3D space; and the determining of the physical position of the robotically-manipulated surgical instrument at the surgical area comprises adjusting a 3D kinematic-based position of the robotically-manipulated surgical instrument in the 3D space based on the 3D image-based position of the object of interest in the 3D space. ### Claim 19 (depends on 15) The method of claim 15, wherein the determining of the physical position of the robotically-manipulated surgical instrument at the surgical area comprises adjusting a kinematic-based position of the robotically-manipulated surgical instrument based on an image-based position of the object of interest indicated by the observation. ### Claim 20 (depends on 15) The method of claim 15, further comprising registering, based on the imagery, the robotically-manipulated surgical instrument, which is mounted on a first cart, with an endoscope that captures the imagery, the endoscope mounted on a second cart that is separate from the first cart, the registering comprising determining a physical relationship between the first cart and the second cart.