# US 2010/0168562 A1 · Fiducial marker design and detection for locating surgical instrument in images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2010/0168562 A1
- Document ID: US20100168562A1
- Status: Published application
- Kind: A1
- Application number: 12/428,657
- Filed: 2009-04-23
- Published: 2010-07-01
- Earliest claimed: 2008-12-31
- Assignees: Intuitive Surgical, Inc.
- Inventors: Tao Zhao; Wenyi Zhao; Daniel J. Halabe; Brian D. Hoffman; William C Nowlin
- Web page: https://patentgrep.com/patent/US20100168562A1

## CPC

- A61B 34/30: Surgical robots
- A61B 34/37: Leader-follower robots (A61B34/35 takes precedence)
- A61B 90/94: coded with symbols, e.g. text
- A61B 2034/2055: {Optical tracking systems}
- A61B 2034/2065: {Tracking using image or pattern recognition}
- A61B 2090/3937: {Visible markers}
- A61B 2090/3983: {Reference marker arrangements for use with image guided surgery}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)

## Abstract

The present disclosure relates to systems, methods, and tools for tool tracking using image-derived data from one or more tool-located reference features. A method includes: capturing a first image of a tool that includes multiple features that define a first marker, where at least one of the features of the first marker includes an identification feature; determining a position for the first marker by processing the first image; determining an identification for the first marker by using the at least one identification feature by processing the first image; and determining a tool state for the tool by using the position and the identification of the first marker.

## Claims

### Claim 1

A robotic surgical method comprising:
capturing a first image of a surgical tool, the surgical tool including a plurality of features defining a first marker, at least one of the features of the first marker comprising an identification feature;
determining a position for the first marker by processing the captured first image;
determining an identification of the first marker by using the at least one identification feature by processing the first image; and
determining a tool state for the imaged tool by using said position for the first marker and said identification of the first marker.

### Claim 2 (depends on 1)

A method as in claim 1, wherein the first marker comprises redundant features defining error-checking data, the method further comprising:
processing the first image to detect the redundant features and read the error-checking data; and
validating the identification of the first marker by verifying that the first marker identification is consistent with the error-checking data.

### Claim 3 (depends on 2)

A method as in claim 2, wherein the redundant features define check sum data, the method further comprising:
processing the first image to read the check sum data; and
validating the identification of the first marker by verifying that the first marker identification is consistent with the check sum data.

### Claim 4 (depends on 1)

A method as in claim 1, wherein the tool includes a plurality of markers, each marker having at least one identification feature associated with an identification of the marker, the identifications of the markers of the tool differing from each other, wherein the processing of the first image is performed bar a processor having data indicating, for each marker, an associated predetermined positional relationship between the marker and a joint of the surgical tool, and wherein determining the tool state comprises determining a position of the tool by using the position of the first marker, the identification of the first marker, and the associated positional relationship data for the first marker.

### Claim 5 (depends on 4)

A method as in claim 4, wherein the first image includes a second marker of the tool, the method further comprising:
determining a position for the second marker by processing the first image;
determining the identification of the second marker by processing the first image; and
determining a tool state for the imaged tool by using said position for the second marker, said identification of the second marker, and said predetermined positional relationship data associated with the second marker.

### Claim 6 (depends on 4)

A method as in claim 4, wherein the tool includes a second marker, the second marker obscured in the first image, the method further comprising:
moving the tool after determining the tool state by using the first marker;
capturing a second image of the moved tool, the second image encompassing the second marker, the first marker being obscured in the second image;
determining a position for the second marker by processing the second image;
determining the identification of the second marker by processing the second image; and
determining a moved tool state for the tool by using said position for the second marker, said identification of the second marker, and said predetermined positional relationship data associated with the second marker.

### Claim 7 (depends on 1)

A method as in claim 1, further comprising:
capturing a second image of the surgical tool, wherein the first and second images are captured by using a stereoscopic endoscope; and
determining three-dimensional positional data for the first marker by processing the first and second images, wherein the tool state is determine in three dimensions or more.

### Claim 8 (depends on 1)

A method as in claim 1, wherein the surgical tool includes a plurality of markers, each marker having at least one localizer feature and at least one identification feature with a known positional relationship relative to the at least one localizer feature, wherein the position of the first marker is determined by using said at least one localizer feature, and wherein the identification of the first marker is determined by identifying said at least one localizer feature of the first marker and by reading said at least one identification feature according to the known positional relationship between the localizer feature and said at least one identification feature.

### Claim 9 (depends on 1)

A method as in claim 1, wherein the plurality of features includes at least one localizer feature, and wherein the determining an identification of the first marker step comprises:
processing the first image so as to identify the at least one localizer feature;
selecting a candidate identity for the first marker;
generating a candidate view of a marker having the candidate identity by using the identified at least one localizer feature; and
comparing the candidate view with the first image so as to verify that the selected candidate identity is the first marker identity.

### Claim 10 (depends on 9)

A method as in claim 9, wherein the selecting a candidate identity for the first marker step comprises generating an estimated pose for the surgical tool by using at least one prior tool state from a prior image of the tool or joint data from a robotic actuation system effecting movement of the tool, wherein the selected candidate identity results in a candidate pose for the surgical tool that is within a predetermined deviation of the estimated pose for the surgical tool.

### Claim 11 (depends on 10)

A method as in claim 10, wherein the first image encompasses another surgical tool, each tool having an identity, and further comprising associating an identity with the imaged tool having the first marker by verifying that the candidate identity for the first marker results in a candidate pose that is within a predetermined deviation of the estimated pose for the surgical tool having the first marker.

### Claim 12 (depends on 1)

A method as in claim 1, wherein a Maximum Stable Extremal Region (MSER) or adaptive thresholding is used to determine positional data for the first marker.

### Claim 13

A robotic surgery system, comprising:
a surgical tool having a plurality of features defining a first marker, at least one of the features of the first marker comprising an identification feature;
an imaging device for capturing a first image of said tool during use and output first image data in response thereto; and
a processor coupled with the imaging device, the processor adapted to process said captured first image data so as to:
determine positional data for the first marker;
determine an identification of the first marker by using the identification feature; and
determine tool state data for the imaged tool by using said first marker positional data and said identification of the first marker.

### Claim 14 (depends on 13)

A system as in claim 13, further comprising a tangible medium comprising machine-readable instructions executable by said processor for processing said captured first image.

### Claim 15 (depends on 13)

A system as in claim 13, further comprising an input for non-endoscopically derived tool state data, the non-endoscopically derived tool state data derived from robotic joints supporting the tool, and wherein the processor is further configured to process the non-endoscopically derived tool state information and the image-derived tool state information for tracking the state of the tool.

### Claim 16 (depends on 13)

A system as in claim 13, wherein the imaging device is adapted to capture a second image at substantially the same time as the first image, and output second image data in response thereto, and wherein the processor is further configured so as to determine three-dimensional positional data for the first marker by processing the first and second image data.

### Claim 17 (depends on 16)

A system as in claim 16, wherein the imaging device comprises a stereoscopic endoscope.

### Claim 18 (depends on 13)

A system as in claim 13, wherein said first marker comprises redundant features defining error-checking data, and wherein the processor is further configured to process said first image data so as to:
detect the first marker redundant features;
read the error-checking data; and
validate the identification of the first marker by verifying that the first marker identification is consistent with the error-checking data.

### Claim 19 (depends on 18)

A system as in claim 18, wherein the redundant features define check-sum data, and wherein the processor is further configured to process said first image data so as to read the check-sum data; and validate the identification of the first marker by verifying that the first marker identification is consistent with the check-sum data.

### Claim 20 (depends on 13)

A system as in claim 13, wherein a plurality of markers are distributed around the tool, wherein the processor comprises data for each marker indicating an associated marker identification and an associated positional relationship between the marker and a joint of the surgical tool, and wherein the markers have identification features that differ sufficiently for the processor to determine the identification of the markers encompassed within the first image.

### Claim 21 (depends on 20)

A system as in claim 20, wherein at least one of the markers comprises at least one localizer feature that is shared with an adjacent marker.

### Claim 22 (depends on 20)

A system as in claim 20, wherein the features of each of the markers are arranged in a two-dimensional pattern.

### Claim 23 (depends on 22)

A system as in claim 22, wherein each of the markers uses a plurality of circles as localizer features.

### Claim 24 (depends on 22)

A system as in claim 22, wherein each of the markers uses a plurality of corners as localizer features.

### Claim 25 (depends on 24)

A system as in claim 24, wherein the corners comprise saddle points.

### Claim 26 (depends on 22)

A system as in claim 22, wherein each of the markers comprises three localizer features.

### Claim 27 (depends on 22)

A system as in claim 22, wherein each of the markers comprises four localizer features.

### Claim 28 (depends on 22)

A system as in claim 22, wherein each of the markers comprises four circles and a bar as localizer features.

### Claim 29 (depends on 13)

A system as in claim 13, wherein the first marker comprises text.

### Claim 30 (depends on 29)

A system as in claim 29, wherein the first marker comprises text modified to increase positional data or discriminative features.

### Claim 31 (depends on 13)

A system as in claim 13, wherein the processor is coupled with a display and the processor outputs modified first image data to the display that alters a portion of the first image corresponding to a reference feature.

### Claim 32

A surgical tool for use with a robotic surgery system, the surgery system including an imaging device for capturing an image of the surgical tool during use and a processor coupled with the imaging device for processing the captured image so as to determine image-derived positional information for the surgical tool, the surgical tool comprising a plurality of markers, each marker having at least one identification feature, the identification features of each marker differing sufficiently for the surgery system to discriminate between the markers based on images encompassing the markers.

### Claim 33

A robotic surgical method comprising:
capturing a first image of a surgical tool, the surgical tool including a plurality of features defining a plurality of markers, each marker having a predetermined positional relationship with the surgical tool, the first image including one of the markers;
determining a position for the imaged marker by processing the first image;
generating an estimated tool state for the tool by using at least one prior tool state from a prior image of the tool or joint data from a robotic actuation system effectuating movement of the tool; and
determining a tool state for the tool by using said position for the imaged marker, said predetermined positional relationship between the surgical tool and the imaged marker, and said estimated tool state for the tool.
