# US 2010/0168763 A1 · Configuration marker design and detection for instrument tracking

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2010/0168763 A1
- Document ID: US20100168763A1
- Status: Published application
- Kind: A1
- Application number: 12/428,691
- Filed: 2009-04-23
- Published: 2010-07-01
- Earliest claimed: 2008-12-31
- Assignees: Intuitive Surgical, Inc.
- Inventors: TAO ZHAO; WENYI ZHAO; WILLIAM C. NOWLIN
- Web page: https://patentgrep.com/patent/US20100168763A1

## CPC

- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 34/30: Surgical robots
- A61B 34/37: Leader-follower robots (A61B34/35 takes precedence)
- A61B 90/94: coded with symbols, e.g. text
- A61B 2034/102: {Modelling of surgical devices, implants or prosthesis}
- A61B 2034/2065: {Tracking using image or pattern recognition}
- A61B 2090/371: {with simultaneous use of two cameras}
- A61B 2090/3983: {Reference marker arrangements for use with image guided surgery}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)

## Abstract

The present disclosure relates to systems, methods, and tools for tool tracking using image-derived data from one or more tool located reference features. A method includes: directing illuminating light from a light source onto a robotic surgical tool within a patient body, wherein the tool includes a plurality of primitive features having known positions on the tool, and wherein each feature includes a spherical reflective surface; capturing stereo images of the plurality of primitive features when the tool is within the patient body, wherein the stereo images are captured by an image capture device adjacent the illumination source so that the illumination light reflected from the imaged primitive features toward the image capture device substantially aligns with spherical centers of the surfaces of the imaged primitive features, and determining a position for the tool by processing the stereo images so as to locate the spherical centers of the imaged primitive features by using the reflected light.

## Claims

### Claim 1

A surgical robotic tool tracking method comprising:
directing illuminating light from a light source onto a robotic surgical tool within a patient body, wherein the tool includes a plurality of primitive features having known positions on the tool, and wherein each feature includes a spherical reflective surface:
capturing stereo images of a plurality of the primitive features when the tool is within the patient body, wherein the stereo images are captured by a stereo image capture device adjacent the illumination source so that the illumination light reflected from the imaged primitive features toward the image capture device substantially aligns with spherical centers of the surfaces of the imaged primitive features; and
determining a position for the tool by processing the stereo images so as to locate the spherical centers of the imaged primitive features by using the reflected light.

### Claim 2 (depends on 1)

A method as in claim 1, wherein the illuminating light includes a visible light spectrum.

### Claim 3 (depends on 1)

A method as in claim 1, wherein determining a position for the tool by processing the image is accomplished so as to identify at least one of the primitive features by using specular reflected light.

### Claim 4 (depends on 1)

A method as in claim 1, further comprising processing said stereo images so as to determine three-dimensional positional data for the spherical centers of the imaged primitive features.

### Claim 5 (depends on 4)

A method as in claim 4, wherein a constellation algorithm is used to identify a pattern of the primitive features in the first image.

### Claim 6 (depends on 5)

A method as in claim 5, further comprising:
generating an estimated tool state for the tool by using at least one prior tool state from a prior image of the tool or joint data from a robotic actuation system effecting movement of the tool; and
using the estimated tool state in the constellation algorithm.

### Claim 7 (depends on 4)

A method as in claim 4, further comprising:
capturing stereo images for a plurality of time points;
generating an estimated tool state for the plurality of time points; and
rejecting any incompatible pattern detection by using a robust estimation technique.

### Claim 8 (depends on 7)

A method as in claim 7, wherein the robust estimation technique used is a Random Sample Consensus (RANSAC) technique.

### Claim 9 (depends on 4)

A method as in claim 4, wherein a model based image signature is used in the identification of a primitive feature in said images.

### Claim 10 (depends on 4)

A method as in claim 4, further comprising:
processing the stereo images so as to identify a natural feature of the tool in both of said images;
determining a three-dimensional position for the identified natural feature; and
determining an image-derived tool state by using the three-dimensional position for the natural feature in combination with the three-dimensional positional data for the imaged primitive features.

### Claim 11 (depends on 1)

A method as in claim 1, further comprising:
generating an estimated tool state for the tool by using at least one prior tool state from a prior image of the tool or joint data from a robotic actuation system effecting movement of the tool; and
using the estimated tool state to reject an incompatible pattern detection.

### Claim 12 (depends on 1)

A method as in claim 1, wherein at least one of the primitive features comprises a convex or concave spherical reflective surface aligned with a joint axis of the tool, the surface being defined by a joint structure.

### Claim 13

A robotic surgery system, comprising:
a robotic surgical tool having a plurality of primitive features having know positions on the tool, and wherein each feature comprises a spherical reflective surface;
a light source oriented to transmit illumination light within a patient body;
a stereo image capture device adjacent the illumination source so that the illumination light reflected from the primitive features toward the image capture device substantially aligns with a spherical centers of the spherical surfaces; and
a processor coupled with the image capture device and configured for determining a position for the tool by processing stereo images so as to locate the spherical centers of imaged primitive features by using the reflected light.

### Claim 14 (depends on 13)

A system as in claim 13, further comprising a tangible medium comprising machine-readable instructions executable by said processor for processing said stereo images.

### Claim 15 (depends on 13)

A system as in claim 13, wherein the processor is configured to determine a position for the tool by processing the stereo images so as to identify at least one of the primitive features by using specular reflected light.

### Claim 16 (depends on 13)

A system as in claim 13, wherein one of the primitive features is aligned with a joint axis of the tool, the surface being defined by a joint structure.

### Claim 17 (depends on 13)

A system as in claim 13, wherein the processor is configured so as to determine three-dimensional positional data for the spherical centers of the imaged primitive features by processing the stereo images.

### Claim 18 (depends on 17)

A system as in claim 17, wherein the imaging device comprises a stereoscopic endoscope.

### Claim 19 (depends on 13)

A system as in claim 13, wherein the spherical reflective surface comprises a convex or concave surface.

### Claim 20 (depends on 13)

A system as in claim 13, wherein the plurality of features includes three or more features.

### Claim 21 (depends on 20)

A system as in claim 20, wherein the plurality of features includes four or more features.

### Claim 22

A surgical tool for use With a robotic surgery system, the surgery system including a stereo imaging device for capturing stereo images of the surgical tool during use and a processor coupled with the imaging device for processing the captured stereo images so as to determine image-derived positional information for the surgical tool, the surgical tool comprising a plurality of primitive features with each primitive feature comprising a spherical reflective surface.

### Claim 23

An object tracking system, comprising:
an object having a plurality of primitive features with each primitive feature comprising a spherical reflective surface;
a light source oriented to transmit illumination light toward the object;
a stereo image capture device for capturing stereo images of said object, the image device being disposed adjacent the illumination source so that illumination light reflected from a plurality of primitive features toward the image capture device substantially aligns with spherical centers of the spherical surfaces, the image device outputting image data for said stereo images; and
a processor coupled with the image capture device and configured to process said image data so as to:
determine three-dimensional position data for three or more of the imaged primitive features; and
determine a position for the tool by processing the three-dimensional position data.

### Claim 24

A method for estimating the pose of a surgical tool comprising three or more substantially corner-less primitive features having known positions on the tool, the method comprising:
using a stereoscopic endoscope to capture stereo images of three or more of the primitive features, the stereo images including a first image and a second image;
extracting at least three of the primitive features from the first image;
extracting at least three of the primitive features from the second image;
determining correspondences between extracted primitive features by using image signatures;
using the determined correspondences to determine three-dimension positions for at least three of the primitive features;
identifying a pattern in the extracted primitive features that corresponds to a pattern of the tool primitive features; and
estimating a pose for the surgical tool by using the identified pattern and the determined three-dimensional positions.
