# US 2010/0164950 A1 · Efficient 3-d telestration for local robotic proctoring

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2010/0164950 A1
- Document ID: US20100164950A1
- Status: Published application
- Kind: A1
- Application number: 12/465,020
- Filed: 2009-05-13
- Published: 2010-07-01
- Earliest claimed: 2008-12-31
- Assignees: Intuitive Surgical, Inc.
- Inventors: Wenyi Zhao; Chenyu Wu; David Hirvonen; Christopher J. Hasser; Brian E. Miller; Catherine J. Mohr; Myriam J. Curet; Tao Zhao; Simon Dimaio; Brian D. Hoffman
- Web page: https://patentgrep.com/patent/US20100164950A1

## CPC

- A61B 34/25: {User interfaces for surgical systems}
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- G06T 7/593: from stereo images
- G06T 11/00: Two-dimensional [2D] image generation
- H04N 13/20: Image signal generators
- A61B 2034/2057: {Details of tracking cameras}
- A61B 2034/2065: {Tracking using image or pattern recognition}
- A61B 2090/371: {with simultaneous use of two cameras}
- A61B 2090/373: {using light, e.g. by using optical scanners}
- G06T 2207/10068: Endoscopic image
- G06T 2207/20096: Interactive definition of curve of interest
- G06T 2207/20101: Interactive definition of point of interest, landmark or seed
- G06T 2207/30004: Biomedical image processing
- H04N 13/10: Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N 13/30: Image reproducers (optical systems for producing stereoscopic or other three-dimensional [3D] effects G02B30/00)
- H04N 2013/0092: {Image segmentation from stereoscopic image signals}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An apparatus is configured to show telestration in 3-D to a surgeon in real time. A proctor is shown one side of a stereo image pair, such that the proctor can draw a telestration line on the one side with an input device. Points of interest are identified for matching to the other side of the stereo image pair. In response to the identified points of interest, regions and features are identified and used to match the points of interest to the other side. Regions can be used to match the points of interest. Features of the first image can be matched to the second image and used to match the points of interest to the second image, for example when the confidence scores for the regions are below a threshold value. Constraints can be used to evaluate the matched points of interest, for example by excluding bad points.

## Claims

### Claim 1

A method of three-dimensional telestration for a user, the method comprising:
identifying selected points of interest of a first image;
displaying the first image on a user display with a first telestration mark at the selected points of interest;
selectively matching the selected points of interest to a second image to determine matched points of interest in response to the identification of the selected points of interest; and
displaying the second image on the user display with a second telestration mark at the matched points of interest such that the first telestration mark and the second telestration mark are presented to appear in three dimensions to the user.

### Claim 2 (depends on 1)

The method of claim 1, wherein the selected points of interest of the first image are matched to the second image without a disparity map, so that no matching of the first image to the second image occurs throughout most of the first image and most of the second image.

### Claim 3 (depends on 1)

The method of claim 1 further comprising inputting points of interest of the first image with an input device to draw the first telestration mark, and wherein the selected points of interest are identified from the points of interest.

### Claim 4 (depends on 1)

The method of claim 1, wherein identifying selected points of interest comprises at least one of selecting raw data points, fitting raw data points to a curve, or interpolating raw data points.

### Claim 5 (depends on 1)

The method of claim 1, wherein identifying the selected points of interest comprises identifying the raw data points, such that the identified raw data points of the first image are matched to the second image.

### Claim 6 (depends on 1)

The method of claim 1, wherein identifying the points of interest comprises fitting the raw data points to a curve such that the curve comprises the selected points of interest.

### Claim 7 (depends on 6)

The method of claim 6, wherein the curve comprises a spline.

### Claim 8 (depends on 1)

The method of claim 1, wherein identifying the points of interest comprises interpolating the raw data points to determine an interpolated data point of the first image, such that the interpolated data point of the first image is matched to the second image.

### Claim 9 (depends on 1)

The method of claim 1, wherein selectively matching comprises at least one of selectively matching regions, selectively matching features, selectively interpolating features, or selectively interpolating previously matched points of interest.

### Claim 10 (depends on 9)

The method of claim 9:
wherein region match confidence scores are determined when the regions are selectively matched;
wherein feature match scores are determined when the features are selectively matched;
wherein the features are selectively matched in response to the region match confidence scores;
wherein the features are selectively interpolated in response to the feature match scores; and
wherein the previously matched points of interest are selectively interpolated in response to the feature match scores;
such that locations a first portion of the matched points of interest are determined in response high region match confidence scores, a second portion of the matched points of interest are determined in response to successful feature match scores, and a third portion of the matched points of interest are determined in response to unsuccessful feature match scores.

### Claim 11 (depends on 9)

The method of claim 9:
wherein selectively matching comprises selectively matching the regions, in which the regions are identified from the first image in response to the selected points of interest and corresponding matched regions of the second image are determined from the regions; and
wherein the regions are matched to the corresponding matched regions to determine locations of the matched points of interest.

### Claim 12 (depends on 11)

The method of claim 11, wherein each of the regions from the first image comprises a portion of the first image so as to include at least one of the selected points of interest within the region, each of the corresponding matched regions comprising a portion of the second image so as to include at least one of the matched points of interest within the corresponding matched region of the second image.

### Claim 13 (depends on 11)

The method of claim 11:
wherein region match confidence scores are determined for each of the regions matched to the second image; and
wherein the features are selectively matched for each of the regions having a low confidence score.

### Claim 14 (depends on 11)

The method of claim 11:
wherein selectively matching comprises selectively matching the features;
wherein selectively matching the features comprises identifying the features in response to the selected point of interest; and
wherein the features are matched to the second image to determine the matched points of interest.

### Claim 15 (depends on 14)

The method of claim 14:
wherein each of the features is identified with a descriptor comprising at least one vector having a location, an orientation, and a magnitude; and
wherein the or each vector is determined in response to a gradient of intensities of pixels of the first image.

### Claim 16 (depends on 14)

The method of claim 14:
wherein feature match confidence scores are determined for each of the features matched to the second image; and
wherein successfully matched features are interpolated to determine matched points of interest.

### Claim 17 (depends on 14)

The method of claim 14:
wherein each region comprises a portion of the image; and
wherein at least some of the features are located within insufficiently matched regions.

### Claim 18 (depends on 1)

The method of claim 1, wherein the identified points of interest are matched to the second image with a constraint comprising at least one of a focus constraint or an epi-polar constraint.

### Claim 19 (depends on 1)

The method of claim 1:
wherein the first image and the second image comprise a pair of real time stereoscopic images from a robotic surgery system; and
wherein the pair of stereoscopic images are shown to appear in three dimensions to the user.

### Claim 20 (depends on 1)

The method of claim 1 wherein the first image comprises a first series of real time digital images and the second image comprises a second series of real time digital images, such that the identified points of interest correspond to identified points of the first series of real time digital images and the matched points of interest correspond to matched points of the second series of real time digital images.

### Claim 21 (depends on 20)

The method of claim 20, wherein a location of the second image series corresponding to a first matched point of interest of the first series is determined before a second point of interest is identified from the first series.

### Claim 22 (depends on 21)

The method of claim 21, further comprising:
sequentially highlighting the points of interest of the first series; and
determining the corresponding locations of the matched points of interest of second series while selecting the points of interest of the first series.

### Claim 23 (depends on 1)

The method of claim 1:
wherein a first portion the selected points of interest are successfully matched to the second image to determine successfully matched points of interest of the second image;
wherein a second portion of the selected points of interest are not successfully matched to the second image and comprise unmatched points of interest of the first image; and
wherein the matched points of interest are shown to the user such that the user sees the selected points of interest corresponding to the matched points of interest in three dimensions and such that the user sees selected points of interest corresponding to the unmatched points of interest without three-dimensional viewing.

### Claim 24 (depends on 1)

The method of claim 1:
wherein the marks fade automatically after a period of time, in which the period of time is programmable by the user;
wherein the period of time is within a range from about one second to about ten seconds; and
wherein the marks are erased by at least one of the user or a second user before the marks fade.

### Claim 25 (depends on 1)

The method of claim 1:
wherein a second user inputs telestration marks on the first image, a first portion of the first image corresponds to a valid matching region for matching to the second image, and a second portion of the first image outside the matching region corresponds to a non-matching region;
wherein the telestration marks within the matching region are selectively identified for matching to the second image and the telestration marks within the non-matching region are not selectively identified for matching to the second image; and
wherein the telestration marks within the second portion are shown with the first image on the display and not shown with the second image on the display.

### Claim 26

A three-dimensional telestration display method comprising:
capturing a stereoscopic pair of images comprising a first image and a second image;
receiving a telestration input for the first image;
identifying selected first points of interest of the first image from the telestration input;
matching the selected first points of interest in the first image to corresponding second points of interest in the second image; and
outputting to a stereoscopic display;
wherein the display comprises the first image with a first telestration mark associated with the first selected points of interest;
wherein the display comprises the second image with a second telestration mark associated with the second points of interest;
wherein the first and second images appear as a three-dimensional image in the display; and
wherein the first and second telestration marks appear as a three-dimensional telestration mark in the display.

### Claim 27 (depends on 26)

The method of claim 26, wherein the first and second telestration marks appear to a viewer as a single three-dimensional mark in the display.

### Claim 28

An apparatus to show three-dimensional telestration a user, the apparatus comprising:
a source of images comprising a first image and a second image;
a first user display comprising a first display and a second display to present the first image and the second image in three-dimensional appearance to the user;
a second user display configured to show at least one of the first image or the second image to a second user;
a second user input device configured to receive input from the second user to draw first telestration marks on the first image shown on the second user display; and
a processor system comprising a tangible medium configured to receive the input from the second user input device, identify selected points of interest of the first image in response to the input, and selectively match the selected points of interest to the second image to determine matched points of interest, wherein the processor system is configured to display second telestration marks at the matched points of interest such that the first telestration marks and the second telestration marks appear in three dimensions to the first user.

### Claim 29 (depends on 28)

The apparatus of claim 28:
wherein processor system further comprises a first user input device;
wherein the processor system is coupled to the first user input device such that the first user can control the telestration marks shown on the display;
wherein the first user input device comprises a camera control coupled to the processor system such that the user can adjust a camera of the source of the pair of stereo images; and
wherein telestration marks drawn by the second user are erased in response to the first user touching the camera control.

### Claim 30

An apparatus to show three-dimensional telestration a user, the apparatus comprising:
a source of a pair of stereo images comprising a left image and a right image;
a first user display comprising a left display and a right display to present the left image and the right image in three-dimensional appearance to the user;
a first input device configured to receive input from the user;
a second user display configured to show at least one of the left image or the right image to a second user; and
a second input device configured to receive input from the second user to draw telestration marks on the at least one of the left image or the right image shown on the second user display;
wherein the first input device and the second input device are each configured to select one of the left image or the right image for the second user to draw the telestration marks at the second user display.

### Claim 31 (depends on 30)

The apparatus of claim 30:
wherein the second display is coupled to the first display for the second user to show to the first user the telestration marks drawn on the selected one of the left image or the right image shown on the second display; and
wherein the first display is configured to show the telestration marks on one of the left display or the right display in response to the selected one of the left image of the right image.

### Claim 32 (depends on 30)

The apparatus of claim 30 further comprising:
a processor system comprising a tangible medium configured to receive the input from the first input device and the input from the second input device;
wherein the processor system is configured to show the telestration marks on only one of the left display or the right display in response to the selected one of the left image or the right image such that the marks are not shown in three dimensions.
