# US 11,896,441 B2 · Systems and methods for measuring a distance using a stereoscopic endoscope

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,896,441 B2
- Document ID: US11896441B2
- Status: Granted
- Kind: B2
- Application number: 17/051,515
- Filed: 2019-05-02
- Published: 2021-07-22
- Granted: 2024-02-13
- Earliest claimed: 2018-05-03
- Assignees: Intuitive Surgical Operations, Inc.
- Applicants: Intuitive Surgical Operations, Inc.
- Inventors: Rohitkumar Godhani; Brian D. Hoffman
- Web page: https://patentgrep.com/patent/US11896441B2

## CPC

- A61B 90/36: {Image-producing devices or illumination devices not otherwise provided for}
- G06T 7/60: Analysis of geometric attributes
- A61B 1/00193: {adapted for stereoscopic vision}
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 34/25: {User interfaces for surgical systems}
- A61B 34/30: Surgical robots
- A61B 34/37: Leader-follower robots (A61B34/35 takes precedence)
- A61B 34/70: {Manipulators specially adapted for use in surgery}
- A61B 2034/2046: {Tracking techniques}
- A61B 2034/301: {for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes}
- A61B 2090/061: {for measuring dimensions, e.g. length}
- A61B 2090/364: {Correlation of different images or relation of image positions in respect to the body}
- A61B 2090/367: {creating a 3D dataset from 2D images using position information}
- G06T 2207/10012: Stereo images
- G06T 2207/10068: Endoscopic image
- G06T 2207/30004: Biomedical image processing
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A measurement system accesses first and second images captured respectively from first and second vantage points by first and second cameras included within a stereoscopic endo scope located at a surgical area associated with a patient. The measurement system receives user input designating a user-selected two-dimensional (“2D”) endpoint corresponding to a feature within the surgical area as represented in the first image, and identifies, based on the user-selected 2D endpoint, a matched 2D endpoint corresponding to the feature as represented in the second image. Based on the user-selected and matched 2D endpoints, the measurement system defines a three-dimensional (“3D”) endpoint corresponding to the feature within the surgical area. The measurement system then determines a distance from the 3D endpoint to an additional 3D endpoint corresponding to an additional feature within the surgical area. Corresponding systems and methods are also described.

## Claims

### Claim 1

A measurement system comprising:
at least one physical computing device that
accesses a first image captured from a first vantage point by a first camera included within a stereoscopic endoscope located at a surgical area associated with a patient;
accesses a second image captured from a second vantage point by a second camera included within the stereoscopic endoscope, the second vantage point stereoscopic to the first vantage point;
receives, from a user of the measurement system, user input designating a user-selected two-dimensional (“2D”) endpoint corresponding to a feature within the surgical area as represented in the first image;
identifies, based on the user-selected 2D endpoint, a matched 2D endpoint corresponding to the feature as represented in the second image;
defines, based on the user-selected and matched 2D endpoints, a three-dimensional (“3D”) endpoint corresponding to the feature within the surgical area; and
determines a distance from the 3D endpoint to an additional 3D endpoint corresponding to an additional feature within the surgical area.

### Claim 2 (depends on 1)

The measurement system of claim 1, wherein the at least one physical computing device further:
receives, from the user prior to determining the distance, user input designating an additional user-selected 2D endpoint corresponding to the additional feature as represented in the first image;
identifies, based on the additional user-selected 2D endpoint, an additional matched 2D endpoint corresponding to the additional feature as represented in the second image; and
defines, based on the additional user-selected and matched 2D endpoints, the additional 3D endpoint corresponding to the additional feature.

### Claim 3 (depends on 1)

The measurement system of claim 1, wherein:
the user-selected 2D endpoint is a pixel block including a first number of rows and a first number of columns; and
the at least one physical computing device identifies the matched 2D endpoint by:
identifying, within the second image, a rectangular area including a plurality of candidate pixel blocks each including the first number of rows and the first number of columns, the rectangular area including a second number of pixels rows greater than the first number of rows and a second number of columns greater than the first number of columns and at least two times greater than the second number of rows;
analyzing the candidate pixel blocks in the plurality of candidate pixel blocks included within the rectangular area to assign, to each of the candidate pixel blocks, a respective similarity metric with respect to the user-selected 2D endpoint; and
based on the analyzing, identifying as the matched 2D endpoint corresponding to the feature a particular candidate pixel block that is assigned a highest similarity metric of the respective similarity metrics assigned to the analyzed candidate pixel blocks in the plurality of candidate pixel blocks.

### Claim 4 (depends on 3)

The measurement system of claim 3, wherein the at least one physical computing device further identifies the matched 2D endpoint by:
tracking kinematics of at least one of the stereoscopic endoscope located at the surgical area and a robotically-manipulated surgical instrument located at the surgical area; and
performing at least one of the identifying of the rectangular area including the plurality of candidate pixel blocks and the analyzing of the candidate pixel blocks to assign the respective similarity metrics to the candidate pixel blocks based on the tracked kinematics.

### Claim 5 (depends on 3)

The measurement system of claim 3, wherein the respective similarity metric assigned to each individual candidate pixel block in the plurality of candidate pixel blocks is based on at least one of
a sum of absolute differences between each pixel in the individual candidate pixel block and a corresponding pixel in the pixel block of the user-selected 2D endpoint;
a sum of squared differences between each pixel in the individual candidate pixel block and the corresponding pixel of the pixel block of the user-selected 2D endpoint;
a normalized cross correlation between each pixel in the individual candidate pixel block and the corresponding pixel of the pixel block of the user-selected 2D endpoint;
a census transformation;
a rank transformation; and
a sum of absolute differences between horizontal and vertical gradients.

### Claim 6 (depends on 1)

The measurement system of claim 1, wherein:
subsequent to the identification of the matched 2D endpoint corresponding to the feature as represented in the second image and prior to the definition of the 3D endpoint corresponding to the feature, the at least one physical computing device further:
identifies, based on the matched 2D endpoint, a reverse-matched 2D endpoint corresponding to the feature as represented in the first image, and
determines, based on the user-selected and reverse-matched 2D endpoints, a match confidence value for the matched 2D endpoint; and
the definition of the 3D endpoint corresponding to the feature is further based on the match confidence value for the matched 2D endpoint.

### Claim 7 (depends on 1)

The measurement system of claim 1, wherein the at least one physical computing device receives the user input designating the user-selected 2D endpoint from the user by
providing the first image for display on a monoscopic display screen communicatively coupled with the at least one physical computing device and configured to be used by an assistant to a surgeon associated with a surgical procedure performed on the patient; and
receiving the user input as the user input is detected by a user interface associated with the display screen.

### Claim 8 (depends on 1)

The measurement system of claim 1, wherein the at least one physical computing device receives the user input designating the user-selected 2D endpoint from the user by
providing the first image for display on each display screen in a stereoscopic pair of display screens communicatively coupled with the at least one physical computing device and configured for use by a surgeon associated with a surgical procedure performed on the patient; and
receiving the user input as the user input is detected by a user interface associated with the stereoscopic pair of display screens.

### Claim 9 (depends on 1)

The measurement system of claim 1, wherein the at least one physical computing device receives the user input designating the user-selected 2D endpoint from the user by
providing the first image for display on a display screen communicatively coupled with the at least one physical computing device;
providing a user interface associated with the display screen, the user interface including a pointer object configured to be moved by the user to any point on the display screen; and
receiving the user input designating the user-selected 2D endpoint as a user selection of a particular point on the display screen to which the user has moved the pointer object.

### Claim 10 (depends on 9)

The measurement system of claim 9, wherein the user selection of the particular point is performed dynamically by the movement of the pointer object to the particular point on the display screen without any additional user selection action.

### Claim 11 (depends on 1)

The measurement system of claim 1, wherein the at least one physical computing device determines the distance from the 3D endpoint to the additional 3D endpoint by:
automatically identifying one or more 3D midpoints on a 3D contour that connects the 3D endpoint to the additional 3D endpoint and that runs along a physical surface upon which the 3D endpoint and the additional 3D endpoint are both disposed;
determining, based on the one or more 3D midpoints, intermediate distances for each segment of a linearly-segmented route from the 3D endpoint to the additional 3D endpoint that passes through each adjacent 3D midpoint so as to substantially adhere to the 3D contour between the 3D endpoint and the additional 3D endpoint; and
determining the distance to be a contoured distance from the 3D endpoint to the additional 3D endpoint computed as a sum of the intermediate distances.

### Claim 12 (depends on 1)

The measurement system of claim 1, wherein:
prior to the identification of the matched 2D endpoint corresponding to the feature as represented in the second image, the at least one physical computing device further:
identifies, based on the user-selected 2D endpoint, a low-confidence matched 2D endpoint,
determines a match confidence value for the low-confidence matched 2D endpoint,
determines that the match confidence value is below a predetermined confidence threshold, and
identifies, based on the user-selected 2D endpoint and in response to the determination that the match confidence value is below the predetermined confidence threshold, a replacement 2D endpoint corresponding to the feature, the replacement 2D endpoint distinct from the user-selected 2D endpoint;
the identification of the matched 2D endpoint corresponding to the feature as represented in the second image is based on the user-selected 2D endpoint by being based on the replacement 2D endpoint that is identified based on the user-selected 2D endpoint; and
the definition of the 3D endpoint corresponding to the feature is based on the user-selected 2D endpoint by being based on the replacement 2D endpoint that is identified based on the user-selected 2D endpoint.

### Claim 13 (depends on 1)

The measurement system of claim 1, wherein both the feature and the additional feature are anatomical features within the patient.

### Claim 14 (depends on 1)

The measurement system of claim 1, wherein the feature is an anatomical feature and the additional feature is a non-anatomical feature.

### Claim 15

A measurement system comprising:
a stereoscopic endoscope configured to be located at a surgical area associated with a patient, the stereoscopic endoscope including
a first camera configured to capture a first image from a first vantage point, and
a second camera configured to capture a second image from a second vantage point that is stereoscopic to the first vantage point;
a display screen configured to display the first image;
a user interface associated with the display screen and allowing a user of the measurement system to designate, from a display of the first image on the display screen, a user-selected two-dimensional (“2D”) endpoint corresponding to a feature within the surgical area as represented in the first image; and
at least one physical computing device communicatively coupled to the stereoscopic endoscope, the display screen, and the user interface, and configured to
identify, based on the user-selected 2D endpoint, a matched 2D endpoint corresponding to the feature as represented in the second image,
define, based on the user-selected and matched 2D endpoints, a three-dimensional (“3D”) endpoint corresponding to the feature within the surgical area, and
determine a distance from the 3D endpoint to an additional 3D endpoint corresponding to an additional feature within the surgical area.

### Claim 16 (depends on 15)

The measurement system of claim 15, wherein:
the user interface further allows the user to designate, from the display of the first image on the display screen, an additional user-selected 2D endpoint corresponding to the additional feature as represented in the first image;
the at least one physical computing device is further configured to
identify, based on the additional user-selected 2D endpoint, an additional matched 2D endpoint corresponding to the additional feature as represented in the second image, and
define, based on the additional user-selected and matched 2D endpoints, the additional 3D endpoint corresponding to the additional feature.

### Claim 17 (depends on 15)

The measurement system of claim 15, wherein:
the user-selected 2D endpoint is a pixel block including a first number of rows and a first number of columns; and
the at least one physical computing device is configured to identify the matched 2D endpoint by:
tracking kinematics of at least one of the stereoscopic endoscope located at the surgical area and a robotically-manipulated surgical instrument located at the surgical area;
identifying, within the second image and based on the tracked kinematics, a rectangular area including a plurality of candidate pixel blocks each including the first number of rows and the first number of columns, the rectangular area including a second number of rows greater than the first number of rows and a second number of columns greater than the first number of columns and at least two times greater than the second number of rows;
analyzing, the candidate pixel blocks in the plurality of candidate pixel blocks included within the rectangular area to assign, to each of the candidate pixel blocks, a respective similarity metric with respect to the user-selected 2D endpoint; and
based on the analyzing, identifying as the matched 2D endpoint corresponding to the feature a particular candidate pixel block that is assigned a highest similarity metric of the respective similarity metrics assigned to the analyzed candidate pixel blocks in the plurality of candidate pixel blocks.

### Claim 18

A method comprising:
accessing, by a measurement system, a first image captured from a first vantage point by a first camera included within a stereoscopic endoscope located at a surgical area associated with a patient;
accessing, by the measurement system, a second image captured from a second vantage point by a second camera included within the stereoscopic endoscope, the second vantage point stereoscopic to the first vantage point;
receiving, by the measurement system from a user of the measurement system, user input designating a user-selected two-dimensional (“2D”) endpoint corresponding to a feature within the surgical area as represented in the first image;
identifying, by the measurement system based on the user-selected 2D endpoint, a matched 2D endpoint corresponding to the feature as represented in the second image;
defining, by the measurement system based on the user-selected and matched 2D endpoints, a three-dimensional (“3D”) endpoint corresponding to the feature within the surgical area; and
determining, by the measurement system, a distance from the 3D endpoint to an additional 3D endpoint corresponding to an additional feature within the surgical area.

### Claim 19 (depends on 18)

The method of claim 18, further comprising:
receiving, by the measurement system from the user prior to determining the distance, user input designating an additional user-selected 2D endpoint corresponding to the additional feature as represented in the first image;
identifying, by the measurement system based on the additional user-selected 2D endpoint, an additional matched 2D endpoint corresponding to the additional feature as represented in the second image; and
defining, by the measurement system based on the additional user-selected and matched 2D endpoints, the additional 3D endpoint corresponding to the additional feature.

### Claim 20 (depends on 18)

The method of claim 18, wherein:
the user-selected 2D endpoint is a pixel block including a first number of rows and a first number of columns; and
the identifying of the matched 2D endpoint includes:
tracking kinematics of at least one of the stereoscopic endoscope located at the surgical area and a robotically-manipulated surgical instrument located at the surgical area;
identifying, within the second image and based on the tracked kinematics, a rectangular area including a plurality of candidate pixel blocks each including the first number of rows and the first number of columns, the rectangular area including a second number of rows greater than the first number of rows and a second number of columns greater than the first number of columns and at least two times greater than the second number of rows;
analyzing, the candidate pixel blocks in the plurality of candidate pixel blocks included within the rectangular area to assign, to each of the candidate pixel blocks, a respective similarity metric with respect to the user-selected 2D endpoint; and
based on the analyzing, identifying as the matched 2D endpoint corresponding to the feature a particular candidate pixel block that is assigned a highest similarity metric of the respective similarity metrics assigned to the analyzed candidate pixel blocks in the plurality of candidate pixel blocks.