# US 2010/0166323 A1 · Robust sparse image matching for robotic surgery > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2010/0166323 A1 - Document ID: US20100166323A1 - Status: Published application - Kind: A1 - Application number: 12/465,029 - Filed: 2009-05-13 - Published: 2010-07-01 - Earliest claimed: 2008-12-31 - Assignees: Intuitive Surgical, Inc. - Inventors: Wenyi Zhao; Chenyu Wu; David Hirvonen; Tao Zhao; Brian D. Hoffman; Simon Dimaio - Web page: https://patentgrep.com/patent/US20100166323A1 ## CPC - G06T 7/33: using feature-based methods - G06V 10/757: {Matching configurations of points or features} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Systems, methods, and devices are used to match images. Points of interest from a first image are identified for matching to a second image. In response to the identified points of interest, regions and features can be identified and used to match the points of interest to a corresponding second image or second series of images. Regions can be used to match the points of interest when regions of the first image are matched to the second image with high confidence scores, for example above a threshold. Features of the first image can be matched to the second image, and these matched features may be used to match the points of interest to the second image, for example when the confidence scores for the regions are below the threshold value. Constraint can be used to evaluate the matched points of interest, for example by excluding bad points. ## Claims ### Claim 1 A method of matching images, the method comprising: identifying selected points of interest of a first image for matching to a second image; and selectively matching the selected points of interest to the second image to determine matched points of interest of the second image. ### Claim 2 (depends on 1) The method of claim 1 wherein most of the first image is not matched to the second image. ### Claim 3 (depends on 1) The method of claim 1 wherein each of the selected points of interest is matched to the second image such that each of the matched points of interest of the second image corresponds to one of the selected points of interest of the first image. ### Claim 4 (depends on 1) The method of claim 1 wherein selectively matching comprises matching each of the selected points of interest to the second image with a first image processing method and matching insufficiently matched selected points of interest with a second image processing method in response to output of the first image processing method. ### Claim 5 (depends on 4) The method of claim 4 wherein the first image processing method comprises region matching and the second image processing method comprises feature matching. ### Claim 6 (depends on 5) The method of claim 5 wherein features of the first image are matched to the second image so as to determine corresponding locations of the images for insufficiently matched regions. ### Claim 7 (depends on 1) The method of claim 1 wherein selectively matching comprises: determining a region of the first image for each of the selected points of interest; matching each region of the first image to a second image to determine a confidence score for a matched region of the second image; identifying a feature of the first image for each of the regions with a low confidence score; matching each feature of the first image to the second image to determine a confidence score for a matched feature of the second image; and determining a matched point of interest of the second image for each point of interest in response to: a location of the matched region when the confidence score for the region is high, and a location of the matched feature when the confidence score for the feature is high. ### Claim 8 (depends on 7) The method of claim 7 wherein the matched point of interest of the second image is determined in response to a global offset of the second image when the confidence score for tile feature is low. ### Claim 9 (depends on 7) The method of claim 7: wherein at least some of the selected points of interest are matched to the second image with interpolation when the confidence score for the feature is low; and wherein the interpolation is determined based on the matched points of interest determined with at least one of the location of the matched region or the location of the matched feature. ### Claim 10 (depends on 7) The method of claim 7: wherein at least some of the regions comprise high confidence regions having confidence scores above a region threshold value; and wherein at least some of the locations of the points of interest are determined in response to the high confidence regions, the high confidence regions being sufficiently matched. ### Claim 11 (depends on 10) The method of claim 10: wherein at least some of the regions comprise low confidence regions having confidence scores below the region threshold value such that the low confidence regions are insufficiently matched; and wherein at least some of the locations of the points of interest are determined in response to matched features associated with low confidence regions. ### Claim 12 (depends on 7) The method of claim 7: wherein the regions of the first image are matched to the second image with cross-correlation to determine a correlation surface for each region; and wherein each confidence score is determined in response to the correlation surface. ### Claim 13 (depends on 7) The method of claim 7: wherein at least some of the features comprise high confidence features having feature confidence scores above a feature threshold value such that the high confidence features are sufficiently matched; and wherein at least some of the locations of the points of interest are determined in response to the high confidence features. ### Claim 14 (depends on 13) The method of claim 13: wherein at least some of the features comprise low confidence features having feature confidence scores below the feature threshold value such that the low confidence features are insufficiently matched; and wherein at least some of the features of the first image comprise additional features identified in the first image and matched to the second image in response to the low confidence features. ### Claim 15 (depends on 7) The method of claim 7 wherein the features of the first image are matched to the second image with a scale invariant feature transform. ### Claim 16 (depends on 1) The method of claim 1, wherein a location of the second image corresponding to a first point of interest is determined before a second point of interest is identified. ### Claim 17 (depends on 16) The method of claim 16, further comprising sequentially highlighting the points of interest and the corresponding locations while selecting the points of interest. ### Claim 18 (depends on 7) The method of claim 7 wherein each region comprises a portion of the first image, the portion having an area of no more than about twenty percent of a total area of the first image. ### Claim 19 (depends on 1) The method of claim 1 wherein locations of the matched points of interest are determined in response to at least one of the focus constraint or the soft epipolar constraint. ### Claim 20 (depends on 19) The method of claim 19 wherein at least one of the matched points of interest is excluded in response to the locations of the points of interest exceeding the at least one of the focus constraint or the soft epipolar constraint. ### Claim 21 (depends on 19) The method of claim 19 wherein regions and features of the first image are matched to the second image in response to the at least one of the focus constraint or the soft epipolar constraint. ### Claim 22 (depends on 19) The method of claim 19: wherein the locations of the matched points of interest are determined in response to the focus constraint; wherein each of the identified selected points of interest has a first horizontal location of the first image and each of the matched points of interest has a second horizontal location of the second image; and wherein the focus constraint constrains the second horizontal location in response to a difference of the first horizontal location and the second horizontal location. ### Claim 23 (depends on 22) The method of claim 22 wherein at least one of the matched points of interest is excluded when the difference exceeds the focus constraint. ### Claim 24 (depends on 22) The method of claim 22 wherein the regions and features of the first image are matched to the second image in response to the focus constraint. ### Claim 25 (depends on 22) The method of claim 22 further comprising: determining a global image offset of the first image relative to the second image, the global image offset comprising a horizontal global offset; wherein the focus constraint constrains the second horizontal location in response to the horizontal global offset and the difference of the first horizontal location and the second horizontal location. ### Claim 26 (depends on 19) The method of claim 19: wherein the second location is determined in response to the soft epipolar constraint; wherein each of the points of interest comprises a first vertical location for the first image and a second vertical location for the second image; and wherein the soft epipolar constraint constrains the second vertical location in response to the first vertical location. ### Claim 27 (depends on 26) The method of claim 26: wherein the global image offset comprises a vertical global offset; and wherein the soft epipolar constraint constrains the second vertical location in response to the vertical global offset and the vertical location. ### Claim 28 (depends on 1) The method of claim 1 wherein the first image comprises a left image of a tissue structure and the second image comprises a right image of the tissue structure to display the tissue structure in a three-dimensional appearance. ### Claim 29 (depends on 1) The method of claim 1: wherein the first image comprises a first series of real time images; and wherein the second image comprises a second series of real time images. ### Claim 30 A system for matching images, the system comprising: a source of a first image and a second image; and a processor system coupled to the source and comprising a tangible medium configured to identify selected points of interest of the first image and selectively match the selected points of interest to the second image. ### Claim 31 (depends on 30) The system of claim 30 further comprising: an input device configured to input locations of the first image; wherein the processor system is configured to identify the selected points of interest in response to the input locations.