# US 11,896,364 B2 · Systems and methods for registration of multiple vision systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,896,364 B2 - Document ID: US11896364B2 - Status: Granted - Kind: B2 - Application number: 17/532,039 - Filed: 2021-11-22 - Published: 2022-03-17 - Granted: 2024-02-13 - Earliest claimed: 2012-08-14 - Assignees: Intuitive Surgical Operations, Inc. - Applicants: Intuitive Surgical Operations, Inc. - Inventors: Tao Zhao; Prashant Chopra - Web page: https://patentgrep.com/patent/US11896364B2 ## CPC - A61B 5/066: {Superposing sensor position on an image of the patient, e.g. obtained by ultrasound or x-ray imaging} - A61B 1/000094: {extracting biological structures} - A61B 1/009: {with bending or curvature detection of the insertion part} - A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis - G06T 7/33: using feature-based methods - A61B 5/062: {using magnetic field} - A61B 2034/2065: {Tracking using image or pattern recognition} - G06T 2207/10068: Endoscopic image - G06T 2207/30021: Catheter; Guide wire - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A non-transitory machine-readable media storing instructions is provided. The instructions cause one or more processors to: define a subregion of a model of an anatomic region of a patient anatomy, the subregion corresponding to an area surrounding a tip of an image capture probe located within the anatomic region, the subregion including a plurality of virtual tissue structures—a set of virtual tissue structures of the plurality of virtual tissue structures is situated distally from the tip of the probe; compare a tissue structure in an image received from a camera positioned at the tip of the probe to a portion of the plurality of virtual tissue structures to identify a closest matched virtual tissue structure; and based on identification of the closest matched virtual tissue structure, register the image to the model to identify a virtual probe position for the tip of the probe with respect to the model. ## Claims ### Claim 1 A non-transitory machine-readable media storing instructions that, when run by one or more processors, cause the one or more processors to: define a subregion of a model of an anatomic region of a patient anatomy, the subregion corresponding to an area surrounding a tip of an image capture probe located within the anatomic region, the subregion including a plurality of virtual tissue structures, wherein a set of virtual tissue structures of the plurality of virtual tissue structures is situated distally from the tip of the image capture probe; compare a tissue structure in an image received from a camera positioned at the tip of the image capture probe to at least a portion of the plurality of virtual tissue structures to identify a closest matched virtual tissue structure; and based on identification of the closest matched virtual tissue structure, perform a registration of the image to the model of the anatomic region to identify a virtual probe position for the tip of the image capture probe with respect to the model. ### Claim 2 (depends on 1) The non-transitory machine-readable media of claim 1, wherein the instructions further cause the one or more processors to guide movement of the image capture probe within the patient anatomy based on the registration. ### Claim 3 (depends on 1) The non-transitory machine-readable media of claim 1, wherein the instructions further cause the one or more processors to identify a position of the tip of the image capture probe based on information received from a sensor system. ### Claim 4 (depends on 3) The non-transitory machine-readable media of claim 3, wherein defining the subregion of the model comprises defining boundaries of the subregion based on a stackup of sensor tolerances for the sensor system. ### Claim 5 (depends on 1) The non-transitory machine-readable media of claim 1, wherein defining the subregion of the model comprises defining boundaries of the subregion based on anatomic movement that includes at least one of patient breathing or tissue deformation in response to medical instrumentation. ### Claim 6 (depends on 1) The non-transitory machine-readable media of claim 1, wherein the instructions further cause the one or more processors to: identify structural aspects of the tissue structure in the image; and identify a subset of the plurality of virtual tissue structures that match the structural aspects. ### Claim 7 (depends on 6) The non-transitory machine-readable media of claim 6, wherein comparing the tissue structure to at least the portion of the plurality of virtual tissue structures comprises comparing the tissue structure to the subset of the plurality of virtual tissue structures to identify the closest matched virtual tissue structure. ### Claim 8 (depends on 6) The non-transitory machine-readable media of claim 6, wherein comparing the tissue structure to at least the portion of the plurality of virtual tissue structures comprises: calculating an optimum similarity measure for each of the plurality of virtual tissue structures in the subset; and selecting a virtual tissue structure in the subset having a greatest optimum similarity measure as the closest matched virtual tissue structure. ### Claim 9 (depends on 1) The non-transitory machine-readable media of claim 1, wherein the instructions further cause the one or more processors to generate the model of the anatomic region, wherein the model is three-dimensional. ### Claim 10 (depends on 1) The non-transitory machine-readable media of claim 1, wherein the instructions further cause the one or more processors to generate a registered virtual image from the model based on the registration. ### Claim 11 (depends on 10) The non-transitory machine-readable media of claim 10, wherein the instructions further cause the one or more processors to display either the registered virtual image or information extracted from the registered virtual image overlaid on the image received from the camera. ### Claim 12 (depends on 11) The non-transitory machine-readable media of claim 11, wherein displaying either the registered virtual image or the information extracted from the registered virtual image comprises displaying the information extracted from the registered virtual image overlaid on the image received from the camera with a transparency that varies depending on a quality of match between the registered virtual image and the image. ### Claim 13 (depends on 10) The non-transitory machine-readable media of claim 10, wherein the instructions further cause the one or more processors to: generate at least one of a micro-tracking image or a macro-tracking image from the registered virtual image, wherein the micro-tracking image and the macro-tracking image each includes a marker indicating the tip of the image capture probe registered with the model; and display the at least one of the micro-tracking image or the macro-tracking image on a display system. ### Claim 14 An apparatus comprising: a processor; and a memory storing computer-readable instructions that, when executed by the processor, cause the apparatus to: define a subregion of a model of an anatomic region of a patient anatomy, the subregion corresponding to an area surrounding a tip of an image capture probe located within the anatomic region, the subregion including a plurality of virtual tissue structures, wherein a set of virtual tissue structures of the plurality of virtual tissue structures is situated distally from the tip of the image capture probe; compare a tissue structure in an image received from a camera positioned at the tip of the image capture probe to at least a portion of the plurality of virtual tissue structures to identify a closest matched virtual tissue structure; and based on identification of the closest matched virtual tissue structure, perform a registration of the image to the model of the anatomic region to identify a virtual probe position for the tip of the image capture probe with respect to the model. ### Claim 15 (depends on 14) The apparatus of claim 14, wherein boundaries of the subregion are defined based on a stackup of sensor tolerances for a sensor system, wherein the sensor system is coupled to the image capture probe. ### Claim 16 (depends on 14) The apparatus of claim 14, wherein boundaries of the subregion are defined based on anatomic movement that includes at least one of patient breathing or tissue deformation in response to medical instrumentation. ### Claim 17 (depends on 14) The apparatus of claim 14, wherein the instructions further cause the apparatus to identify structural aspects of the tissue structure in the image and identify a subset of the plurality of virtual tissue structures that match the structural aspects. ### Claim 18 (depends on 17) The apparatus of claim 17, wherein the instructions further cause the apparatus to compare the tissue structure to the subset of the plurality of virtual tissue structures to identify the closest matched virtual tissue structure. ### Claim 19 (depends on 14) The apparatus of claim 14, wherein the instructions further cause the apparatus to guide movement of the image capture probe within the patient anatomy based on the registration. ### Claim 20 (depends on 14) The apparatus of claim 14, wherein the instructions further cause the apparatus to generate the model of the anatomic region, wherein the model is three-dimensional.