Methods and systems for computer-aided surgery using intra-operative video acquired by a free moving camera

A method for computer-aided execution of surgery and diagnosis in anatomical regions comprising rigid, non-deformable parts or tissues, the method comprising:

• (i) attaching a visual marker, referred to as the World Marker or WM, in an arbitrary location in the surface of the rigid anatomical part of interest, the visual marker comprising at least one planar facet with a known pattern whose system of coordinates defines a global or world system of coordinates;
• (ii) acquiring, by a free-moving camera, intra-operative images and video of an operating field, articular joint, or anatomical cavity;
• (iii) using a touch-probe, which is a tool or instrument comprising a handgrip and a tip, and that has at least one visual marker, referred to as Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates where a position of the tip PT is known, for pin-pointing points and/or outlining curves in the surface of the rigid anatomical part, while keeping both WM and TM of the probe in the Field-of-View or FOV of the camera to be visible in the acquired images and video;
• (iv) receiving, by an image processing system that may correspond to, or may be part of, a computer and/or any other computing device, the images and video acquired in (iii), which are processed for determining the location of the touch-probe and reconstructing in 3D the pin-pointed points and/or outlined curves that are stored in memory in global or world coordinates;
• (v) using the 3D reconstruction results to perform measurements, make inferences, or overlay a surgical plan with the patient's anatomy, in which case a 3D registration method may be used for determining the transformation that maps local coordinates in a surgical plan into world coordinates in the WM attached to anatomy of interest;
• (vi) performing the surgical procedure with the required instruments, where each instrument has at least one visual marker, also referred to as Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates in which the position of relevant points and parts in the instrument are known, while keeping both WM and TMs of the instruments in the FOV of the camera to be visible in the acquired images and video;
• (vii) receiving, by the image processing system, the images and video acquired in (vi), that are processed such that instruments can be located in world coordinates and real-time guidance can be provided by relating these locations with 3D reconstructions results and/or overlaid surgical plan;
• and
• (viii) outputting, by the image processing system, processing results and guidance information to a display, by either overlaying information in the input video using augmented reality (AR) techniques, or by animating models in a virtual reality (VR) environment;
• where the camera is calibrated such that image points u expressed in pixel coordinates are mapped into image points x expressed in metric units according to
• x =f−1(u;
• k, ξ)
• with f =1 being the inverse of a projection function f and where k and ξ stand respectively for camera intrinsic and distortion parameters.

A system for video based computer-aided surgery and diagnosis in anatomical regions of a patient comprising rigid, non-deformable anatomical parts or tissues comprising:

• (i). a free-moving camera that is meant to acquire intra-operative images and video of an operating field, articular joint or anatomical cavity;
• (ii). a visual marker, referred to as a World Marker or WM, which is an object that can be rigidly attached to a surface, and that comprises at least one planar facet with a known pattern having a system of coordinates that defines a global or world system of coordinates;
• (iii). a touch-probe that is a tool or instrument comprising a handgrip and a tip, and that has at least one visual marker, referred to as a Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates where a position of the tip PT is known;
• (iv). one or more surgical instruments for surgical execution, where each surgical instrument has at least one visual marker, also referred to as a Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates in which a position of relevant points and parts in the instrument are known;
• and
• (v). an image processing system, that may correspond to or may be part of a computer and/or any other computing device, that receives as input images and video acquired by the camera (i), as well as commands that may be entered by a button panel, keyboard, camera buttons, foot switch and/or any other input interface, and that outputs processing results and guidance information to a display;
• wherein the system is configured to execute steps comprising:
• (vi). attachment of the WM of (ii) in an arbitrary location in a surface of a rigid anatomical part of interest, such that the WM of (ii) and the rigid anatomical part of interest do not move one with respect to the other;
• (vii). 3D reconstruction of points in the surface of the rigid anatomical part of interest by using the touch-probe of (iii) to pin-point the points in the surface while keeping both the WM and the TM of the probe in a Field-of-View or FOV of the camera of (i) for determining the location of the touch-probe in the world system of coordinates to create 3D reconstruction results;
• (viii). use the 3D reconstruction results to perform measurements, make inferences, or overlay a surgical plan with an anatomical region of the patient, in which case a 3D registration method may be used for determining a transformation that maps local coordinates in a surgical plan into world coordinates in the WM attached to the anatomical part of interest;
• and
• (ix). assisted execution of a clinical procedure by using the surgical instruments of (iv) while keeping both WM and TMs of the surgical instruments in the FOV of the camera of (i) such that the surgical instruments can be located in world coordinates and real-time guidance can be provided by relating surgical instrument locations in world coordinates;
• wherein an arbitrary point P is reconstructed in global or world coordinates by acquiring an image as described in (vii) that undergoes the following processing steps:
• detect, identify, and estimate a 3D pose C of the WM in camera coordinates;
• detect, identify, and estimate a 3D pose {circumflex over (T)} of the TM in camera coordinates;
• reconstruct a point P that is in contact with the tip of the touch-probe by making
• (
• P
• 1
• )
• =
• C
• -
• 1
• ⁢
• T
• ^
• ⁡
• (
• P
• T
• 1
• )
• ;
• and
• store 3D coordinates of point P in a memory.

A system for video based computer-aided surgery and diagnosis in anatomical regions of a patient comprising rigid, non-deformable anatomical parts or tissues comprising:

• (i). a free-moving camera that is meant to acquire intra-operative images and video of an operating field, articular joint or anatomical cavity;
• (ii). a visual marker, referred to as a World Marker or WM, which is an object that can be rigidly attached to a surface, and that comprises at least one planar facet with a known pattern having a system of coordinates that defines a global or world system of coordinates;
• (iii). a touch-probe that is a tool or instrument comprising a handgrip and a tip, and that has at least one visual marker, referred to as a Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates where a position of the tip PT is known;
• (iv). one or more surgical instruments for surgical execution, where each surgical instrument has at least one visual marker, also referred to as a Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates in which a position of relevant points and parts in the instrument are known;
• and
• (v). an image processing system, that may correspond to or may be part of a computer and/or any other computing device, that receives as input images and video acquired by the camera (i), as well as commands that may be entered by a button panel, keyboard, camera buttons, foot switch and/or any other input interface, and that outputs processing results and guidance information to a display;
• wherein the system is configured to execute steps comprising:
• (vi). attachment of the WM of (ii) in an arbitrary location in a surface of a rigid anatomical part of interest, such that the WM of (ii) and the rigid anatomical part of interest do not move one with respect to the other;
• (vii). 3D reconstruction of points in the surface of the rigid anatomical part of interest by using the touch-probe of (iii) to pin-point the points in the surface while keeping both the WM and the TM of the probe in a Field-of-View or FOV of the camera of (i) for determining the location of the touch-probe in the world system of coordinates to create 3D reconstruction results;
• (viii). use the 3D reconstruction results to perform measurements, make inferences, or overlay a surgical plan with an anatomical region of the patient, in which case a 3D registration method may be used for determining a transformation that maps local coordinates in a surgical plan into world coordinates in the WM attached to the anatomical part of interest;
• and
• (ix).assisted execution of a clinical procedure by using the surgical instruments of (iv) while keeping both WM and TMs of the surgical instruments in the FOV of the camera of (i) such that the surgical instruments can be located in world coordinates and real-time guidance can be provided by relating surgical instrument locations in world coordinates;
• wherein the assisted execution of the clinical procedure is accomplished by acquiring continuous video according to (ix), the video comprising a plurality of images, with each consecutive image undergoing the following processing steps in real-time:
• a. detect, identify, and estimate a 3D pose C of the WM in camera coordinates;
• b. detect, identify, and estimate a 3D pose {circumflex over (T)} of the TM of each of the one or more surgical instruments in camera coordinates;
• c. compute the 3D pose T of TM in WM coordinates using the following equation:
• T =C−1 {circumflex over (T)}
• d. map the information of surgical instrument calibration, that can be points, axes, or CAD models, into world coordinates using the rigid transformation T;
• e. relate the surgical instrument calibration information with 3D data stored in memory to make measurements and inferences for the purpose of real-time guidance;
• and
• f. display aiding features either by using Augmented Reality (AR), in which case guidance information is overlaid in image or by animating a Virtual Reality (VR) 3D model.