# US 11,998,401 B2 · Aligning pre-operative scan images to real-time operative images for a mediated-reality view of a surgical site > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,998,401 B2 - Document ID: US11998401B2 - Status: Granted - Kind: B2 - Application number: 17/744,110 - Filed: 2022-05-13 - Published: 2022-08-25 - Granted: 2024-06-04 - Earliest claimed: 2019-01-23 - Assignees: Proprio, Inc. - Applicants: Proprio, Inc. - Inventors: Samuel R. Browd; James Andrew Youngquist; Adam Gabriel Jones - Web page: https://patentgrep.com/patent/US11998401B2 ## CPC - A61B 90/37: {Surgical systems with images on a monitor during operation} - A61B 90/39: {Markers, e.g. radio-opaque or breast lesions markers} - G06T 7/33: using feature-based methods - G06T 7/73: using feature-based methods - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - A61B 2017/00216: {with eye tracking or head position tracking control} - A61B 2034/2065: {Tracking using image or pattern recognition} - A61B 2090/363: {Use of fiducial points} - A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image} - A61B 2090/367: {creating a 3D dataset from 2D images using position information} - A61B 2090/368: {changing the image on a display according to the operator's position} - A61B 2090/371: {with simultaneous use of two cameras} - A61B 2090/372: {Details of monitor hardware} - A61B 2090/373: {using light, e.g. by using optical scanners} - A61B 2090/376: {using X-rays, e.g. fluoroscopy} - A61B 2090/378: {using ultrasound} - A61B 2090/3925: {ultrasonic} - A61B 2090/3966: {Radiopaque markers visible in an X-ray image} - A61B 2090/3983: {Reference marker arrangements for use with image guided surgery} - A61B 2090/3991: {having specific anchoring means to fixate the marker to the tissue, e.g. hooks} - A61B 2090/502: {Headgear, e.g. helmet, spectacles} - G06T 2207/10016: Video; Image sequence - G06T 2207/10081: Computed x-ray tomography [CT] - G06T 2207/30196: Human being; Person - G06T 2207/30204: Marker - G06T 2207/30208: Marker matrix - G06T 2210/41: Medical - G06T 2219/2004: Aligning objects, relative positioning of parts - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A mediated-reality system for surgical applications incorporates pre-operative images and real-time captured images of a surgical site into a visualization presented on a head-mounted display worn by a surgeon during a surgical procedure. The mediated-reality system tracks the surgeon's head position and generates real-time images of the surgical site from a virtual camera perspective corresponding to the surgeon's head position to mimic the natural viewpoint of the surgeon. The mediated-reality system furthermore aligns the pre-operative images with the real-time images from the virtual camera perspective and presents a mediated-reality visualization of the surgical site with the aligned pre-operative three-dimensional images or a selected portion thereof overlaid on the real-time images representing the virtual camera perspective. ## Claims ### Claim 1 A method comprising: identifying, based on images of a patient in a first position and post-positioning images of the patient in a second position, a set of corresponding features visible in the images of the patient in the first position and the post-positioning images, wherein the images of the patient in the first position are captured prior to a surgery and the post-positioning images are captured after the patient is positioned for the surgery; applying a first transformation to the images of the patient in the first position to substantially align locations of the corresponding features in the images of the patient in the first position to respective locations in the post-positioning images to generate initial transformed images; receiving real-time images of the patient captured at least in part during the surgery; identifying three-dimensional coordinates of features in images of the patient in the first position and three-dimensional coordinates of features in the real-time images of the patient; applying a second transformation to the initial transformed images of the patient in the first position based on the three-dimensional coordinates of features in images of the patient in the first position and three-dimensional coordinates of features in the real-time images of the patient to refine an alignment of the locations of features in the initial transformed images to locations of corresponding features in the real-time images to generate refined transformed images; overlaying the refined transformed images over the real-time images of the patient to generate an initial mediated reality view; and providing the initial mediated reality view to a display device for display. ### Claim 2 (depends on 1) The method of claim 1, wherein the set of corresponding features include anatomical features of a body of the patient. ### Claim 3 (depends on 1) The method of claim 1, wherein the set of corresponding features include fiducial markers on a patch applied to a body of the patient. ### Claim 4 (depends on 3) The method of claim 3, wherein the patch comprises a flexible adhesive patch positioned proximate to an anatomical structure of interest. ### Claim 5 (depends on 3) The method of claim 3, wherein the fiducial markers comprise three-dimensional structures including one or more raised surfaces, and wherein each of the fiducial markers has a unique geometry relative to other fiducial markers on the patch. ### Claim 6 (depends on 3) The method of claim 3, wherein the fiducial markers are arranged in a grid on the patch. ### Claim 7 (depends on 3) The method of claim 3, wherein the fiducial markers comprise materials of different densities that are distinguishable in the real-time images. ### Claim 8 (depends on 3) The method of claim 3, wherein the patch comprises perforated boundaries for tearing the patch into sections. ### Claim 9 (depends on 1) The method of claim 1, further comprising: identifying, based on the images of the patient in the first position, coordinates in a three-dimensional image space corresponding to locations of fiducial markers present on a patch applied to the patient; and identifying, based on the real-time images, coordinates in a three-dimensional real-time image space corresponding to locations of the fiducial markers present on the patch applied to the patient. ### Claim 10 A non-transitory computer-readable storage medium comprising stored instructions, the instructions comprising instructions for generating a mediated reality view of a surgical site, the instructions causing one or more processors of a mediated reality system to execute processing comprising operations to: identify, based on images of a patient in a first position and post-positioning images of the patient in a second position, a set of corresponding features visible in the images of the patient in the first position and the post-positioning images, wherein the images of the patient in the first position are captured prior to a surgery and the post-positioning images are captured after the patient is positioned for the surgery; apply a first transformation to the images of the patient in the first position to substantially align locations of the corresponding features in the images of the patient in the first position to respective locations in the post-positioning images to generate initial transformed images; receiving real-time images of the patient captured at least in part during the surgery; identify three-dimensional coordinates of features in images of the patient in the first position and three-dimensional coordinates of features in the real-time images of the patient; apply a second transformation to the initial transformed images of the patient in the first position based on the three-dimensional coordinates of features in images of the patient in the first position and three-dimensional coordinates of features in the real-time images of the patient to refine an alignment of the locations of features in the initial transformed images to locations of corresponding features in the real-time images to generate refined transformed images; overlay the refined transformed images over the real-time images of the patient to generate an initial mediated reality view; and provide the initial mediated reality view to a display device for display. ### Claim 11 (depends on 10) The non-transitory computer-readable storage medium of claim 10, wherein the set of corresponding features include anatomical features of a body of the patient. ### Claim 12 (depends on 10) The non-transitory computer-readable storage medium of claim 10, wherein the set of corresponding features include fiducial markers on a patch applied to a body of the patient. ### Claim 13 (depends on 12) The non-transitory computer-readable storage medium of claim 12, wherein the patch comprises a flexible adhesive patch positioned proximate to an anatomical structure of interest. ### Claim 14 (depends on 12) The non-transitory computer-readable storage medium of claim 12, wherein the fiducial markers comprise three-dimensional structures including one or more raised surfaces, and wherein each of the fiducial markers has a unique geometry relative to other fiducial markers on the patch. ### Claim 15 (depends on 12) The non-transitory computer-readable storage medium of claim 12, wherein the fiducial markers are arranged in a grid on the patch. ### Claim 16 (depends on 12) The non-transitory computer-readable storage medium of claim 12, wherein the fiducial markers comprise materials of different densities that are distinguishable in the real-time images. ### Claim 17 (depends on 12) The non-transitory computer-readable storage medium of claim 12, wherein the patch comprises perforated boundaries for tearing the patch into sections. ### Claim 18 (depends on 10) The non-transitory computer-readable storage medium of claim 10, further comprising stored instructions that when executed cause the mediated reality system to: identify, based on the images of the patient in the first position, coordinates in a three-dimensional image space corresponding to locations of fiducial markers present on a patch applied to the patient; and identify, based on the real-time images, coordinates in a three-dimensional real-time image space corresponding to locations of the fiducial markers present on the patch applied to the patient. ### Claim 19 A mediated reality system comprising: a display device; a processor; and a non-transitory computer-readable storage medium comprising stored instructions, the instructions comprising instructions for generating a mediated reality view of a surgical site, the instructions when executed by the processor of the mediated reality system, cause the mediated reality system to execute processing comprising operations to: identify, based on images of a patient in a first position and post-positioning images of the patient in a second position, a set of corresponding features visible in the images of the patient in the first position and the post-positioning images, wherein the images of the patient in the first position are captured prior to a surgery and the post-positioning images are captured after the patient is positioned for the surgery; apply a first transformation to the images of the patient in the first position to substantially align locations of the corresponding features in the images of the patient in the first position to respective locations in the post-positioning images to generate initial transformed images; receiving real-time images of the patient captured at least in part during the surgery; identify three-dimensional coordinates of features in images of the patient in the first position and three-dimensional coordinates of features in the real-time images of the patient; apply a second transformation to the initial transformed images of the patient in the first position based on the three-dimensional coordinates of features in images of the patient in the first position and three-dimensional coordinates of features in the real-time images of the patient to refine an alignment of the locations of features in the initial transformed images to locations of corresponding features in real-time images to generate refined transformed images; overlay the refined transformed images over the real-time images of the patient to generate an initial mediated reality view; and provide the initial mediated reality view to the display device for display. ### Claim 20 (depends on 19) The mediated reality system of claim 19, wherein the display device comprises a head-mounted display device.