# US 12,004,821 B2 · Systems, methods, and devices for generating a hybrid image > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,004,821 B2 - Document ID: US12004821B2 - Status: Granted - Kind: B2 - Application number: 17/591,904 - Filed: 2022-02-03 - Published: 2023-08-03 - Granted: 2024-06-11 - Earliest claimed: 2022-02-03 - Assignees: Medtronic Navigation, Inc. - Applicants: Medtronic Navigation, Inc. - Inventors: Stephanie Elizabeth Wiegel; Aditya R. Dalvi; Andrew J. Koert - Web page: https://patentgrep.com/patent/US12004821B2 ## CPC - A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis - A61B 6/5247: {combining images from an ionising-radiation diagnostic technique and a non-ionising radiation diagnostic technique, e.g. X-ray and ultrasound} - A61B 90/37: {Surgical systems with images on a monitor during operation} - G06T 11/60: Creating or editing images; Combining images with text - A61B 2034/2051: {Electromagnetic tracking systems} - A61B 2034/2055: {Optical tracking systems} - A61B 2034/2063: {Acoustic tracking systems, e.g. using ultrasound} - A61B 2090/364: {Correlation of different images or relation of image positions in respect to the body} - A61B 2090/3735: {Optical coherence tomography [OCT]} - A61B 2090/374: {NMR or MRI} - A61B 2090/376: {using X-rays, e.g. fluoroscopy} - A61B 2090/3762: {using computed tomography systems [CT]} - A61B 2090/378: {using ultrasound} - A61B 2090/3784: {both receiver and transmitter being in the instrument or receiver being also transmitter} - G06T 2210/41: Medical - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Systems, methods, and devices for generating a hybrid image are provided. A preoperative image may be received. A first intraoperative image may be received from an imaging device. The preoperative image and the first intraoperative image may be input into a hybrid model that combines the first intraoperative image with the preoperative image. A hybrid image comprising the preoperative image combined with the first intraoperative image may be generated with an output of the hybrid model. ## Claims ### Claim 1 A system for generating a hybrid image comprising: an imaging device; a processor; and a memory storing data for processing by the processor, the data, when processed, causing the processor to: receive a preoperative image depicting at least one anatomical element; receive a first intraoperative image from the imaging device depicting the at least one anatomical element; input the preoperative image and the first intraoperative image into a hybrid model, the hybrid model configured to combine the first intraoperative image with the preoperative image; generate, with an output of the hybrid model, a hybrid image comprising the preoperative image combined with the first intraoperative image; receive pose information of the imaging device, the pose information correlating to one or more poses of the imaging device when the first intraoperative image was obtained; generate a three-dimensional representation of the at least one anatomical element based on the preoperative image, the first intraoperative image, and the pose information, the three-dimensional representation including a pose of the at least one anatomical element; receive a second intraoperative image; detect movement in the at least one anatomical element based on a difference in the first intraoperative image and the second intraoperative image; and update the pose of the at least one anatomical element in the three-dimensional representation when movement is detected in the at least one anatomical element. ### Claim 2 (depends on 1) The system of claim 1, wherein the preoperative image comprises an x-ray image and the first intraoperative image comprises an ultrasound image. ### Claim 3 (depends on 1) The system of claim 1, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to: input the preoperative image and the second intraoperative image into the hybrid model; and generate, with an output of the hybrid model, an updated hybrid image comprising the preoperative image and the second intraoperative image, wherein the second intraoperative image replaces the first intraoperative image. ### Claim 4 (depends on 3) The system of claim 3, wherein the preoperative image, the first intraoperative image, and the second intraoperative image each depict at least one anatomical element, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to: detect movement in the at least one anatomical element based on a difference in the first intraoperative image and the second intraoperative image; and replace the first intraoperative image with the second intraoperative image when movement is detected. ### Claim 5 (depends on 1) The system of claim 1, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to: generate a preoperative three-dimensional representation of one or more anatomical elements using the preoperative image; and combine the preoperative three-dimensional representation with the three-dimensional representation. ### Claim 6 (depends on 1) The system of claim 1, further comprising a navigation system configured to use the hybrid image for navigation of at least one surgical instrument, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to: generate instructions for navigating the at least one surgical instrument based on the hybrid image. ### Claim 7 (depends on 1) The system of claim 1, wherein the hybrid image is generated by at least one of merging the first intraoperative image with the preoperative image or the first intraoperative image is overlaid onto the preoperative image. ### Claim 8 (depends on 1) The system of claim 1, wherein the hybrid model is trained using historical preoperative images and historical intraoperative images. ### Claim 9 (depends on 1) The system of claim 1, wherein the preoperative image comprises hard tissue information and the first intraoperative image comprises soft tissue information, wherein a first set of pixels represents the hard tissue information and a second set of pixels represents the soft tissue information. ### Claim 10 A device for generating a hybrid image comprising: a processor; and a memory storing data for processing by the processor, the data, when processed, causing the processor to: receive a preoperative image depicting at least one anatomical element; receive a first intraoperative image depicting the at least one anatomical element; input the preoperative image and the first intraoperative image into a hybrid model, the hybrid model configured to combine the first intraoperative image with the preoperative image; generate, with an output of the hybrid model, a hybrid image comprising the preoperative image and the first intraoperative image; receive pose information of the imaging device, the pose information correlating to one or more poses of the imaging device when the first intraoperative image was obtained; generate a three-dimensional representation of the at least one anatomical element based on the preoperative image, the first intraoperative image, and the pose information, the three-dimensional representation including a pose of the at least one anatomical element; receive a second intraoperative image; detect movement in the at least one anatomical element based on a difference in the first intraoperative image and the second intraoperative image; and update the pose of the at least one anatomical element in the three-dimensional representation when movement is detected in the at least one anatomical element. ### Claim 11 (depends on 10) The device of claim 10, wherein the preoperative image comprises an x-ray image and the first intraoperative image comprises an ultrasound image. ### Claim 12 (depends on 10) The device of claim 10, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to: input the preoperative image and the second intraoperative image into the hybrid model; and generate an updated hybrid image comprising the preoperative image and the second intraoperative image. ### Claim 13 (depends on 12) The device of claim 12, wherein the preoperative image, the first intraoperative image, and the second intraoperative image each depict at least one anatomical element, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to: detect movement in the at least one anatomical element based on a difference in the first intraoperative image and the second intraoperative image. ### Claim 14 (depends on 10) The device of claim 10, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to: generate a preoperative three-dimensional representation of one or more anatomical elements using the preoperative image; and combine the three-dimensional representation with the preoperative three-dimensional representation. ### Claim 15 (depends on 10) The device of claim 10, wherein the hybrid image is generated by at least one of merging the first intraoperative image with the preoperative image or the first intraoperative image is overlaid onto the preoperative image. ### Claim 16 (depends on 10) The device of claim 10, wherein the hybrid model is trained using historical preoperative images and historical intraoperative images. ### Claim 17 (depends on 10) The device of claim 10, wherein the preoperative image comprises hard tissue information and the first intraoperative image comprises soft tissue information. ### Claim 18 A system for navigating using a hybrid image comprising: an imaging device; a navigation system; a processor; and a memory storing data for processing by the processor, the data, when processed, causing the processor to: receive a preoperative image depicting at least one anatomical element; receive an intraoperative image from the imaging device depicting the at least one anatomical element; input the preoperative image and the first intraoperative image into a hybrid model, the hybrid model configured to combine the first intraoperative image with the preoperative image; generate, with the output of the hybrid model, a hybrid image comprising the preoperative image and the first intraoperative image; receive pose information of the imaging device, the pose information correlating to one or more poses of the imaging device when the first intraoperative image was obtained; generate a three-dimensional representation of the at least one anatomical element based on the preoperative image, the first intraoperative image, and the pose information, the three-dimensional representation including a pose of the at least one anatomical element; receive a second intraoperative image; detect movement in the at least one anatomical element based on a difference in the first intraoperative image and the second intraoperative image; and update the pose of the at least one anatomical element in the three-dimensional representation when movement is detected in the at least one anatomical element.