# US 10,368,956 B2 · MR-levelcheck-2: method for localization of structures in projection images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,368,956 B2 - Document ID: US10368956B2 - Status: Granted - Kind: B2 - Application number: 15/434,123 - Filed: 2017-02-16 - Published: 2017-08-17 - Granted: 2019-08-06 - Earliest claimed: 2016-02-16 - Assignees: The Johns Hopkins University - Applicants: The Johns Hopkins University - Inventors: Jeffrey H. Siewerdsen; Wathudurage Tharindu De Silva; Ali Uneri; Michael Ketcha; Sureerat Reaungamornrat; Jean-Paul Wolinsky - Web page: https://patentgrep.com/patent/US10368956B2 ## CPC - A61B 90/36: {Image-producing devices or illumination devices not otherwise provided for} - A61B 5/055: involving electronic [EMR] or nuclear [NMR] magnetic resonance, e.g. magnetic resonance imaging - A61B 5/4566: {Evaluating the spine (A61B5/4561 takes precedence)} - G01R 33/5608: {Data processing and visualization specially adapted for MR, e.g. for feature analysis and pattern recognition on the basis of measured MR data, segmentation of measured MR data, edge contour detection on the basis of measured MR data, for enhancing measured MR data in terms of signal-to-noise ratio by means of noise filtering or apodization, for enhancing measured MR data in terms of resolution by means for deblurring, windowing, zero filling, or generation of gray-scaled images, colour-coded images or images displaying vectors instead of pixels (image data processing or generation, in general G06T)} - G06T 7/11: Region-based segmentation - G06T 7/33: using feature-based methods - A61B 2090/364: {Correlation of different images or relation of image positions in respect to the body} - A61B 2090/376: {using X-rays, e.g. fluoroscopy} - A61B 2576/02: specially adapted for a particular organ or body part - G06T 2200/04: involving 3D image data - G06T 2207/10088: Magnetic resonance imaging [MRI] - G06T 2207/10116: X-ray image - G06T 2207/30012: Spine; Backbone - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G01R: MEASURING ELECTRIC VARIABLES; MEASURING MAGNETIC VARIABLES (indicating correct tuning of resonant circuits H03J3/12) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An embodiment in accordance with the present invention provides a technique for localizing structures of interest in projection images (e.g., x-ray projection radiographs or fluoroscopy) based on structures defined in a preoperative 3D image (e.g., MR or CT). Applications include, but are not limited to, spinal interventions. The present invention achieves 3D-2D image registration (and particularly allowing use with a preoperative MR image) by segmenting the structures of interest in the preoperative 3D image and generating a simulated projection of the segmented structures to be aligned with the 2D projection image. Other applications include various clinical scenarios involving 3D-2D image registration, such as image-guided cranial neurosurgery, orthopedic surgery, biopsy, and radiation therapy. ## Claims ### Claim 1 A method, comprising: receiving, by a device, 3D magnetic resonance image data in the form of a preoperative 3D magnetic resonance image of a subject; segmenting, by the device, structures of interest in the preoperative 3D magnetic resonance image of the subject, the structures of interest being segmented by using an automatic segmentation algorithm, and the structures of interest being segmented to mitigate non-correspondence of anatomical features between the 3D magnetic resonance image data and 2D image data, the 3D magnetic resonance image data and the 2D image data being acquired from different imaging modalities; generating, by the device, a simulated projection of the segmented structures of interest to be aligned with 2D image data of the subject; providing, by the device and for display, a visual representation of the structures of interest; and providing, by the device and for display, a visual representation of the 3D magnetic resonance image data registered onto the 2D image data of the subject. ### Claim 2 (depends on 1) The method of claim 1, further comprising: registering the 3D magnetic resonance image data onto the 2D image data of the subject for a spine. ### Claim 3 (depends on 1) The method of claim 1, further comprising: segmenting the 3D magnetic resonance image data with an objective function defined by: min u ⁡ ( x ) ∈ { 0 , 1 } ⁢ 〈 1 - u , D 1 〉 + 〈 u , D 2 〉 + ∫ g ⁡ ( x ) ⁢  ∇ u ⁡ ( x )  ⁢ dx . ### Claim 4 (depends on 1) The method of claim 1, further comprising: segmenting the 3D magnetic resonance image data with one chosen from a group consisting of: projecting MRI intensities within each segmented vertebrae; dilating the segmentation to include a region approximating a bony cortex; and projecting a binary region and projecting a vertebral body dilated to include the bony cortex. ### Claim 5 (depends on 1) The method of claim 1, further comprising: adding labels to the visual representation of the 3D magnetic resonance image registered onto the 2D image data. ### Claim 6 (depends on 1) The method of claim 1, further comprising: applying annotations within the preoperative 3D magnetic resonance image of the subject to mark locations of the structures of interest. ### Claim 7 (depends on 6) The method of claim 6, further comprising: initializing segmentation using the annotations applied within the preoperative 3D magnetic resonance image of the subject. ### Claim 8 (depends on 1) The method of claim 1, further comprising: applying forward projection to the segmented structures of interest. ### Claim 9 (depends on 1) The method of claim 1, where mitigating the non-correspondence of the anatomical features comprises: mitigating the non-correspondence of the anatomical features using one selected from a group consisting of: signal intensities, and image gradients. ### Claim 10 (depends on 4) The method of claim 4, wherein the MRI intensities are T2-weighted signal values. ### Claim 11 A system, comprising: one or more memories; and one or more processors, communicatively coupled to the one or more memories, to: receive 3D magnetic resonance image data in form of a preoperative 3D magnetic resonance image of a subject; segment structures of interest in the preoperative 3D magnetic resonance image of the subject, the structures of interest being segmented by using an automatic segmentation algorithm, and the structures of interest being segmented to mitigate non-correspondence of anatomical features between the 3D magnetic resonance image data and 2D image data, the 3D magnetic resonance image data and the 2D image data being acquired from different imaging modalities; generate a simulated projection of the segmented structures to be aligned with 2D image data of the subject; provide, for display, a visual representation of the structures of interest; and provide, for display, a visual representation of the 3D magnetic resonance image data registered onto the 2D image data of the subject. ### Claim 12 (depends on 11) The system of claim 11, where the one or more processors are further to: register 3D magnetic resonance image data to 2D image data for a spine. ### Claim 13 (depends on 11) The system of claim 11, where the one or more processors are to: segment the 3D magnetic resonance image data with an objective function defined by: min u ⁡ ( x ) ∈ { 0 , 1 } ⁢ 〈 1 - u , D 1 〉 + 〈 u , D 2 〉 + ∫ g ⁡ ( x ) ⁢  ∇ u ⁡ ( x )  ⁢ dx . ### Claim 14 (depends on 11) The system of claim 11, where the one or more processors are further to: segment the 3D magnetic resonance image data with one chosen from a group consisting of: projecting MRI intensities within each segmented vertebrae; dilating the segmentation to include a region approximating a bony cortex; projecting a binary region; and projecting a vertebral body dilated to include the bony cortex. ### Claim 15 (depends on 11) The system of claim 11, where the one or more processors are further to: add labels to the visual representation of the 3D magnetic resonance image registered onto the 2D image data of the subject. ### Claim 16 (depends on 11) The system of claim 11, where the one or more processors are further to: apply annotations within the preoperative 3D magnetic resonance image to mark locations of the structures of interest. ### Claim 17 (depends on 16) The system of claim 16, where the one or more processors are further to: initialize segmentation using the annotations applied within the preoperative 3D magnetic resonance image. ### Claim 18 (depends on 11) The system of claim 11, where the one or more processors are further to: apply forward projection to the segmented structures of interest. ### Claim 19 A method, comprising: receiving, by a device, 3D magnetic resonance image data in the form of a preoperative 3D magnetic resonance image of a subject; segmenting, by the device, structures of interest in the preoperative 3D magnetic resonance image of the subject; segmenting, by the device, the 3D magnetic resonance image data with an objective function: min u ⁡ ( x ) ∈ { 0 , 1 } ⁢ 〈 1 - u , D 1 〉 + 〈 u , D 2 〉 + ∫ g ⁡ ( x ) ⁢  ∇ u ⁡ ( x )  ⁢ dx ; generating, by the device, a simulated projection of the segmented structures of interest to be aligned with 2D image data of the subject; providing, by the device and for display, a visual representation of the structures of interest; and providing, by the device and for display, a visual representation of the 3D magnetic resonance image data registered onto the 2D image data of the subject. ### Claim 20 (depends on 19) The method of claim 19, further comprising: adding labels to the visual representation of the 3D magnetic resonance image registered onto the 2D image data. ### Claim 21 (depends on 19) The method of claim 19, where the structures of interest are segmented to mitigate non-correspondence of anatomical features between the 3D magnetic resonance image data and the 2D image data, and the 3D magnetic resonance image data and the 2D image data are acquired from different imaging modalities. ### Claim 22 A system, comprising: one or more memories; one or more processors, communicatively coupled to the one or more memories, to: receive 3D magnetic resonance image data in the form of a preoperative 3D magnetic resonance image of a subject; segment structures of interest in the preoperative 3D magnetic resonance image of the subject; segment the 3D magnetic resonance image data with an objective function: min u ⁡ ( x ) ∈ { 0 , 1 } ⁢ 〈 1 - u , D 1 〉 + 〈 u , D 2 〉 + ∫ g ⁡ ( x ) ⁢  ∇ u ⁡ ( x )  ⁢ dx ; generate a simulated projection of the segmented structures of interest to be aligned with 2D image data of the subject; provide for display a visual representation of the structures of interest; and provide for display a visual representation of the 3D magnetic resonance image data registered onto the 2D image data of the subject.