# US 12,602,783 B2 · System and methods for automatic image alignment of three-dimensional image > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,783 B2 - Document ID: US12602783B2 - Status: Granted - Kind: B2 - Application number: 18/453,954 - Filed: 2023-08-22 - Published: 2025-02-27 - Granted: 2026-04-14 - Earliest claimed: 2023-08-22 - Assignees: GE Precision Healthcare LLC - Applicants: GE Precision Healthcare LLC - Inventors: Deepa Anand; Bipul Das; Vanika Singhal; Rakesh Mullick; Sandeep Dutta; Amy L Deubig; Maud Bonnard; Christine Smith - Web page: https://patentgrep.com/patent/US12602783B2 ## CPC - G06T 7/0012: {Biomedical image inspection} - G06T 7/73: using feature-based methods - G06T 7/60: Analysis of geometric attributes - G06T 15/00: Three-dimensional [3D] image rendering - G06V 10/24: Aligning, centring, orientation detection or correction of the image - G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components - G01R 33/5608: {Data processing and visualization specially adapted for MR, e.g. for feature analysis and pattern recognition on the basis of measured MR data, segmentation of measured MR data, edge contour detection on the basis of measured MR data, for enhancing measured MR data in terms of signal-to-noise ratio by means of noise filtering or apodization, for enhancing measured MR data in terms of resolution by means for deblurring, windowing, zero filling, or generation of gray-scaled images, colour-coded images or images displaying vectors instead of pixels (image data processing or generation, in general G06T)} - G06T 2207/10081: Computed x-ray tomography [CT] - G06T 2207/10088: Magnetic resonance imaging [MRI] - G06T 2207/10136: 3D ultrasound image - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20132: Image cropping - G06T 2207/30004: Biomedical image processing - G06V 2201/03: Recognition of patterns in medical or anatomical images - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G01R: MEASURING ELECTRIC VARIABLES; MEASURING MAGNETIC VARIABLES (indicating correct tuning of resonant circuits H03J3/12) ## Abstract The current disclosure provides systems and methods for automatic image alignment of three-dimensional (3D) medical image volumes. The method includes pre-processing the 3D medical image volume by selecting a sub-volume of interest, detecting anatomical landmarks in the sub-volume using a deep neural network, estimating transformation parameters based on the anatomical landmarks to adjust rotation angles and translation of the sub-volume, adjusting the rotation angles and translation to produce a first aligned sub-volume, determining confidence in the transformation parameters based on the first aligned sub-volume, and iteratively refining the transformation parameters if the confidence is below a predetermined threshold. The disclosed approach for automated image alignment reduces the need for manual alignment and, increases a probability of the 3D image volume converging to a desired orientation compared to conventional approaches. ## Claims ### Claim 1 A method for automatic image alignment of a three-dimensional (3D) medical image volume, an imaging modality of the 3D medical image volume including at least one of computed tomography (CT), magnetic resonance imaging (MRI), and ultrasound, the method comprising: pre-processing the 3D medical image volume by selecting a sub-volume of interest; detecting a plurality of anatomical landmarks in the sub-volume of interest using a deep neural network trained to identify anatomical landmarks; estimating transformation parameters based on the plurality of anatomical landmarks to adjust rotation angles and translation of the sub-volume of interest; adjusting rotation angles and translation of the sub-volume of interest using the transformation parameters to produce a first aligned sub-volume of interest; determining confidence in the transformation parameters based on the first aligned sub-volume of interest; refining the transformation parameters iteratively if the confidence is below a predetermined threshold; adjusting rotation angles and translation of the first aligned sub-volume of interest using the refined transformation parameters to produce a second aligned sub-volume of interest; determining confidence in the refined transformation parameters based on the second aligned sub-volume of interest by: projecting the second aligned sub-volume of interest onto a plurality of projection planes to produce a plurality of projection images; and mapping the plurality of projection images to a corresponding plurality of confidence scores using a confidence estimation network; and determining confidence in the refined transformation parameters through majority voting using the plurality of confidence scores corresponding to the plurality of projection images. ### Claim 2 (depends on 1) The method of claim 1, wherein the 3D medical image volume is an image sequence comprising a plurality of two-dimensional (2D) images, and wherein pre-processing the 3D medical image volume by selecting the sub-volume of interest comprises: selecting a subset of the plurality of 2D images including the sub-volume of interest; and cropping the subset of the plurality of 2D images to remove regions outside the sub-volume of interest. ### Claim 3 (depends on 1) The method of claim 1, wherein selecting the sub-volume of interest comprises: identifying a pair of bilaterally symmetrical anatomical features in the 3D medical image volume, wherein the pair of bilaterally symmetrical anatomical features comprise a first anatomical feature and a second anatomical feature; and selecting the sub-volume containing the first anatomical feature, and not containing the second anatomical feature. ### Claim 4 (depends on 1) The method of claim 1, wherein the plurality of anatomical landmarks comprises at least three distinct landmarks. ### Claim 5 (depends on 1) The method of claim 1, wherein estimating transformation parameters based on the plurality of anatomical landmarks to adjust rotation angles and translation of the sub-volume of interest comprises: selecting a subset of landmarks from the plurality of anatomical landmarks detected in the sub-volume of interest, wherein the subset of landmarks include at least three landmarks; and determining an anatomical plane of interest with respect to an image coordinate system based on the subset of landmarks. ### Claim 6 (depends on 5) The method of claim 5, wherein adjusting rotation angles and translation of the sub-volume of interest using the transformation parameters to produce the first aligned sub-volume of interest includes, performing multiplanar reformation of the sub-volume of interest along the anatomical plane of interest based on one or more geometric constraints imposed on the subset of landmarks. ### Claim 7 (depends on 1) The method of claim 1, wherein determining confidence in the transformation parameters based on the first aligned sub-volume of interest comprises: detecting the plurality of anatomical landmarks in the first aligned sub-volume of interest using the deep neural network; comparing positions of the plurality of anatomical landmarks in the first aligned sub-volume of interest to one or more pre-determined geometric constraints to determine a degree of geometric constraint satisfaction; and determining the confidence in the transformation parameters based on the degree of geometric constraint satisfaction. ### Claim 8 (depends on 1) The method of claim 1, wherein the transformation parameters include rotation angles and translation vectors. ### Claim 9 (depends on 1) The method of claim 1, wherein refining the transformation parameters iteratively includes: detecting the plurality of anatomical landmarks in the first aligned sub-volume of interest using the deep neural network; estimating refined transformation parameters based on the plurality of anatomical landmarks to adjust rotation angles and translation of the first aligned sub-volume of interest; determining confidence in the refined transformation parameters based on the second aligned sub-volume of interest; and further refining the refined transformation parameters iteratively if the confidence is below the predetermined threshold. ### Claim 10 An image processing system, comprising: a display device; a non-transitory memory including instructions; and a processor, wherein, when executing the instructions, the processor causes the image processing system to: receive a three-dimensional (3D) image volume, an imaging modality of the 3D image volume including at least one of computed tomography (CT), magnetic resonance imaging (MRI), and ultrasound; pre-process the 3D image volume by selecting a sub-volume of interest; detect a plurality of landmarks in the sub-volume of interest using a deep neural network trained to identify anatomical landmarks; estimate transformation parameters based on the plurality of landmarks to adjust rotation angles and translation of the sub-volume of interest; adjust rotation angles and translation of the sub-volume of interest using the transformation parameters to produce a first aligned sub-volume of interest; determine confidence in the transformation parameters based on the first aligned sub-volume of interest; refine the transformation parameters iteratively if the confidence is below a predetermined threshold; adjusting rotation angles and translation of the first aligned sub-volume of interest using the refined transformation parameters to produce a second aligned sub-volume of interest; determining confidence in the refined transformation parameters based on the second aligned sub-volume of interest by: projecting the second aligned sub-volume of interest onto a plurality of projection planes to produce a plurality of projection images; and mapping the plurality of projection images to a corresponding plurality of confidence scores using a confidence estimation network; and determining confidence in the refined transformation parameters through majority voting using the plurality of confidence scores corresponding to the plurality of projection images. ### Claim 11 (depends on 10) The image processing system of claim 10, wherein the plurality of landmarks comprises at least three distinct landmarks. ### Claim 12 (depends on 10) The image processing system of claim 10, wherein the processor further causes the image processing system to: select a subset of landmarks from the plurality of landmarks detected in the sub-volume of interest, wherein the subset of landmarks includes at least three landmarks; and determine an anatomical plane of interest with respect to an image coordinate system based on the subset of landmarks. ### Claim 13 (depends on 12) The image processing system of claim 12, wherein the processor further causes the image processing system to: perform multiplanar reformation of the sub-volume of interest along the anatomical plane of interest. ### Claim 14 (depends on 10) The image processing system according to claim 10, wherein the processor further causes the image processing system to: detect the plurality of landmarks in the first aligned sub-volume of interest using the deep neural network; compare positions of the plurality of landmarks in the first aligned sub-volume of interest to one or more pre-determined geometric constraints to determine a degree of geometric constraint satisfaction; and determine the confidence in the transformation parameters based on the degree of geometric constraint satisfaction. ### Claim 15 (depends on 10) The image processing system of claim 10, wherein the processor further causes the image processing system to: project the first aligned sub-volume of interest onto a projection plane to produce a projection image, wherein the projection plane is parallel to a plurality of 2D images comprising the first aligned sub-volume of interest; and map the projection image to a confidence score using a confidence estimation network. ### Claim 16 A method comprising: receiving a three-dimensional (3D) image volume comprising a first plurality of two-dimensional (2D) images parallel to a first plane, an imaging modality of the 3D image volume including at least one of computed tomography (CT), magnetic resonance imaging (MRI), and ultrasound; pre-processing the 3D image volume by selecting a subset of the first plurality of 2D images to produce a sub-volume of interest including an anatomical region of interest; detecting at least three landmarks in the sub-volume of interest using a deep neural network trained to identify anatomical landmarks; estimating transformation parameters, including an anatomical plane of interest intersecting the sub-volume of interest, based on the at least three landmarks; performing multiplanar reformatting of the sub-volume of interest along the anatomical plane of interest to produce a first aligned sub-volume of interest, wherein the first aligned sub-volume of interest comprises a second plurality of 2D images parallel to the anatomical plane of interest; determining confidence in the transformation parameters based on the first aligned sub-volume of interest; refining the transformation parameters iteratively if the confidence is below a predetermined threshold; adjusting rotation angles and translation of the first aligned sub-volume of interest using the refined transformation parameters to produce a second aligned sub-volume of interest; determining confidence in the refined transformation parameters based on the second aligned sub-volume of interest by: projecting the second aligned sub-volume of interest onto a plurality of projection planes to produce a plurality of projection images; and mapping the plurality of projection images to a corresponding plurality of confidence scores using a confidence estimation network; and determining confidence in the refined transformation parameters through majority voting using the plurality of confidence scores corresponding to the plurality of projection images. ### Claim 17 (depends on 16) The method of claim 16, wherein detecting the at least three landmarks includes: detecting a plurality of landmarks comprising more than three landmarks in the sub-volume of interest; for each landmark of the plurality of landmarks: encoding the landmark as a feature vector; and comparing the feature vector against a distribution of previously encoded feature vectors of a same landmark type as the landmark; determining a probability of the landmark belonging to the distribution of previously encoded feature vectors; and determining a confidence in the landmark based on the probability. ### Claim 18 (depends on 17) The method of claim 17, the method further comprising, filtering the plurality of landmarks based on confidence, to obtain the at least three landmarks. ### Claim 19 (depends on 16) The method of claim 16, wherein determining confidence in the transformation parameters based on the first aligned sub-volume of interest comprises: detecting the at least three landmarks in the first aligned sub-volume of interest using the deep neural network; comparing positions of the at least three landmarks in the first aligned sub-volume of interest to one or more pre-determined geometric constraints, to determine a degree of geometric constraint satisfaction determining a geometric constraint confidence based on the degree of geometric constraint satisfaction; projecting the first aligned sub-volume of interest onto a projection plane to produce a projection image; mapping the projection image to a projection confidence using a confidence estimation network; and determining an overall confidence for the first aligned sub-volume by combining the geometric constraint confidence with the projection confidence.