# US 2019/0378274 A1 · Joint registration and segmentation of images using deep learning > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0378274 A1 - Document ID: US20190378274A1 - Status: Published application - Kind: A1 - Application number: 16/001,566 - Filed: 2018-06-06 - Published: 2019-12-12 - Earliest claimed: 2018-06-06 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Rahil Garnavi; Zongyuan Ge; Dwarikanath Mahapatra; Suman Sedai - Web page: https://patentgrep.com/patent/US20190378274A1 ## CPC - G06T 7/0016: {involving temporal comparison} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/08: Learning methods - G06N 3/086: using evolutionary algorithms, e.g. genetic algorithms or genetic programming - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G06T 7/174: involving the use of two or more images - G06T 7/33: using feature-based methods - G06T 7/337: {involving reference images or patches} - G06T 7/38: Registration of image sequences - G06N 3/048: Activation functions - G06T 2207/10008: from scanner, fax or copier - G06T 2207/10081: Computed x-ray tomography [CT] - G06T 2207/10088: Magnetic resonance imaging [MRI] - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A system for registering and segmenting images includes an image scanner configured to acquire an image pair including a first image at a first time and a second image at a second time that is after the first time. A joint registration and segmentation server receives the image pair from the image scanner and simultaneously performs joint registration and segmentation on the image pair using a single deep learning framework. A computer vision processor receives an output of the joint registration and segmentation server and characterizes how a condition has progressed from the first time to the second time therefrom. A user terminal presents the characterization to a user. ## Claims ### Claim 1 A system for registering and segmenting images, comprising: an image scanner configured to acquire an image pair including a first image at a first time and a second image at a second time that is after the first time; a joint registration and segmentation server configured to receive the image pair from the image scanner and to simultaneously perform joint registration and segmentation on the image pair using a single deep learning framework; a computer vision processor configured to receive an output of the joint registration and segmentation server and to characterize how a condition has progressed from the first time to the second time therefrom; and a user terminal configured to present the characterization to a user. ### Claim 2 (depends on 1) The system of claim 1, wherein the image scanner is a computed tomography (CT) scanner, a magnetic resonance imager (MRI), a camera, or a satellite-based imager. ### Claim 3 (depends on 1) The system of claim 1, wherein the first time and the second time are at different days and the first image and the second image are acquired in distinct imaging sessions. ### Claim 4 (depends on 1) The system of claim 1, wherein the joint registration and segmentation server is configured to perform joint registration and segmentation on the image pair using a neural network that has been trained, before the acquisition of the first and second images, using training data. ### Claim 5 (depends on 4) The system of claim 4, wherein the neural network includes generative adversarial networks (GANs) which may use one of the first and second images as a floating image and the other of the first and second images as a reference image. ### Claim 6 (depends on 1) The system of claim 1, wherein the output of the joint registration and segmentation server includes a segmentation mask and a registered image. ### Claim 7 (depends on 1) The system of claim 1, wherein the computer vision processor is configured to perform computer-aided diagnosis. ### Claim 8 (depends on 1) The system of claim 1, wherein the user terminal includes a display device for displaying the characterization to the user. ### Claim 9 A method for registering and segmenting images, comprising: acquiring an image pair including a first image at a first time and a second image at a second time that is after the first time; simultaneously performing joint registration and segmentation on the image pair using a single deep learning framework; and characterizing how a condition has progressed from the first time to the second time from a result of the joint registration and segmentation of the image pair. ### Claim 10 (depends on 9) The method of claim 9, wherein the first and second images are computed tomography (CT) images, magnetic resonance imager (MRI) images, camera images, or satellite images. ### Claim 11 (depends on 9) The method of claim 9, wherein the first time and the second time are at different days and the first image and the second image are acquired in distinct imaging sessions. ### Claim 12 (depends on 9) The method of claim 9, wherein the joint registration and segmentation is performed on the image pair using a neural network that has been trained, before the acquisition of the first and second images, using training data. ### Claim 13 (depends on 12) The method of claim 12, wherein the neural network includes generative adversarial networks (GANs) which may use one of the first and second images as a floating image and the other of the first and second images as a reference image. ### Claim 14 (depends on 9) The method of claim 9, wherein the result of the joint registration and segmentation of the image pair includes a segmentation mask and a registered image. ### Claim 15 (depends on 9) The method of claim 9, wherein the characterizing of how the condition has progressed from the first time to the second time is part of a computer-aided diagnosis. ### Claim 16 A computer program product for registering and segmenting images, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to: simultaneously performing, by the computer, joint registration and segmentation on an image pair using a single deep learning framework, the image pair including a first image at a first time and a second image at a second time that is after the first time; and characterizing, by the computer, how a condition has progressed from the first time to the second time from a result of the joint registration and segmentation of the image pair. ### Claim 17 (depends on 16) The computer program product of claim 16, wherein the first and second images are computed tomography (CT) images, magnetic resonance imager (MRI) images, camera images, or satellite images. ### Claim 18 (depends on 16) The computer program product of claim 16, wherein the joint registration and segmentation is performed on the image pair using a neural network that has been trained, before the acquisition of the first and second images, using training data. ### Claim 19 (depends on 18) The computer program product of claim 18, wherein the neural network includes generative adversarial networks (GANs) which may use one of the first and second images as a floating image and the other of the first and second images as a reference image. ### Claim 20 (depends on 16) The computer program product of claim 16, wherein the result of the joint registration and segmentation of the image pair includes a segmentation mask and a registered image.