# US 10,769,791 B2 · Systems and methods for cross-modality image segmentation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,769,791 B2 - Document ID: US10769791B2 - Status: Granted - Kind: B2 - Application number: 16/138,926 - Filed: 2018-09-21 - Published: 2019-04-18 - Granted: 2020-09-08 - Earliest claimed: 2017-10-13 - Assignees: Beijing Keya Medical Technology Co., Ltd. - Applicants: Beijing Keya Medical Technology Co., Ltd. - Inventors: Qi Song; Shanhui Sun; Youbing Yin; Kunlin Cao - Web page: https://patentgrep.com/patent/US10769791B2 ## CPC - G06V 10/82: using neural networks - G01R 33/4812: {MR combined with X-ray or computed tomography [CT]} - G01R 33/5608: {Data processing and visualization specially adapted for MR, e.g. for feature analysis and pattern recognition on the basis of measured MR data, segmentation of measured MR data, edge contour detection on the basis of measured MR data, for enhancing measured MR data in terms of signal-to-noise ratio by means of noise filtering or apodization, for enhancing measured MR data in terms of resolution by means for deblurring, windowing, zero filling, or generation of gray-scaled images, colour-coded images or images displaying vectors instead of pixels (image data processing or generation, in general G06T)} - G06F 18/24133: {Distances to prototypes} - G06F 18/253: {of extracted features} - G06T 7/0012: {Biomedical image inspection} - G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215) - G06T 7/11: Region-based segmentation - G06V 10/764: using classification, e.g. of video objects - G06V 10/806: {of extracted features} - G06T 2207/10081: Computed x-ray tomography [CT] - G06T 2207/10088: Magnetic resonance imaging [MRI] - G06T 2207/10104: Positron emission tomography [PET] - G06T 2207/10108: Single photon emission computed tomography [SPECT] - G06T 2207/10116: X-ray image - G06T 2207/10132: Ultrasound image - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06V 2201/03: Recognition of patterns in medical or anatomical images - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G01R: MEASURING ELECTRIC VARIABLES; MEASURING MAGNETIC VARIABLES (indicating correct tuning of resonant circuits H03J3/12) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Embodiments of the disclosure provide systems and methods for segmenting a medical image. The system includes a communication interface configured to receive the medical image acquired by an image acquisition device. The system also includes a memory configured to store a plurality of learning networks jointly trained using first training images of a first imaging modality and second training images of a second imaging modality. The system further includes a processor, configured to segment the medical image using a segmentation network selected from the plurality of learning networks. ## Claims ### Claim 1 A system for segmenting a medical image, comprising: a communication interface configured to receive the medical image acquired by an image acquisition device; a memory configured to store a plurality of learning networks jointly trained using first training images of a first imaging modality and second training images of a second imaging modality, wherein the plurality of learning networks are jointly trained with a cross-modality learning network using a loss function constructed using outputs by the cross-modality learning network and the plurality of learning networks; and a processor, configured to segment the medical image using a segmentation network selected from the plurality of learning networks. ### Claim 2 (depends on 1) The system of claim 1, wherein the learning network is a convolutional neural network. ### Claim 3 (depends on 1) The system of claim 1, wherein the first imaging modality and the second imaging modality are selected from a group of Computed Tomography (CT), magnetic resonance imaging (MRI), functional MRI, cone beam computed tomography (CBCT), positron emission tomography (PET), single-photon emission computed tomography (SPECT), X-ray, optical tomography, fluorescence imaging, ultrasound imaging, and radiotherapy portal imaging. ### Claim 4 (depends on 1) The system of claim 1, wherein the communication interface is further configured to receive the plurality of learning networks from a training device, wherein the plurality of learning networks include a first-modality learning network and a second-modality learning network, wherein the training device is configured to jointly train the cross-modality learning network, the first-modality learning network, and the second-modality learning network, wherein the cross-modality learning network includes a convolutional base network, and an auxiliary network for learning image features from the first training images and second training images, wherein the first-modality learning network includes the convolutional base network, and a first classification network for segmenting the first training images, and wherein the second-modality learning network includes the convolutional base network, and a second classification network for segmenting the second training images. ### Claim 5 (depends on 4) The system of claim 4, wherein the processor is further configured to: determine that the medical image is of the first imaging modality; and select the first-modality learning network as the segmentation network for segmenting the medical image. ### Claim 6 (depends on 4) The system of claim 4, wherein the training device is further configured to: register the first training images with the second training images; select first image patches from first training images and second image patches from the second training images; pair the first image patches with the second image patches based on the registration; and jointly train the cross-modality learning network, the first-modality learning network, and the second-modality learning network using the paired image patches. ### Claim 7 (depends on 6) The system of claim 6, wherein the training device is further configured to determine a label for each pair of first image patch and second image patch, the label indicating a distance between physical positions of the first image patch and the second image patch of the pair. ### Claim 8 (depends on 6) The system of claim 6, wherein the training device is further configured to apply the image features learned by an auxiliary network to register the first training images with the second training images. ### Claim 9 (depends on 4) The system of claim 4, wherein to jointly train the networks, the training device is configured to use the loss function, wherein the loss function is constructed using outputs by the cross-modality learning network, the first-modality learning network, and the second-modality learning network. ### Claim 10 A method for segmenting a medical image, comprising: receiving the medical image acquired by an image acquisition device; receiving, from a training device, a plurality of learning networks jointly trained using first training images of a first imaging modality and second training images of a second imaging modality, wherein the plurality of learning networks are jointly trained with a cross-modality learning network using a loss function constructed using outputs by the cross-modality learning network and the plurality of learning networks; and segmenting, by a processor, the medical image using a segmentation network selected from the plurality of learning networks. ### Claim 11 (depends on 10) The method of claim 10, wherein the learning network is a convolutional neural network. ### Claim 12 (depends on 10) The method of claim 10, wherein the first imaging modality and the second imaging modality are selected from a group of Computed Tomography (CT), magnetic resonance imaging (MRI), functional MRI, cone beam computed tomography (CBCT), positron emission tomography (PET), single-photon emission computed tomography (SPECT), X-ray, optical tomography, fluorescence imaging, ultrasound imaging, and radiotherapy portal imaging. ### Claim 13 (depends on 10) The method of claim 10, wherein the plurality of learning networks include a first-modality learning network and a second-modality learning network, wherein the method further comprises jointly training the cross-modality learning network, the first-modality learning network, and the second-modality learning network, wherein the cross-modality learning network includes a convolutional base network, and an auxiliary network for learning image features from the first training images and second training images, wherein the first-modality learning network includes the convolutional base network, and a first classification network for segmenting the first training images, and wherein the second-modality learning network includes the convolutional base network, and a second classification network for segmenting the second training images. ### Claim 14 (depends on 13) The method of claim 13, further comprising: determining that the medical image is of the first imaging modality; and selecting the first-modality learning network as the segmentation network for segmenting the medical image. ### Claim 15 (depends on 13) The method of claim 13, wherein jointly training the networks further comprises: registering the first training images with the second training images; selecting first image patches from first training images and second image patches from the second training images; pairing the first image patches with the second image patches based on the registration; and jointly training the cross-modality learning network, the first-modality learning network, and the second-modality learning network using the paired image patches. ### Claim 16 (depends on 15) The method of claim 15, wherein jointly training the networks further comprises determining a label for each pair of first image patch and second image patch, the label indicating a distance between physical positions of the first image patch and the second image patch of the pair. ### Claim 17 (depends on 15) The system of claim 15, wherein registering the first training images with the second training images further comprises applying the image features learned by an auxiliary network to improve the registration. ### Claim 18 (depends on 13) The method of claim 13, wherein jointly training the networks further comprises using the loss function, wherein the loss function is constructed using outputs by the cross-modality learning network, the first-modality learning network, and the second-modality learning network. ### Claim 19 A non-transitory computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by at least one processor, performs a method for segmenting a medical image, the method comprising: receiving the medical image acquired by an image acquisition device; receiving a plurality of learning networks jointly trained using first training images of a first imaging modality and second training images of a second imaging modality, wherein the plurality of learning networks are jointly trained with a cross-modality learning network using a loss function constructed using outputs by the cross-modality learning network and the plurality of learning networks; and segmenting the medical image using a segmentation network selected from the plurality of learning networks. ### Claim 20 (depends on 19) The non-transitory computer-readable medium of claim 19, wherein the plurality of learning networks include a first-modality learning network and a second-modality learning network, wherein the method further comprises jointly training the cross-modality learning network, the first-modality learning network, and the second-modality learning network, wherein the cross-modality learning network includes a convolutional base network, and an auxiliary network for learning image features from the first training images and second training images, wherein the first-modality learning network includes the convolutional base network, and a first classification network for segmenting the first training images, and wherein the second-modality learning network includes the convolutional base network, and a first classification network for segmenting the second training images.