Multimodal medical image fusion method based on darts network

A multimodal medical image fusion method based on a differentiable architecture search (DARTS) network, the method comprising:

• S1. performing a network architecture search on a preset DARTS model by using multimodal medical image data to obtain a DARTS network architecture applicable to the multimodal medical image data, wherein the DARTS network architecture comprises one or more cells connected in series, each cell comprises a plurality of nodes, each of the nodes is connected to outputs of two preceding cells or other nodes in a current cell by means of different convolution operations, and channels of the nodes are combined into an output of the cell;
• S2. constructing a multimodal medical image fusion network, wherein the multimodal medical image fusion network comprises a multi-channel DARTS network module, a feature fusion module, and an upsampling module, wherein the multi-channel DARTS network module is formed by a plurality of parallel DARTS network architectures,
• the DARTS network architecture is configured to downsample an input image to obtain a corresponding feature map, the feature fusion module is configured to perform feature fusion on a feature map outputted by a dual-channel DARTS network module, and the upsampling module is configured to perform convolution upsampling on a fused feature to obtain a fusion image with the same size as the input image;
• S3. training the multimodal medical image fusion network by using the multimodal medical image data through an unsupervised learning method;
• and
• S4. inputting a multimodal medical image to be fused into a trained multimodal medical image fusion model to obtain a fusion result,
• wherein the different convolution operations in step S1 comprise:
• a depthwise separable convolution with a convolution kernel size of 3, a depthwise separable convolution with a convolution kernel size of 5, a standard convolution with a convolution kernel size of 3, a standard convolution with a convolution kernel size of 5, a dilated convolution with a convolution kernel size of 3, a dilated convolution with a convolution kernel size of 5, and skip connections.

A multimodal medical image fusion method based on a differentiable architecture search (DARTS) network, the method comprising:

• S1. performing a network architecture search on a preset DARTS model by using multimodal medical image data to obtain a DARTS network architecture applicable to the multimodal medical image data, wherein the DARTS network architecture comprises one or more cells connected in series, each cell comprises a plurality of nodes, each of the nodes is connected to outputs of two preceding cells or other nodes in a current cell by means of different convolution operations, and channels of the nodes are combined into an output of the cell;
• S2. constructing a multimodal medical image fusion network, wherein the multimodal medical image fusion network comprises a multi-channel DARTS network module, a feature fusion module, and an upsampling module, wherein the multi-channel DARTS network module is formed by a plurality of parallel DARTS network architectures,
• the DARTS network architecture is configured to downsample an input image to obtain a corresponding feature map, the feature fusion module is configured to perform feature fusion on a feature map outputted by a dual-channel DARTS network module, and the upsampling module is configured to perform convolution upsampling on a fused feature to obtain a fusion image with the same size as the input image;
• S3. training the multimodal medical image fusion network by using the multimodal medical image data through an unsupervised learning method;
• and
• S4. inputting a multimodal medical image to be fused into a trained multimodal medical image fusion model to obtain a fusion result.