Method, device, and storage medium for pancreatic mass segmentation, diagnosis, and quantitative patient management

A device for pancreatic mass diagnosis and patient management, comprising:

• a memory storing a computer program;
• and
• a processor configured to execute the computer program stored in the memory to:
• receive computed tomography (CT) images of a pancreas of a patient during a multi-phase CT scan, the CT images including a plurality of three-dimensional (3D) images of the pancreas for each phase of the multiple phases and the pancreas of the patient including a mass;
• perform a segmentation process on the CT images of the pancreas and the mass to obtain a segmentation mask of the pancreas the mass of the patient, wherein the processor is further configured to:
• concatenate the CT images of the pancreas of the patient to form a four-dimensional (4D) input X, wherein X∈RN×W×H×D, wherein R represents a matrix, W is an image width, H is an image height, D is an image depth, the multi-phase CT scan is an N-phase CT scan, and N is a positive integer,
• feed the 4D input X into a segmentation network trained on an input Y which is one-hot encoded by combining a pancreas dataset with masses of radiologist-segmented pancreatic ductal adenocarcinoma (PDAC) and radiologist-segmented non-PDAC, to obtain the segmentation mask of the pancreas, wherein Y∈RK×W×H×D, K is a task label indicating a task including a classification task between the masses of the PDAC and the non-PDAC, and a treatment recommendation task, the segmentation network includes an nnUNet as a backbone network and a pixel-level segmentation loss function Lseg,
• L
• s
• ⁢
• e
• ⁢
• g
• =
• -
• ∑
• w
• ,
• h
• ,
• d
• ⁢
• ∑
• k
• K
• ⁢
• y
• k
• ,
• w
• ,
• h
• ,
• d
• ⁢
• log
• ⁢
• ⁢
• (
• F
• ⁡
• (
• x
• )
• k
• ,
• w
• ,
• h
• ,
• d
• )
• -
• 2
• ⁢
• ∑
• k
• K
• ⁢
• ∑
• w
• ,
• h
• ,
• d
• ⁢
• F
• ⁡
• (
• x
• )
• k
• ,
• w
• ,
• h
• ,
• d
• ⁢
• y
• k
• ,
• w
• ,
• h
• ,
• d
• ∑
• w
• ,
• h
• ,
• d
• ⁢
• F
• ⁡
• (
• x
• )
• k
• ,
• w
• ,
• h
• ,
• d
• +
• ∑
• w
• ,
• h
• ,
• d
• ⁢
• y
• k
• ,
• w
• ,
• h
• ,
• d
• ,
• F(x) is a softmax output of the nnUNet, and F(x)∈RK×W×H×D;
• perform a mask-to-mesh process on the segmentation mask of the pancreas of the patient to obtain a mesh model including 156 vertices of the pancreas of the patient;
• perform a classification process on the mesh model of the pancreas of the patient to identify a type and a grade of a segmented pancreatic mass;
• and
• output updated CT images of the pancreas of the patient, the updated CT images including the segmented pancreatic mass highlighted thereon and the type and the grade of the segmented pancreatic mass annotated thereon.

A non-transitory storage medium storing a computer program, when being executed by a processor, the computer program performing:

• receiving computed tomography (CT) images of a pancreas of a patient during a multi-phase CT scan, the CT images including a plurality of three-dimensional (3D) images of the pancreas for each phase of the multiple phases and the pancreas of the patient including a mass;
• performing a segmentation process on the CT images of the pancreas and the mass to obtain a segmentation mask of the pancreas and the mass of the patient;
• performing a mask-to-mesh process on the segmentation mask of the pancreas and the mass of the patient to obtain a mesh model of the pancreas and the mass of the patient;
• performing a classification process on the mesh model of the pancreas and the mass of the patient to identify a type and a grade of a segmented pancreatic mass, by performing:
• encoding an initialized feature vector hp0 for each vertex p;
• feeding the initialized feature vector hp0 into a graph-based residual convolutional network (Graph-ResNet) to perform the classification process under a pixel level, a vertex level, and a global level;
• encoding a combined feature vector hνp;
• and
• feeding the combined feature vector hνp into a fully connected global classification layer to identify the type and the grade of the segmented pancreatic mass;
• and
• outputting updated CT images of the pancreas of the patient, the updated CT images including the segmented pancreatic mass highlighted thereon and the type and the grade of the segmented pancreatic mass annotated thereon.