Neonatal pain identification from neonatal facial expressions

A system for identifying a pain condition in a neonate of interest, the system comprising:

• an image data interface connectable to receive image data comprising a face of the neonate of interest;
• processing circuitry configured as a cascaded Neonatal Convolutional Neural Network (N-CNN) connected to receive and process the image data and to determine whether the neonate of interest is experiencing a pain condition or a no-pain condition, wherein the cascaded N-CNN is trained using a neonatal pain assessment database (NPAD) comprising images of a plurality of neonate faces acquired under the pain condition and images of the plurality of neonate faces acquired under the no-pain condition, and wherein the trained cascaded N-CNN is further configured to:
• preprocess the image data comprising the face of the neonate of interest to generate a plurality of preprocessed frames focused on the face of the neonate of interest;
• perform a combination of a convolution layer and a max pooling operation of the plurality of preprocessed frames at a right branch of the N-CNN to extract prominent features from the plurality of preprocessed frames, at a left branch of the N-CNN to extract generic features from the plurality of preprocessed frames and at a central branch of the N-CNN to extract deep features from the plurality of preprocessed frames;
• merging the prominent features from the plurality of preprocessed frames extracted at the right branch, the generic features extracted at the left branch and the deep features extracted at the central branch to generate merged results;
• and
• perform a combination of a convolution layer and a max pooling operation of the merged results to determine if the neonate of interest is experiencing the pain condition or the no-pain condition.

A non-transitory computer readable storage media having computer-executable instructions, when executed by a processor for:

• training a neonatal convolutional neural network (N-CNN) using a neonatal pain assessment database, the neonatal pain assessment database (NPAD) comprising images of a plurality of neonate faces acquired under a pain condition and images of the plurality of neonate faces acquired under a no-pain condition, to establish a trained N-CNN, monitoring a face of a neonate of interest with a video image capture device to capture image data of the face of the neonate of interest, applying the trained N-CNN to the image data captured by the video image capture device to determine if the neonate of interest is experiencing the pain condition or the no-pain condition, wherein applying the trained N-CNN to the image data captured by the video image capture device comprises preprocessing the image data of the face of the neonate of interest to generate a plurality of preprocessed frames focused on the face of the neonate of interest, performing a combination of a convolution layer and a max pooling operation on each of the plurality of preprocessed frames at a right branch of the N-CNN to extract prominent features from the plurality of preprocessed frames, at a left branch of the N-CNN to extract generic features from the plurality of preprocessed frames and at a central branch of the N-CNN to extract deep features from the plurality of preprocessed frames, merging the prominent features from the plurality of preprocessed frames extracted at the right branch, the generic features extracted at the left branch and the deep features extracted at the central branch to generate merged results and performing a combination of convolution and max pooling of the merged results to determine if the neonate of interest is experiencing the pain condition or the no-pain condition, and providing an output from the N-CNN indicating whether the neonate of interest is experiencing the pain condition or the no-pain condition.