# US 11,992,331 B2 · Neonatal pain identification from neonatal facial expressions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,992,331 B2 - Document ID: US11992331B2 - Status: Granted - Kind: B2 - Application number: 17/073,568 - Filed: 2020-10-19 - Published: 2021-02-04 - Granted: 2024-05-28 - Earliest claimed: 2018-04-19 - Assignees: University of South Florida - Applicants: University of South Florida - Inventors: Ghadh Alzamzmi; Dmitry Goldgof; Rangachar Kasturi; Terri Ashmeade; Yu Sun; Rahul Paul; Md Sirajus Salekin - Web page: https://patentgrep.com/patent/US11992331B2 ## CPC - A61B 5/4824: {Touch or pain perception evaluation (A61B5/4821 takes precedence)} - A61B 5/7267: {involving training the classification device} - A61B 5/1079: {using optical or photographic means} - A61B 5/1176: {Recognition of faces} - G06N 3/02: Neural networks - G16H 30/40: for processing medical images, e.g. editing - G16H 40/63: for local operation - G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems - A61B 2503/045: Newborns, e.g. premature baby monitoring - A61B 2576/00: Medical imaging apparatus involving image processing or analysis (A61B1/00009, A61B6/52 and A61B8/52 take precedence) - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA ## Abstract A Neonatal CNN (N-CNN) is provided for detecting neonatal pain emotion based upon facial recognition. A cascaded N-CNN is trained using a Neonatal Pain Assessment Database (NPAD) to automatically identify a neonatal patient experience pain in real-time. These results show that the automatic recognition of neonatal pain provided by the embodiments of the present invention is a viable and more efficient alternative to the current standard of pain assessment. ## Claims ### Claim 1 A computer-implemented method for identifying when a neonate of interest is experiencing pain, the method comprising: training a neonatal convolutional neural network (N-CNN) using a neonatal pain assessment database (NPAD), the neonatal pain assessment database (NPAD) comprising images of a plurality of neonate faces acquired under a pain condition and images of the plurality of neonate faces acquired under a no-pain condition, to establish a trained N-CNN; monitoring a face of the neonate of interest with a video image capture device to capture image data of the face of the neonate of interest; applying the trained N-CNN to the image data captured by the video image capture device to determine if the neonate of interest is experiencing the pain condition or the no-pain condition, wherein applying the trained N-CNN to the image data further comprises: preprocessing the image data captured by the video image capture device to generate a plurality of preprocessed frames focused on the face of the neonate of interest; performing a combination of a convolution layer and a max pooling operation on each of the plurality of preprocessed frames at a right branch of the N-CNN to extract prominent features from the plurality of preprocessed frames, at a left branch of the N-CNN to extract generic features from the plurality of preprocessed frames and at a central branch of the N-CNN to extract deep features from the plurality of preprocessed frames; merging the prominent features from the plurality of preprocessed frames extracted at the right branch, the generic features extracted at the left branch and the deep features extracted at the central branch to generate merged results; performing a combination of a convolution layer and a max pooling operation of the merged results to determine if the neonate of interest is experiencing the pain condition or the no-pain condition; and providing an output from the N-CNN indicating whether the neonate of interest is experiencing the pain condition or the no-pain condition. ### Claim 2 (depends on 1) The method of claim 1, wherein the images of the neonatal pain assessment database (NPAD) comprise: a first set of images of the plurality of neonate faces acquired prior to application of the pain condition, as a baseline; a second set of images of the plurality of neonate faces acquired during a preparation period prior to the application of the pain condition; a third set of images of the plurality of neonate faces acquired during the application of the pain condition; a fourth set of images of the plurality of neonate faces acquired one minute post completion of the application of the pain condition; a fifth set of images of the plurality of neonate faces acquired two minutes post completion of the application of the pain condition; a sixth set of images of the plurality of neonate faces acquired three minutes post completion of the application of the pain condition; a seventh set of images of the plurality of neonate faces acquired four minutes post completion of the application of the pain condition; and an eighth set of images of the plurality of neonate faces acquired five minutes post completion of the application of the pain condition. ### Claim 3 (depends on 2) The method of claim 2, further comprising using a Neonatal Infant Pain Scale (NIPS) performed by a medical professional to classify each of the acquired first set of images, the acquired second set of images, the acquired third set of images, the acquired fourth set of images, the acquired fifth set of images, the acquired sixth set of images, the acquired seventh set of images and the acquired eight set of images as being acquired under the pain condition or being acquired under the no-pain condition. ### Claim 4 (depends on 3) The method of claim 3, further comprising: applying a face tracker to each of the plurality of images in each of the first, second, third, fourth, fifth, sixth, seventh and eighth sets of images classified as being acquired under the pain condition and classified as being under the no-pain condition to detect the face of the neonate in each of the plurality of images in each of the first, second, third, fourth, fifth, sixth, seventh and eighth sets of images; identifying a plurality of facial landmark points on the face of the neonate detected in each of the plurality of images, each of the facial landmark points having an associated set of coordinates; registering and cropping each of the plurality of images based upon the set of coordinates of the facial landmark points to focus on the face of the neonate detected in each of the plurality of images; and removing any images of the plurality of images that are determined to be the same, thereby establishing the neonatal pain assessment database (NPAD). ### Claim 5 (depends on 4) The method of claim 4, further comprising performing data augmentation to increase a number of images in the neonatal pain assessment database (NPAD) for training the N-CNN. ### Claim 6 (depends on 1) The method of claim 1, wherein preprocessing the image data captured by the video image capture device to generate the plurality of preprocessed frames focused on the face of the neonate of interest further comprises: applying a face tracker to the image data captured by the video image capture device to detect the face of the neonate of interest in each of a plurality of frames; identifying a plurality of facial landmark points on the face of the neonate of interest in each of the plurality of frames, each of the facial landmark points having an associated set of coordinates; and registering and cropping each of the plurality of frames based upon the set of coordinates of the facial landmark points to generate the plurality of preprocessed frames focused on the face of the neonate of interest. ### Claim 7 (depends on 1) The method of claim 1, wherein performing the combination of the convolution layer and the max pooling operation on each of the plurality of preprocessed frames at the right branch of the N-CNN to extract the prominent features from the plurality of preprocessed frames, at the left branch of the N-CNN to extract the generic features from the plurality of preprocessed frames and at the central branch of the N-CNN to extract the deep features from the plurality of preprocessed frames further comprises: performing the max pooling operation on each of the plurality of preprocessed frames using a 10×10 filter in the right branch of the N-CNN to extract the prominent features from the plurality of preprocessed frames; performing the convolution layer with 64 filters of size 5×5 and performing the max pooling operation using a 3×3 filter, performing another convolution layer with 64 filters of size 5×5 and performing another max pooling operation using the 3×3 filter in the central branch of the N-CNN to extract the generic features from the plurality of preprocessed frames; performing the convolution layer with 64 filters of size 5×5 and performing the max pooling operation using a 10×10 filter in the left branch of the N-CNN to extract the deep features from the plurality of preprocessed frames; and performing the convolution layer with 64 filters of size 2×2 and the max pooling operation using a 2×2 filter on the merged results to determine if the neonate of interest is experiencing the pain condition or the no-pain condition. ### Claim 8 A system for identifying a pain condition in a neonate of interest, the system comprising: an image data interface connectable to receive image data comprising a face of the neonate of interest; processing circuitry configured as a cascaded Neonatal Convolutional Neural Network (N-CNN) connected to receive and process the image data and to determine whether the neonate of interest is experiencing a pain condition or a no-pain condition, wherein the cascaded N-CNN is trained using a neonatal pain assessment database (NPAD) comprising images of a plurality of neonate faces acquired under the pain condition and images of the plurality of neonate faces acquired under the no-pain condition, and wherein the trained cascaded N-CNN is further configured to: preprocess the image data comprising the face of the neonate of interest to generate a plurality of preprocessed frames focused on the face of the neonate of interest; perform a combination of a convolution layer and a max pooling operation of the plurality of preprocessed frames at a right branch of the N-CNN to extract prominent features from the plurality of preprocessed frames, at a left branch of the N-CNN to extract generic features from the plurality of preprocessed frames and at a central branch of the N-CNN to extract deep features from the plurality of preprocessed frames; merging the prominent features from the plurality of preprocessed frames extracted at the right branch, the generic features extracted at the left branch and the deep features extracted at the central branch to generate merged results; and perform a combination of a convolution layer and a max pooling operation of the merged results to determine if the neonate of interest is experiencing the pain condition or the no-pain condition. ### Claim 9 (depends on 8) The system of claim 8, further comprising: a video image capture device coupled to the image data interface, the video image capture device for monitoring the face of the neonate of interest to capture the image data comprising the face of the neonate of interest. ### Claim 10 (depends on 9) The system of claim 9, wherein the trained cascaded N-CNN is further configured to preprocess the image data comprising the face of the neonate of interest captured by the video capture device by: applying a face tracker to the image data comprising the face of the neonate of interest captured by the video camera to detect the face of the neonate of interest in each of a plurality of frames, identifying a plurality of facial landmark points on the face of the neonate of interest in each of the plurality of frames, each of the facial landmark points having an associated set of coordinates and registering and cropping each of the plurality of frames based upon the set of coordinates of the facial landmark points to generate the plurality of preprocessed frames focused on the face of the neonate of interest. ### Claim 11 (depends on 10) The system of claim 10, wherein the trained cascaded N-CNN is further configured to identify whether the neonate of interest is experiencing the pain condition or the no-pain condition by: performing the max pooling operation on each of the plurality of preprocessed frames in the right branch of the N-CNN using a 10×10 filter to extract the prominent features, performing the convolution layer with 64 filters of size 5×5 and performing the max pooling operation using a 3×3 filter, performing another convolution layer with 64 filters of size 5×5 and performing another max pooling operation using a 3×3 filter in the central branch of the N-CNN to extract the generic features, performing the convolution layer with 64 filters of size 5×5 and performing the max pooling operation using a 10×10 filter in the left branch of the N-CNN to extract the deep features, merging the prominent, generic, and deep features from the right branch, the central branch and the left branch, respectfully, to generate the merged results and performing the combination of the convolution layer with 64 filters of size 2×2 and the max pooling operation using a 2×2 filter on the merged results. ### Claim 12 (depends on 8) The system of claim 8, further comprising: an output interface configured to provide the determination whether the neonate of interest is experiencing the pain condition or the no-pain condition to a user. ### Claim 13 (depends on 8) The system of claim 8, wherein the processing circuitry is further configured as a N-CNN training module and the neonatal pain assessment database (NPAD) is stored in a memory module, the NPAD comprising: a first set of images of the plurality of neonate faces acquired prior to application of the pain condition, as a baseline; a second set of images of the plurality of neonate faces acquired during a preparation period prior to the application of the pain condition; a third set of images of the plurality of neonate faces acquired during the application of the pain condition; a fourth set of images of the plurality of neonate faces acquired one minute post completion of the application of the pain condition; a fifth set of images of the plurality of neonate faces acquired two minutes post completion of the application of the pain condition; a sixth set of images of the plurality of neonate faces acquired three minutes post completion of the application of the pain condition; a seventh set of images of the plurality of neonate faces acquired four minutes post completion of the application of the pain condition; and an eighth set of images of the plurality of neonate faces acquired five minutes post completion of the application of the pain condition. ### Claim 14 (depends on 13) The system of claim 13, wherein the N-CNN training module is further configured for using a Neonatal Infant Pain Scale (NIPS) performed by a medical professional to classify each of the acquired first set of images, the acquired second set of images, the acquired third set of images, the acquired fourth set of images, the acquired fifth set of images, the acquired sixth set of images, the acquired seventh set of images and the acquired eight set of images of the plurality of neonate faces as being acquired under the pain condition or being acquired under the no-pain condition. ### Claim 15 (depends on 14) The system of claim 14, wherein the N-CNN training module is further configured for: applying a face tracker to each of the plurality of images in each of the first, second, third, fourth, fifth, sixth, seventh and eighth sets of images classified as being acquired under the pain condition and classified as being under the no-pain condition to detect the face of the neonate in each of the plurality of images in each of the first, second, third, fourth, fifth, sixth, seventh and eighth set of images; identifying a plurality of facial landmark points on the face of the neonate detected in each of the plurality of images, each of the facial landmark points having an associated set of coordinates; registering and cropping each of the plurality of images based upon the set of coordinates of the facial landmark points to focus on the face of the neonate; and removing any images of the plurality of images that are determined to be the same, thereby establishing the neonatal pain assessment database (NPAD). ### Claim 16 (depends on 14) The system of claim 14, wherein the cascaded N-CNN training module is further configured for performing data augmentation to increase a number of images in the neonatal pain assessment database (NPAD). ### Claim 17 A non-transitory computer readable storage media having computer-executable instructions, when executed by a processor for: training a neonatal convolutional neural network (N-CNN) using a neonatal pain assessment database, the neonatal pain assessment database (NPAD) comprising images of a plurality of neonate faces acquired under a pain condition and images of the plurality of neonate faces acquired under a no-pain condition, to establish a trained N-CNN, monitoring a face of a neonate of interest with a video image capture device to capture image data of the face of the neonate of interest, applying the trained N-CNN to the image data captured by the video image capture device to determine if the neonate of interest is experiencing the pain condition or the no-pain condition, wherein applying the trained N-CNN to the image data captured by the video image capture device comprises preprocessing the image data of the face of the neonate of interest to generate a plurality of preprocessed frames focused on the face of the neonate of interest, performing a combination of a convolution layer and a max pooling operation on each of the plurality of preprocessed frames at a right branch of the N-CNN to extract prominent features from the plurality of preprocessed frames, at a left branch of the N-CNN to extract generic features from the plurality of preprocessed frames and at a central branch of the N-CNN to extract deep features from the plurality of preprocessed frames, merging the prominent features from the plurality of preprocessed frames extracted at the right branch, the generic features extracted at the left branch and the deep features extracted at the central branch to generate merged results and performing a combination of convolution and max pooling of the merged results to determine if the neonate of interest is experiencing the pain condition or the no-pain condition, and providing an output from the N-CNN indicating whether the neonate of interest is experiencing the pain condition or the no-pain condition.