US 12,223,707 B1 · Systems and methodologies for automated classification of images of stool in diapers

Abstract

A method involves use of multiple convolutional neural networks and multiple segmentation masks to programmatically generate a stool rating for a digital image of a diaper with stool. The method includes generating, by a first convolutional neural network, a first mask representing an identification of an area of the digital image that corresponds to stool, and a second mask representing an identification of an area of the digital image that corresponds to a diaper. The method further includes generating a third mask representing an intersection of the first and second masks, and generating a modified digital image utilizing the third mask. The method further includes determining, by a second convolutional neural network, a stool rating for the digital image of the diaper with stool by utilizing the modified digital image as input for the second convolutional neural network.

A method involving use of multiple convolutional neural networks and multiple segmentation masks to programmatically generate a rating for one or more test images, the method comprising:

(a) training, using a first plurality of digital images of stool, a first downstream layer of a first convolutional neural network to identify one or more areas of a digital image that correspond to stool, such training comprising, for each respective batch of digital images of the first plurality of digital images, for each of a plurality of iterations,
(i) calculating, by the first convolutional neural network for each respective digital image of the respective batch of digital images, a class probability value for each pixel of the respective digital image, each class probability value being calculated based on one or more parameters associated with one or more layers of the first convolutional neural network, the one or more parameters including one or more weight parameters and one or more bias parameters,
(ii) calculating, by the first convolutional neural network based on a loss function, a respective loss value for the respective batch, such calculating comprising
(A) comparing, for each respective pixel of each respective digital image of the respective batch, the calculated respective class probability value for the respective pixel to a respective encoded truth mask representing an indication of pixels of the respective digital image that were manually identified by a person as corresponding to stool, and
(B) determining the respective loss value for the respective batch based on summing up loss values determined based on the comparisons of the calculated class probability values for the pixels of the respective digital images with the encoded truth masks,
(iii) updating one or more parameters of the first downstream layer of the first convolutional neural network, such updating comprising
(A) calculating a gradient of a matrix of the calculated class probability values,
(B) starting from this calculated gradient of the matrix of the calculated class probability values, backpropagating into the first downstream layer of the first convolutional neural network and calculating gradients for parameters associated with this layer, including weight parameters and bias parameters associated with this layer,
(C) performing, for each respective parameter of a set of one or more parameters of the first downstream layer of the neural network, a parameter update based on a corresponding calculated gradient for that respective parameter and a step size value;
(b) training, using a second plurality of digital images of diapers, a second downstream layer of the first convolutional neural network to identify an area of a digital image that corresponds to a diaper, such training comprising, for each respective batch of digital images of the second plurality of digital images, for each of a plurality of iterations,
(i) calculating, by the first convolutional neural network for each respective digital image of the respective batch of digital images, a class probability value for each pixel of the respective digital image, each class probability value being calculated based on one or more parameters associated with one or more layers of the first convolutional neural network, the one or more parameters including one or more weight parameters and one or more bias parameters,
(ii) calculating, by the first convolutional neural network based on a loss function, a respective loss value for the respective batch, such calculating comprising
(A) comparing, for each respective pixel of each respective digital image of the respective batch, the calculated respective class probability value for the respective pixel to a respective encoded truth mask representing an indication of pixels of the respective digital image that were manually identified by a person as corresponding to a diaper, and
(B) determining the respective loss value for the respective batch based on summing up loss values determined based on the comparisons of the calculated class probability values for the pixels of the respective digital images with the encoded truth masks,
(iii) updating one or more parameters of the second downstream layer of the first convolutional neural network, such updating comprising
(A) calculating a gradient of a matrix of the calculated class probability values,
(B) starting from this calculated gradient of the matrix of the calculated class probability values, backpropagating into the second downstream layer of the first convolutional neural network and calculating gradients for parameters associated with this layer, including weight parameters and bias parameters associated with this layer,
(C) performing, for each respective parameter of a set of one or more parameters of the second downstream layer of the first convolutional neural network, a parameter update based on a corresponding calculated gradient for that respective parameter and a step size value;
(c) training, using a third plurality of digital images of diapers with stool, a second convolutional neural network to classify stool depicted in an image, such training comprising
(i) generating, for each respective digital image of the third plurality of digital images, a respective first mask by providing the respective digital image to the first convolutional neural network as input and receiving as output the generated respective first mask representing an identification of one or more areas of the respective digital image that correspond to stool,
(ii) generating, for each respective digital image of the third plurality of digital images, a respective second mask by providing the respective digital image to the first convolutional neural network as input and receiving as output the generated respective second mask representing an identification of an area of the respective digital image that corresponds to a diaper,
(iii) generating, for each respective digital image of the third plurality of digital images, a respective third mask representing an intersection of the respective first mask for the respective digital image and the respective second mask for the respective digital image,
(iv) generating a fourth plurality of digital images from the third plurality of digital images by, for each respective digital image of the third plurality of digital images, applying the generated respective third mask to the respective digital image,
(v) for each respective batch of digital images of the fourth plurality of digital images, for each of a plurality of iterations,
(A) calculating, by the second convolutional neural network for each respective digital image of the respective batch of digital images, a respective set of class probability values for the respective digital image, each class probability value being calculated based on one or more parameters associated with one or more layers of the second convolutional neural network, the one or more parameters including one or more weight parameters and one or more bias parameters,
(B) calculating, by the second convolutional neural network based on a loss function, a respective loss value for the respective batch, such calculating comprising
(I) calculating, for each respective digital image of the respective batch, a respective loss value based on the calculated respective class probability values and a respective label associated with the respective digital image representing an indication of a classification of stool in the digital image on a rating scale by a human rater, and
(II) determining the respective loss value for the respective batch based on the calculated loss values for the digital images of the respective batch,
(C) updating one or more parameters of the second convolutional neural network based on use of gradient descent, such updating comprising
(I) calculating a gradient of a matrix of the calculated class probability values,
(II) starting from this calculated gradient of the matrix of the calculated class probability values, backpropagating through layers of the second convolutional neural network and calculating gradients for parameters associated with these layers, including weight parameters and bias parameters associated with these layers, this backpropagation involving at least some use of skip connections,
(III) performing, for each respective parameter of a set of one or more parameters of the second convolutional neural network, a parameter update based on a corresponding calculated gradient for that respective parameter and a step size value;
(d) utilizing the trained second convolutional neural network to evaluate a first set of digital images and generate a rating classification for each digital image of the first set of digital images, comprising
(i) generating, for each respective digital image of the first set of digital images, a respective first mask by providing the respective digital image to the first convolutional neural network as input and receiving as output the generated respective first mask representing an identification of one or more areas of the respective digital image that corresponds to stool,
(ii) generating, for each respective digital image of the first set of digital images, a respective second mask by providing the respective digital image to the first convolutional neural network as input and receiving as output the generated respective second mask representing an identification of an area of the respective digital image that corresponds to a diaper,
(iii) generating, for each respective digital image of the first set of digital images, a respective third mask representing an intersection of the respective first mask for the respective digital image and the respective second mask for the respective digital image,
(iv) generating a second set of digital images from the first set of digital images by, for each respective digital image of the first set of digital images, applying the generated respective third mask to the respective digital image,
(v) calculating, by the second convolutional neural network for each respective digital image of the second set of digital images, a respective set of class probability values for the respective digital image, each class probability value being calculated based on one or more parameters associated with one or more layers of the second convolutional neural network, the one or more parameters including one or more weight parameters and one or more bias parameters,
(vi) determining, for each respective digital image of the second set of digital images, based on the calculated respective set of class probability values for that respective digital image, a class to assign the respective digital image to, and associating the corresponding digital image of the first set of digital images with a rating corresponding to that class.

36 more elements

Systems and methodologies for automated classification of images of stool in diapers

Life of this patent

Systems and methodologies for automated classification of images of stool in diapers

Life of this patent

Abstract

Claims6