Automatic Estimation of Ulcerative Colitis Severity from Endoscopy Videos USING ORDINAL MULTI-INSTANCE LEARNING

A non-transitory computer-readable storage medium storing instructions for estimating ulcerative colitis severity depicted in a frame of an endoscopic video, the instructions when executed causing one or more processors to perform steps comprising:

• receiving the frame of the endoscopic video;
• applying a first machine-learned model to the frame of the endoscopic video to estimate a first binary probability that the frame is indicative of ulcerative colitis greater than a first severity level on a baseline severity scale, wherein the first machine-learned model is trained from a set of annotated training endoscopic videos, and wherein each of the set of annotated training endoscopic videos has a respective single label representing a maximum severity of ulcerative colitis observed with respect to the baseline severity scale;
• applying a second machine-learned model to the frame of the endoscopic video to estimate a second binary probability that the frame is indicative of ulcerative colitis greater than a second severity level on the baseline severity scale, the second severity level indicative of more severe ulcerative colitis than the first severity level, wherein the second machine-learned model is trained from the set of annotated training endoscopic videos;
• generating an output severity score for the frame based on at least the first binary probability and the second binary probability;
• and
• outputting the output severity score for the frame of the endoscopic video.

A method for estimating ulcerative colitis severity depicted in an endoscopic video, the method comprising:

• receiving the endoscopic video;
• applying a regression-based machine-learned model to each frame of the endoscopic video to estimate respective frame-level severity scores representing estimated severities of ulcerative colitis in each frame, wherein the machine-learned model is trained from a set of annotated training endoscopic videos, and wherein each of the set of annotated training endoscopic videos has a respective single label representing a maximum severity of ulcerative colitis observed with respect to a baseline severity scale comprising an ordinal set of discrete severity levels;
• determining a maximum frame-level severity score from the respective frame-level severity scores;
• comparing the maximum frame-level severity score to a set of thresholds to select a discrete severity level from the baseline severity scale;
• and
• outputting the discrete severity level.