# US 2021/0334591 A1 · Training set enrichment with insignificantly-abnormal medical images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0334591 A1 - Document ID: US20210334591A1 - Status: Published application - Kind: A1 - Application number: 16/860,086 - Filed: 2020-04-28 - Published: 2021-10-28 - Earliest claimed: 2020-04-28 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Dana Levanony; Shaked Perek; Efrat Hexter - Web page: https://patentgrep.com/patent/US20210334591A1 ## CPC - G06T 7/0016: {involving temporal comparison} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/2431: Multiple classes - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06T 5/77: Retouching; Inpainting; Scratch removal - G06T 7/11: Region-based segmentation - G06V 10/759: {Region-based matching} - G06V 10/764: using classification, e.g. of video objects - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G16H 30/40: for processing medical images, e.g. editing - G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems - G06N 3/045: Combinations of networks - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06V 2201/03: Recognition of patterns in medical or anatomical images - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA ## Abstract A method including: automatically detecting, using at least one machine learning algorithm, one or more abnormalities depicted in a medical image of a patient; automatically determining whether the one or more abnormalities have remained temporally and unchanged, based on an older medical image of the patient; and upon determining that the one or more abnormalities have remained temporally and spatially unchanged: automatically inpainting the one or more abnormalities in the medical image, and automatically enrich a new training set with the inpainted medical image. ## Claims ### Claim 1 A method comprising operating at least one hardware processor to: automatically detect, using at least one machine learning algorithm, one or more abnormalities depicted in a medical image of a patient; automatically determine whether the one or more abnormalities have remained temporally unchanged, based on an older medical image of the patient; and upon determining that the one or more abnormalities have remained temporally unchanged: automatically inpaint the one or more abnormalities in the medical image, to enable enrichment of a new training set with the inpainted medical image. ### Claim 2 (depends on 1) The method of claim 1, wherein the detection of the one or more abnormalities by the at least one machine learning algorithm comprises: applying a machine learning classifier to the medical image of a patient, wherein the machine learning classifier is configured to classify medical images as normal or abnormal, and wherein the application of the machine learning classifier to the medical image results in classification of the medical image as abnormal; and segmenting the one or more abnormalities in the medical image, using an artificial neural network (ANN) configured for segmentation. ### Claim 3 (depends on 2) The method of claim 2, further comprising: prior to applying the machine learning classifier: training the machine learning classifier to detect abnormalities, based on a training set which comprises medical images that are each manually labeled as normal or abnormal; and prior to segmenting the one or more abnormalities: training the ANN to segment abnormalities, based on a training set which comprises medical images in which abnormalities are manually segmented. ### Claim 4 (depends on 2) The method of claim 2, wherein: the determination of whether the one or more abnormalities have remained temporally unchanged comprises comparing the segmented one or more abnormalities with a corresponding area in the older medical image. ### Claim 5 (depends on 1) The method of claim 1, wherein the detection of the one or more abnormalities by the at least one machine learning algorithm comprises: segmenting the one or more abnormalities in the medical image, using an artificial neural network (ANN) configured for segmentation of abnormalities. ### Claim 6 (depends on 5) The method of claim 5, further comprising: prior to segmenting the one or more abnormalities: training the ANN to segment abnormalities, based on a training set which comprises medical images in which abnormalities are manually segmented. ### Claim 7 (depends on 5) The method of claim 5, wherein: the determination of whether the one or more abnormalities have remained temporally unchanged comprises comparing the segmented one or more abnormalities with a corresponding area in the older medical image. ### Claim 8 (depends on 1) The method of claim 1, wherein: the determination of whether the one or more abnormalities have remained temporally unchanged is based on detecting at least one of: a size change, a color change, and a texture change. ### Claim 9 (depends on 1) The method of claim 1, wherein: the at least one machine learning algorithm is configured to detect abnormalities that are selected from the group consisting of: benign lesions, artificial implants, injuries, and interventional tissue modifications. ### Claim 10 (depends on 1) The method of claim 1, further comprising: repeating the steps of claim 1 for multiple medical images of multiple patients, wherein the inpainted medical images are used to enrich a single new training set; manually labeling each of the inpainted medical images as normal; adding, to the single new training set, additional medical images that are labeled as abnormal; and training a new machine learning classifier based on the single new training set. ### Claim 11 (depends on 10) The method of claim 10, wherein the new machine learning classifier is configured, following its training, to detect abnormalities that are selected from the group consisting of: malignant lesions, and premalignant lesions. ### Claim 12 A system comprising: (a) at least one hardware processor; and (b) a non-transitory computer-readable storage medium having program code embodied therewith, the program code executable by said at least one hardware processor to: automatically detect, using at least one machine learning algorithm, one or more abnormalities depicted in a medical image of a patient, automatically determine whether the one or more abnormalities have remained temporally unchanged, based on an older medical image of the patient, and upon determining that the one or more abnormalities have remained temporally unchanged: automatically inpaint the one or more abnormalities in the medical image, to enable enrichment of a new training set with the inpainted medical image. ### Claim 13 (depends on 12) The system of claim 12, wherein the detection of the one or more abnormalities by the at least one machine learning algorithm comprises: applying a machine learning classifier to the medical image of a patient, wherein the machine learning classifier is configured to classify medical images as normal or abnormal, and wherein the application of the machine learning classifier to the medical image results in classification of the medical image as abnormal; and segmenting the one or more abnormalities in the medical image, using an artificial neural network (ANN) configured for segmentation, wherein the determination of whether the one or more abnormalities have remained temporally unchanged comprises comparing the segmented one or more abnormalities with a corresponding area in the older medical image. ### Claim 14 (depends on 12) The system of claim 12, wherein the detection of the one or more abnormalities by the at least one machine learning algorithm comprises: segmenting the one or more abnormalities in the medical image, using an artificial neural network (ANN) configured for segmentation of abnormalities, wherein the determination of whether the one or more abnormalities have remained temporally unchanged comprises comparing the segmented one or more abnormalities with a corresponding area in the older medical image. ### Claim 15 (depends on 12) The system of claim 12, wherein: the determination of whether the one or more abnormalities have remained temporally unchanged is based on detecting at least one of: a size change, a color change, and a texture change. ### Claim 16 (depends on 12) The system of claim 12, wherein the program code if further executable by said at least one hardware processor to: repeat the execution of the program code of claim 12 for multiple medical images of multiple patients, wherein the inpainted medical images are used to enrich a single new training set; manually label each of the inpainted medical images as normal; add, to the single new training set, additional medical images that are labeled as abnormal; and train a new machine learning classifier based on the single new training set. ### Claim 17 A computer program product comprising a non-transitory computer-readable storage medium having program code embodied therewith, the program code executable by at least one hardware processor to: automatically detect, using at least one machine learning algorithm, one or more abnormalities depicted in a medical image of a patient, automatically determine whether the one or more abnormalities have remained temporally unchanged, based on an older medical image of the patient, and upon determining that the one or more abnormalities have remained temporally unchanged: automatically inpaint the one or more abnormalities in the medical image, to enable enrichment of a new training set with the inpainted medical image. ### Claim 18 (depends on 17) The computer program product of claim 17, wherein the detection of the one or more abnormalities by the at least one machine learning algorithm comprises: applying a machine learning classifier to the medical image of a patient, wherein the machine learning classifier is configured to classify medical images as normal or abnormal, and wherein the application of the machine learning classifier to the medical image results in classification of the medical image as abnormal; and segmenting the one or more abnormalities in the medical image, using an artificial neural network (ANN) configured for segmentation, wherein the determination of whether the one or more abnormalities have remained temporally unchanged comprises comparing the segmented one or more abnormalities with a corresponding area in the older medical image. ### Claim 19 (depends on 17) The computer program product of claim 17, wherein the detection of the one or more abnormalities by the at least one machine learning algorithm comprises: segmenting the one or more abnormalities in the medical image, using an artificial neural network (ANN) configured for segmentation of abnormalities, wherein the determination of whether the one or more abnormalities have remained temporally unchanged comprises comparing the segmented one or more abnormalities with a corresponding area in the older medical image. ### Claim 20 (depends on 17) The computer program product of claim 17, wherein: the determination of whether the one or more abnormalities have remained temporally unchanged is based on detecting at least one of: a size change, a color change, and a texture change.