# US 12,216,737 B2 · Systems, methods, and apparatuses for actively and continually fine-tuning convolutional neural networks to reduce annotation requirements > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,216,737 B2 - Document ID: US12216737B2 - Status: Granted - Kind: B2 - Application number: 17/698,805 - Filed: 2022-03-18 - Published: 2022-09-22 - Granted: 2025-02-04 - Earliest claimed: 2021-03-19 - Assignees: Arizona Board of Regents on behalf of Arizona State University - Applicants: Arizona Board of Regents on behalf of Arizona State University - Inventors: Zongwei Zhou; Jae Shin; Jianming Liang - Web page: https://patentgrep.com/patent/US12216737B2 ## CPC - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade} - G06T 7/0012: {Biomedical image inspection} - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G16H 30/40: for processing medical images, e.g. editing - G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems - G06T 2207/10068: Endoscopic image - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30032: Colon polyp - G06T 2207/30101: Blood vessel; Artery; Vein; Vascular - G06V 2201/032: of protuberances, polyps nodules, etc. - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA ## Abstract Described herein are systems, methods, and apparatuses for actively and continually fine-tuning convolutional neural networks to reduce annotation requirements, in which the trained networks are then utilized in the context of medical imaging. The success of convolutional neural networks (CNNs) in computer vision is largely attributable to the availability of massive annotated datasets, such as ImageNet and Places. However, it is tedious, laborious, and time consuming to create large annotated datasets, and demands costly, specialty-oriented skills. A novel method to naturally integrate active learning and transfer learning (fine-tuning) into a single framework is presented to dramatically reduce annotation cost, starting with a pre-trained CNN to seek “worthy” samples for annotation and gradually enhances the (fine-tuned) CNN via continual fine-tuning. The described method was evaluated using three distinct medical imaging applications, demonstrating that it can reduce annotation efforts by at least half compared with random selection. ## Claims ### Claim 1 A system comprising: a memory to store instructions; a processor to execute the instructions stored in the memory; wherein the system is specially configured to actively and continually fine tune a convolutional neural network by performing the following operations: generating a plurality of image candidates via a candidate generator, wherein the plurality of image candidates contain true positive samples of pathology and false positive samples of pathology; determining a worthiness of each of the image candidates for annotation based on one or more of an entropy value and a diversity value calculated for each of a selected number of patches associated with each of the plurality of image candidates; iteratively selecting for annotation, via an active, continual fine-tuning (ACFT) algorithm, a set of worthy image candidates from among the image candidates, based on the determined worthiness of each of the image candidates for annotation, and according to a sampling probability to inject randomization into the selecting; annotating each of the image candidates in the selected set of worthy image candidates with a label; and generating, via data augmentation, a plurality of patches for each labeled image candidate in the selected set of worthy image candidates, wherein the label for each labeled image candidate is applied to each of the plurality of patches associated with that labeled image candidate at an image candidate level. ### Claim 2 (depends on 1) The system of claim 1, further comprising majority selection to determine a dominance category for each labeled image candidate and sort an output of each of the plurality of patches associated with each labeled image candidate in the convolutional neural network by the dominance category. ### Claim 3 (depends on 1) The system of claim 1, wherein the method is applied to a colonoscopy frame classification, wherein an input image is labeled as one or more of: (i) informative, and (ii) non-informative. ### Claim 4 (depends on 1) The system of claim 1, wherein the method is applied to polyp detection to reduce misclassification of polyps based on one or more of variations in: (i) color, (ii) shape, and (iii) size. ### Claim 5 (depends on 1) The system of claim 1, wherein the method is applied to pulmonary embolism detection to improve one or more of: (i) pulmonary embolism diagnosis, and (ii) reading time for CTPA datasets. ### Claim 6 A computer-implemented method for actively and continually fine-tuning a convolutional neural network, wherein the method comprises: generating a plurality of image candidates via a candidate generator, wherein the plurality of image candidates contain true positive samples of pathology and false positive samples of pathology; determining a worthiness of each of the image candidates for annotation based on one or more of an entropy value and a diversity value calculated for each of a selected number of patches associated with each of plurality of image candidates; iteratively selecting for annotation, via an active, continual fine-tuning (ACFT) algorithm, a set of worthy image candidates from among the image candidates based on the determined worthiness of each of the image candidates for annotation, and according to a sampling probability to inject randomization into the selecting; annotating each of the image candidates in the selected set of worthy image candidates with a label; and generating, via data augmentation, a plurality of patches for each labeled image candidate in the selected set of worthy image candidates, wherein the label for each labeled image candidate is applied to each of the plurality of patches associated with that image candidate at an image candidate level. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, further comprising majority selection to determine a dominance category for each labeled image candidate and sort an output of each of the plurality of patches associated with each labeled image candidate in the convolutional neural network by the dominance category. ### Claim 8 (depends on 6) The computer-implemented method of claim 6, wherein the method is applied to a colonoscopy frame classification, wherein an input image is labeled as one or more of: (i) informative, and (ii) non-informative. ### Claim 9 (depends on 6) The computer-implemented method of claim 6, wherein the method is applied to polyp detection to reduce misclassification of polyps based on one or more of variations in: (i) color, (ii) shape, and (iii) size. ### Claim 10 (depends on 6) The computer-implemented method of claim 6, wherein the method is applied to pulmonary embolism detection to improve one or more of: (i) pulmonary embolism diagnosis, and (ii) reading time for CTPA datasets. ### Claim 11 A non-transitory computer readable storage media having instructions stored thereupon that, when executed by system having at least a processor and a memory therein, the instructions cause the processor to actively and continually fine-tune a convolutional neural network by performing the following operations: generating a plurality of image candidates via a candidate generator, wherein the plurality of image candidates contain true positive samples of pathology and false positive samples of pathology; determining a worthiness of each of the image candidates for annotation based on one or more of an entropy value and a diversity value calculated for each of a selected number of patches associated with each of the plurality of image candidates; iteratively selecting for annotation, via an active, continual fine-tuning (ACFT) algorithm, a set of worthy image candidates from among the image candidates based on the determined worthiness of each of the image candidates for annotation, and according to a sampling probability to inject randomization; annotating each of the image candidates in the selected set of worthy image candidates with a label; and generating, via data augmentation, a plurality of patches for each labeled image candidate in the selected set or worthy image candidates, wherein the label for each labeled image candidate is passed on to each of the plurality of patches associated with that image candidate at an image candidate level. ### Claim 12 (depends on 11) The non-transitory computer-readable storage media of claim 11, further comprising majority selection to determine a dominance category for each labeled image candidate and sort an output of each of the plurality of patches generated for each labeled image candidate in the convolutional neural network by dominance category. ### Claim 13 (depends on 11) The non-transitory computer-readable storage media of claim 11, wherein the instructions are configurable to execute a colonoscopy frame classification, wherein an input image is labeled as one or more of: (i) informative, and (ii) non-informative. ### Claim 14 (depends on 11) The non-transitory computer-readable storage media of claim 11, wherein the instructions are configurable to execute a polyp detection to reduce misclassification of polyps based on one or more of variations in: (i) color, (ii) shape, and (iii) size. ### Claim 15 (depends on 11) The non-transitory computer-readable storage media of claim 11, wherein the instructions are configurable to execute a pulmonary embolism detection to improve one or more of: (i) pulmonary embolism diagnosis, and (ii) reading time for CTPA datasets.