System and method of incremental learning for object detection

A method for incremental learning object detection without catastrophic forgetting of previously-learned object classes, the method comprising:

• receiving training images including objects pertaining to an object class unknown to an initial version of a two-stage neural network object detector that is able to detect objects pertaining to at least one previously-learned object class;
• and
• training the two-stage neural network object detector to detect the objects of the object class initially unknown, using one-by-one plural images among the training images, until a predetermined condition is met by:
• inputting one image of the plural images to the initial version of the two-stage neural network object detector to obtain a first region proposal and a first object prediction for the at least one previously-learned object class,
• inputting the one image to a current version of the two-stage neural network object detector to obtain a second region proposal and a second object prediction for the at least one previously-learned object class and the object class initially unknown,
• comparing the first region proposal with the second region proposal to estimate a region proposal distillation loss quantifying a decrease of ability to locate objects of the at least one previously-learned object class,
• comparing the first object prediction with the second object prediction to estimate an object identification distillation loss quantifying decrease of ability to identify the objects of the at least one previously-learned object class,
• comparing the second region proposal with ground-truth labels of the one image to estimate a region proposal network loss for the object class initially unknown,
• comparing the second object prediction with the ground-truth labels to estimate an object identification loss for the object class initially unknown,
• calculating a total loss combining the region proposal distillation loss, the object identification distillation loss, the region proposal network loss and the object identification loss, and
• updating the current version of the two-stage neural network object detector so as to minimize the total loss,
• wherein the predetermined condition is met when the number of training iterations reaches a predetermined number, or when a total loss decrease rate is below a predetermined threshold.

A computer-readable medium containing a computer-readable code that when read by a computer causes the computer to perform a method for incremental learning for object detection in images without catastrophic forgetting of previously-learned one or more object classes, the computer-readable medium comprises:

• one or more non-transitory computer-readable medium and method stored on the one or more non-transitory computer-readable medium, the method comprising:
• training a two-stage neural network object detector to locate and identify objects pertaining to an additional object class in images by iteratively updating the two-stage neural network object detector until an overall detection accuracy criterion is met,
• wherein
• the updating is performed so as to balance minimizing a loss of an initial ability to locate and identify objects pertaining to the previously-learned one or more object classes and maximizing an ability to additionally locate and identify the objects pertaining to the additional object class, and
• assessing whether the overall detection accuracy criterion is met includes:
• determining a region proposal distillation loss based on comparing a region proposal output by an initial version of the two-stage neural network object detector with a current region proposal output by a current version of the two-stage neural network object detector, wherein the region proposal distillation loss quantifies a decrease of ability to locate objects pertaining to a previously-learned object classes;
• and
• determining a previously learned object identification distillation loss based on comparing a previously learned object prediction obtained by the initial version of the two-stage neural network object detector with an object prediction output by the current version of the two-stage neural network object detector.

A computer-readable medium containing a computer-readable code that when read by a computer causes the computer to perform a method for incremental learning for object detection in images without catastrophic forgetting of previously-learned object classes, the computer-readable comprising:

• one or more non-transitory computer-readable medium and method stored on the one or more non-transitory computer-readable medium, the method comprising:
• training the two-stage neural network object detector to detect the objects of the object class initially unknown, using one-by-one plural images among the training images, until a predetermined condition is met by:
• inputting one image of the plural images to the initial version of the two-stage neural network object detector to obtain a first region proposal and a first object prediction for the at least one previously-learned object class,
• inputting the one image to a current version of the two-stage neural network object detector to obtain a second region proposal and a second object prediction for the at least one previously-learned object class and the object class initially unknown,
• comparing the first region proposal with the second region proposal to estimate a region proposal distillation loss quantifying a decrease of ability to locate objects of the at least one previously-learned object class,
• comparing the first object prediction with the second object prediction to estimate an object identification distillation loss quantifying decrease of ability to identify the objects of the at least one previously-learned object class,
• comparing the second region proposal with ground-truth labels of the one image to estimate a region proposal network loss for the object class initially unknown,
• comparing the second object prediction with the ground-truth labels to estimate an object identification loss for the object class initially unknown,
• calculating a total loss combining the region proposal distillation loss, the object identification distillation loss, the region proposal network loss and the object identification loss, and
• updating the current version of the two-stage neural network object detector so as to minimize the total loss,
• wherein the predetermined condition is met when the number of training iterations reaches a predetermined number, or when a total loss decrease rate is below a predetermined threshold.

A system for incremental learning for object detection without catastrophic forgetting of previously-learned object classes, the system comprising:

• a computing device having a user interface and configured to use a two-stage neural network object detector to locate and identify objects and to communicate with a cloud server;
• and
• the cloud server configured to gather training images, to store an initial version of the two-stage neural network object detector that locates and identifies previously-learned object classes, and to communicate with the computing device and Internet,
• wherein
• at least one of the computing device and the cloud server is configured to train the two-stage neural network object detector to locate and identify objects pertaining to an additional object class by iteratively updating the two-stage neural network object detector that is able to locate and identify objects pertaining to previously-learned object classes until an overall detection accuracy criterion is met,
• the updating is performed so as to balance minimizing a loss of an initial ability of the two-stage neural network detector to locate and identify objects pertaining to the one or more object classes and maximizing an ability of the two-stage neural network detector to additionally locate and identify objects pertaining to the additional object class, and
• assessing whether the overall detection accuracy criterion is met includes:
• determining a region proposal distillation loss based on comparing a region proposal output by an initial version of the two-stage neural network object detector with a current region proposal output by a current version of the two-stage neural network object detector, wherein the region proposal distillation loss quantifies a decrease of ability to locate objects pertaining to a previously-learned object classes;
• and
• determining a previously learned object identification distillation loss based on comparing a previously learned object prediction obtained by the initial version of the two-stage neural network object detector with an object prediction output by the current version of the two-stage neural network object detector.