Online active learning with corrective and iterative feedback for streaming data classification

A network node device, comprising:

• processing circuitry operatively coupled to a memory and network interface circuitry;
• wherein the network node device is configured to communicate via the network interface circuitry over a network with a client network node device and at least one human-in-the-loop (HITL) reviewer network node device configured to perform an HITL review process;
• and
• wherein the memory stores instructions executable by the processing circuitry whereby the processing circuitry is configured to:
• initialize trainable parameters of a detection model based on a set of verified samples that includes a representation of the target event;
• receive, via the network interface circuitry, from the client network node device over the network, a first indication including a set of unverified samples;
• and
• iteratively perform, for each of a plurality of iterations for the set of unverified samples until a stopping criterion is satisfied:
• order, based on an informativeness criterion associated with the target event, the set of unverified samples to obtain an ordered set of unverified samples;
• select, based on an adaptive query budget and the ordered set of unverified samples, a query subset of unverified samples for HITL review;
• send, via the network interface circuitry, to the HITL reviewer network node device over the network, a second indication including a first HITL request having the query subset for HITL review;
• receive, via the network interface circuitry, from the HITL reviewer network node device over the network, a third indication including a first HITL response having HITL verification responses for the query subset;
• update the trainable parameters of the detection model based on the HITL verification responses for the query subset;
• apply the updated detection model to remaining unverified samples other than the query subset to obtain prediction results;
• identify, from the prediction results, a positive predicted subset predicted to include the target event;
• send, via the network interface circuitry, to the HITL reviewer network node device over the network, a fourth indication including a second HITL request having the positive predicted subset for HITL review;
• receive, via the network interface circuitry, from the HITL reviewer network node device over the network, a fifth indication including a second HITL response having HITL verification responses indicating which samples of the positive predicted subset include the target event to obtain a true positive subset and which samples of the positive predicted subset do not include the target event to obtain a false positive subset;
• and
• update the trainable parameters of the detection model based on the true positive subset and the false positive subset;
• and
• receive, responsive to satisfaction of the stopping criterion, via the network interface circuitry, from the client network node device over the network, a subsequent indication including a subsequent set of unverified samples.

A system for training a detection model to detect a target event using human-in-the-loop (HITL) verification, the system comprising:

• a client network node device;
• at least one HITL reviewer network node device configured to perform an HITL review process;
• a network node device comprising processing circuitry operatively coupled to memory and network interface circuitry, the network node device being configured to communicate via the network interface circuitry over a network with the client network node device and the at least one HITL reviewer network node device;
• and
• wherein the memory stores instructions executable by the processing circuitry to cause the processing circuitry to:
• initialize trainable parameters of a detection model based on a set of verified samples that includes a representation of the target event;
• receive, from the client network node device over the network, a first indication including a set of unverified samples;
• and
• iteratively, for each of a plurality of iterations for the set of unverified samples until a stopping criterion is satisfied:
• order, based on an informativeness criterion associated with the target event, the set of unverified samples to obtain an ordered set of unverified samples;
• select, based on an adaptive query budget and the ordered set of unverified samples, a query subset of unverified samples for HITL review;
• send, to the at least one HITL reviewer network node device over the network, a second indication including a first HITL request having the query subset for HITL review;
• receive, from the at least one HITL reviewer network node device over the network, a third indication including a first HITL response having HITL verification responses for the query subset;
• update the trainable parameters of the detection model based on the HITL verification responses for the query subset;
• apply the updated detection model to remaining unverified samples other than the query subset to obtain prediction results;
• identify, from the prediction results, a positive predicted subset indicating a predicted presence of the target event;
• send, to the at least one HITL reviewer network node device over the network, a fourth indication including a second HITL request having the positive predicted subset for HITL review;
• receive, from the at least one HITL reviewer network node device over the network, a fifth indication including a second HITL response indicating which samples of the positive predicted subset include the target event to obtain a true positive subset and which samples of the positive predicted subset do not include the target event to obtain a false positive subset;
• and
• update the trainable parameters of the detection model based on the true positive subset and the false positive subset;
• and
• receive, from the client network node device over the network, responsive to satisfaction of the stopping criterion, a subsequent indication including a subsequent set of unverified samples.