# US 2019/0378044 A1 · Processing dynamic data within an adaptive oracle-trained learning system using curated training data for incremental re-training of a predictive model > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0378044 A1 - Document ID: US20190378044A1 - Status: Published application - Kind: A1 - Application number: 16/547,340 - Filed: 2019-08-21 - Published: 2019-12-12 - Earliest claimed: 2013-12-23 - Assignees: Groupon, Inc. - Applicants: Groupon, Inc. - Inventors: Shawn Ryan Jeffery; David Alan Johnston; Jonathan Esterhazy; Gaston L'Huillier; Hernan Enrique Arroyo Garcia - Web page: https://patentgrep.com/patent/US20190378044A1 ## CPC - G06N 20/00: Machine learning - G06N 5/04: Inference or reasoning models - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract In general, embodiments of the present invention provide systems, methods and computer readable media for an adaptive oracle-trained learning framework for automatically building and maintaining models that are developed using machine learning algorithms. In embodiments, the framework leverages at least one oracle (e.g., a crowd) for automatic generation of high-quality training data to use in deriving a model. Once a model is trained, the framework monitors the performance of the model and, in embodiments, leverages active learning and the oracle to generate feedback about the changing data for modifying training data sets while maintaining data quality to enable incremental adaptation of the model. ## Claims ### Claim 1 A system comprising one or more computers configured to implement an adaptive learning framework for automatically building and maintaining a predictive model for processing dynamic data, wherein the adaptive learning framework is configured to include: the predictive model, wherein the model is configured to generate model output from processing an input data instance received by the adaptive learning framework, and wherein the model output includes a judgment and a confidence value representing certainty of the judgment; a training data set from which the predictive model is derived using machine learning; and a training data manager, wherein the training data manager is configured for curating the training data set; and a labeled data reservoir configured to store labeled data instances that have been processed by the predictive model, wherein the labeled data reservoir includes a pool of possible training data, wherein the set of labeled data instances are not included in the training data; and wherein each labeled data instance is associated with a true label representing the instance; and wherein the training data manager is configured to perform operations comprising: determining whether to update the training data set; in an instance in which the training data set is to be updated, selecting a set of labeled data instances from a labeled data reservoir; and updating the training data using the set of labeled data instances. ### Claim 2 (depends on 1) The system of claim 1, wherein determining whether to update the training data set is based at least in part on analyzing the distribution and quality of the training data. ### Claim 3 (depends on 1) The system of claim 1, wherein determining whether to update the training data set is based at least in part on an accuracy assessment of the model performance. ### Claim 4 (depends on 3) The system of claim 3, wherein the accuracy assessment is based on determining whether the confidence value satisfies a confidence threshold value. ### Claim 5 (depends on 2) The system of claim 2, wherein the current model is a classifier predicting to which of a set of predictive categories an input data instance belongs, wherein a true label associated with a labeled data instance identifies the predictive category to which the labeled data instance belongs, and wherein selecting the set of labeled data instances from the labeled data reservoir is based at least in part on maintaining a class balance within the training data. ### Claim 6 (depends on 1) The system of claim 1, wherein the labeled data reservoir includes labeled data instances that are received from multiple sources, and wherein selecting a labeled data instance from the set of labeled data instances comprises: comparing a source of the labeled data instance with a pre-determined source; and selecting the labeled data instance in an instance in which the source of the labeled data instance matches the pre-determined source. ### Claim 7 (depends on 1) The system of claim 1, wherein the operations further comprise: in response to updating the training data, determining whether to re-train the model; in an instance in which the model is re-trained, generating at least one candidate training data set using the updated training data; deriving a candidate model using the candidate training data set; generating an assessment of whether the candidate model performance is improved from the model performance; and instantiating the candidate training data set and the candidate model in the adaptive learning framework in an instance in which the candidate model performance is improved from the model performance. ### Claim 8 (depends on 7) The system of claim 7, wherein generating the assessment of whether the candidate model performance is improved from the model performance includes A/B testing. ### Claim 9 (depends on 8) The system of claim 8, wherein generating the assessment comprises calculating a cross-validation between the candidate model performance and the model performance. ### Claim 10 (depends on 8) The system of claim 8, wherein there are multiple candidate models, and wherein generating the assessment respectively for each of the multiple candidate models is implemented in parallel. ### Claim 11 A computer program product, stored on a non-transitory computer readable medium, comprising instructions that when executed on one or more computers cause the one or more computers to implement an adaptive learning framework for automatically building and maintaining a predictive model for processing dynamic data, wherein the adaptive learning framework is configured to include: the predictive model, wherein the model is configured to generate model output from processing an input data instance received by the adaptive learning framework, and wherein the model output includes a judgment and a confidence value representing certainty of the judgment; a training data set from which the predictive model is derived using machine learning; and a training data manager, wherein the training data manager is configured for curating the training data set; and a labeled data reservoir configured to store labeled data instances that have been processed by the predictive model, wherein the labeled data reservoir includes a pool of possible training data, wherein the set of labeled data instances are not included in the training data; and wherein each labeled data instance is associated with a true label representing the instance; and wherein the training data manager is configured to perform operations comprising: determining whether to update the training data set; in an instance in which the training data set is to be updated, selecting a set of labeled data instances from a labeled data reservoir; and updating the training data using the set of labeled data instances. ### Claim 12 (depends on 11) The computer program product of claim 11, wherein determining whether to update the training data set is based at least in part on analyzing the distribution and quality of the training data. ### Claim 13 (depends on 11) The computer program product of claim 11, wherein determining whether to update the training data set is based at least in part on an accuracy assessment of the model performance. ### Claim 14 (depends on 13) The computer program product of claim 13, wherein the accuracy assessment is based on determining whether the confidence value satisfies a confidence threshold value. ### Claim 15 (depends on 12) The computer program product of claim 12, wherein the current model is a classifier predicting to which of a set of predictive categories an input data instance belongs, wherein a true label associated with a labeled data instance identifies the predictive category to which the labeled data instance belongs, and wherein selecting the set of labeled data instances from the labeled data reservoir is based at least in part on maintaining a class balance within the training data. ### Claim 16 (depends on 11) The computer program product of claim 11, wherein the labeled data reservoir includes labeled data instances that are received from multiple sources, and wherein selecting a labeled data instance from the set of labeled data instances comprises: comparing a source of the labeled data instance with a pre-determined source; and selecting the labeled data instance in an instance in which the source of the labeled data instance matches the pre-determined source. ### Claim 17 (depends on 11) The computer program product of claim 11, wherein the operations further comprise: in response to updating the training data, determining whether to re-train the model; in an instance in which the model is re-trained, generating at least one candidate training data set using the updated training data; deriving a candidate model using the candidate training data set; generating an assessment of whether the candidate model performance is improved from the model performance; and instantiating the candidate training data set and the candidate model in the adaptive learning framework in an instance in which the candidate model performance is improved from the model performance. ### Claim 18 (depends on 17) The computer program product of claim 17, wherein generating the assessment of whether the candidate model performance is improved from the model performance includes A/B testing. ### Claim 19 (depends on 18) The computer program product of claim 18, wherein generating the assessment comprises calculating a cross-validation between the candidate model performance and the model performance. ### Claim 20 (depends on 18) The computer program product of claim 18, wherein there are multiple candidate models, and wherein generating the assessment respectively for each of the multiple candidate models is implemented in parallel.