# US 2021/0240937 A1 · System and method for artificial intelligence driven document analysis, including automated reuse of predictive coding rules based on management and curation of datasets or models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0240937 A1 - Document ID: US20210240937A1 - Status: Published application - Kind: A1 - Application number: 16/881,274 - Filed: 2020-05-22 - Published: 2021-08-05 - Earliest claimed: 2020-01-31 - Assignees: Cs Disco, Inc. - Applicants: CS Disco, Inc. - Inventors: Alan Justin Lockett; Verlyn Michael Fischer; Richard Alan Vestal; Jesse Abraham Ramos; Robert Duane Harrington; Brian Daniel Luskey - Web page: https://patentgrep.com/patent/US20210240937A1 ## CPC - G06N 20/20: Ensemble learning - G06F 40/30: Semantic analysis - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 3/096: Transfer learning - G06N 5/04: Inference or reasoning models - G06N 20/00: Machine learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Artificial intelligence based document analysis systems and methods are disclosed. Embodiments of document analysis systems may allow the reuse of coded datasets defined in association with a particular code by allowing these datasets to be bundled to define a dataset for another code, where that code may be associated with a target corpus of documents. A model can then be trained based on that dataset and used to provide predictive scores for the documents of the target corpora with respect to the code. Furthermore, this code can be applied not just to the target corpus of documents, but additionally can be applied against any other corpora. ## Claims ### Claim 1 A system for document analysis comprising: a processor; a data store, comprising a first target corpus of electronic document and a second target corpus of electronic documents; and a non-transitory computer readable medium comprising instructions for: receiving a definition of a first code in association with the first target corpus; creating a first dataset for the first code; receiving an indication that the first code is to be boosted with a second code, wherein the second code is associated with the second target corpus and the second code is associated with a second dataset, the second dataset comprising a first set of positive signals associated with the second code and documents of the second corpus and a first set of negative signals associated with the second code and documents of the second target corpus; adding the second dataset associated with the second code and the second target corpus to the first dataset of the first code such that the first dataset comprises a boosting dataset including the second dataset comprising the first set of positive signals associated with the second code and documents of the second corpus and the first set of negative signals associated with the second code and documents of the second target corpus; training a first machine learning model for the first code on the boosting dataset of the first dataset; generating predictive scores for the first code for documents of the first target corpus using the first machine learning model; and presenting the predictive scores in association with documents of the first target corpus to a user. ### Claim 2 (depends on 1) The system of claim 1, wherein the first machine learning model is trained only on the boosting dataset. ### Claim 3 (depends on 1) The system of claim 1, wherein the instructions further comprise instructions for: receiving coding decisions for documents of the first target corpus with respect to the first code; storing the coding decisions in association with the first dataset associated with the first code such that the first dataset comprises a native dataset comprising a second set of positive signals associated with the first code and documents of the first corpus and a second set of negative signals associated with the first code and documents of the first corpus; training a second machine learning model for the first code on the native dataset of the first dataset; evaluating the first machine learning model and the second machine learning model to select a best machine learning model based on a test set of documents of the first corpus; generating predictive scores for the first code for documents of the first target corpus using the best machine learning model; and presenting the predictive scores in association with documents of the first target corpus to a user. ### Claim 4 (depends on 3) The system of claim 3, wherein evaluating the first machine learning model and the second machine learning model to select the best machine learning model further comprises, determining a current best machine learning model from the first machine learning model and the second machine learning model, and comparing the current best machine learning model to a previous best machine learning model using the test set of documents of the first target corpus to select the best model ### Claim 5 (depends on 4) The system of claim 4, wherein the first machine learning model for the first code is trained on the boosting dataset of the first dataset and the native dataset of the first dataset. ### Claim 6 (depends on 1) The system of claim 1, wherein the boosting dataset includes a plurality of datasets, each dataset having respective positive signals and negative signals and training the first machine learning model for the first code on the boosting dataset comprises selecting a positive signal and a negative signal from each of the plurality of datasets according to a balancing method. ### Claim 7 (depends on 1) The system of claim 1, wherein indication that the first code is to be boosted with the second code is received through an interface that presents the second code as one of a plurality of codes, each of the plurality of codes presented with an associated textual description in the interface. ### Claim 8 A method for document analysis comprising: receiving a definition of a first code in association with a first target corpus of electronic documents; creating a first dataset for the first code; receiving an indication that the first code is to be boosted with a second code, wherein the second code is associated with a second target corpus of electronic documents and the second code is associated with a second dataset, the second dataset comprising a first set of positive signals associated with the second code and documents of the second corpus and a first set of negative signals associated with the second code and documents of the second target corpus; adding the second dataset associated with the second code and the second target corpus to the first dataset of the first code such that the first dataset comprises a boosting dataset including the second dataset comprising the first set of positive signals associated with the second code and documents of the second corpus and the first set of negative signals associated with the second code and documents of the second target corpus; training a first machine learning model for the first code on the boosting dataset of the first dataset; generating predictive scores for the first code for documents of the first target corpus using the first machine learning model; and presenting the predictive scores in association with documents of the first target corpus to a user. ### Claim 9 (depends on 8) The method of claim 8, wherein the first machine learning model is trained only on the boosting dataset. ### Claim 10 (depends on 8) The method of claim 8, further comprising: receiving coding decisions for documents of the first target corpus with respect to the first code; storing the coding decisions in association with the first dataset associated with the first code such that the first dataset comprises a native dataset comprising a second set of positive signals associated with the first code and documents of the first corpus and a second set of negative signals associated with the first code and documents of the first corpus; training a second machine learning model for the first code on the native dataset of the first dataset; evaluating the first machine learning model and the second machine learning model to select a best machine learning model based on a test set of documents of the first corpus; generating predictive scores for the first code for documents of the first target corpus using the best machine learning model; and presenting the predictive scores in association with documents of the first target corpus to a user. ### Claim 11 (depends on 10) The method of claim 10, wherein evaluating the first machine learning model and the second machine learning model to select the best machine learning model further comprises, determining a current best machine learning model from the first machine learning model and the second machine learning model, and comparing the current best machine learning model to a previous best machine learning model using the test set of documents of the first target corpus to select the best model ### Claim 12 (depends on 11) The method of claim 11, wherein the first machine learning model for the first code is trained on the boosting dataset of the first dataset and the native dataset of the first dataset. ### Claim 13 (depends on 8) The method of claim 8, wherein the boosting dataset includes a plurality of datasets, each dataset having respective positive signals and negative signals and training the first machine learning model for the first code on the boosting dataset comprises selecting a positive signal and a negative signal from each of the plurality of datasets according to a balancing method. ### Claim 14 (depends on 8) The method of claim 8, wherein indication that the first code is to be boosted with the second code is received through an interface that presents the second code as one of a plurality of codes, each of the plurality of codes presented with an associated textual description in the interface. ### Claim 15 A non-transitory computer readable medium, comprising instruction for: receiving a definition of a first code in association with a first target corpus of electronic documents; creating a first dataset for the first code; receiving an indication that the first code is to be boosted with a second code, wherein the second code is associated with a second target corpus of electronic documents and the second code is associated with a second dataset, the second dataset comprising a first set of positive signals associated with the second code and documents of the second corpus and a first set of negative signals associated with the second code and documents of the second target corpus; adding the second dataset associated with the second code and the second target corpus to the first dataset of the first code such that the first dataset comprises a boosting dataset including the second dataset comprising the first set of positive signals associated with the second code and documents of the second corpus and the first set of negative signals associated with the second code and documents of the second target corpus; training a first machine learning model for the first code on the boosting dataset of the first dataset; generating predictive scores for the first code for documents of the first target corpus using the first machine learning model; and presenting the predictive scores in association with documents of the first target corpus to a user. ### Claim 16 (depends on 15) The non-transitory computer readable medium of claim 15, wherein the first machine learning model is trained only on the boosting dataset. ### Claim 17 (depends on 15) The non-transitory computer readable medium of claim 15, further comprising instructions for: receiving coding decisions for documents of the first target corpus with respect to the first code; storing the coding decisions in association with the first dataset associated with the first code such that the first dataset comprises a native dataset comprising a second set of positive signals associated with the first code and documents of the first corpus and a second set of negative signals associated with the first code and documents of the first corpus; training a second machine learning model for the first code on the native dataset of the first dataset; evaluating the first machine learning model and the second machine learning model to select a best machine learning model based on a test set of documents of the first corpus; generating predictive scores for the first code for documents of the first target corpus using the best machine learning model; and presenting the predictive scores in association with documents of the first target corpus to a user. ### Claim 18 (depends on 17) The non-transitory computer readable medium of claim 17, wherein evaluating the first machine learning model and the second machine learning model to select the best machine learning model further comprises, determining a current best machine learning model from the first machine learning model and the second machine learning model, and comparing the current best machine learning model to a previous best machine learning model using the test set of documents of the first target corpus to select the best model ### Claim 19 (depends on 18) The non-transitory computer readable medium of claim 18, wherein the first machine learning model for the first code is trained on the boosting dataset of the first dataset and the native dataset of the first dataset. ### Claim 20 (depends on 15) The non-transitory computer readable medium of claim 15, wherein the boosting dataset includes a plurality of datasets, each dataset having respective positive signals and negative signals and training the first machine learning model for the first code on the boosting dataset comprises selecting a positive signal and a negative signal from each of the plurality of datasets according to a balancing method. ### Claim 21 (depends on 15) The non-transitory computer readable medium of claim 15, wherein indication that the first code is to be boosted with the second code is received through an interface that presents the second code as one of a plurality of codes, each of the plurality of codes presented with an associated textual description in the interface.