Systems and methods for document analysis to produce, consume and analyze content-by-example logs for documents

A method for document analysis, comprising:

• receiving, by a receiving party, a content-by-example log, the content-by-example log including an entry for each withheld document of a set of withheld documents, wherein the set of withheld documents is inaccessible to the receiving party, wherein the entry for each withheld document associates an identifier for the corresponding withheld document with example identifiers for a set of example documents, wherein the set of example documents exemplify the corresponding withheld document, wherein the set of example documents are disclosed documents accessible to the receiving party;
• storing the content-by-example log at a data store;
• analyzing the content-by-example log to determine identifiers of withheld documents of interest by:
• transforming the content-by-example log into a feature vector index, wherein the feature vector index comprises:
• a feature vector associated with each of the identifiers of the withheld documents, wherein the feature vector comprises:
• a set of features determined from the set of example documents for the corresponding withheld document, wherein creating a respective feature vector for each withheld document comprises:
• generating a document feature vector for each example document identified in the content-by-example log as being associated with the withheld document, wherein the document feature vector comprises a weighted set of text based features determined from the respective example document, wherein the feature vector associates with the identifier for the withheld document in the feature vector index by using the document feature vectors generated for each example document of the set of example documents;
• and
• determining the identifiers of withheld documents of interest based on the feature vector index by:
• obtaining labels associated with identifiers of withheld documents;
• obtaining a supervised machine learning model trained at a first time based on obtained labels for documents;
• further training, at a second time after the first time, the supervised machine learning model using newly obtained labels for withheld documents identified in the content-by-example log, wherein the further training at the second time of the supervised machine learning model utilizes features determined from the example documents and provided by the feature vector index, wherein the features are associated with the withheld documents;
• ranking identifiers for withheld documents of the content-by-example log based on the feature vector index using the further trained supervised machine learning model;
• and
• selecting a number of top ranked identifiers of withheld documents as identifiers of the set of withheld documents of interest;
• and
• generating requests for a producing party having the set of withheld documents, wherein the generated requests correspond to at least some of the set of withheld documents of interest by specifying a subset of identifiers of the at least some of the set of withheld documents of interest.

A non-transitory computer readable medium, comprising instructions for:

• receiving, by a receiving party, a content-by-example log, the content-by-example log including an entry for each withheld document of a set of withheld documents, wherein the set of withheld documents is inaccessible to the receiving party, wherein the entry for each withheld document associates an identifier for the corresponding withheld document with example identifiers for a set of example documents, wherein the set of example documents exemplify the corresponding withheld document, wherein the set of example documents are disclosed documents accessible to the receiving party;
• storing the content-by-example log at a data store;
• analyzing the content-by-example log to determine identifiers of withheld documents of interest by:
• transforming the content-by-example log into a feature vector index, wherein the feature vector index comprises:
• a feature vector associated with each of the identifiers of the withheld documents, wherein the feature vector comprises:
• a set of features determined from the set of example documents for the corresponding withheld document, wherein creating a respective feature vector for each withheld document comprises:
• generating a document feature vector for each example document identified in the content-by-example log as being associated with the withheld document, wherein the document feature vector comprises a weighted set of text based features determined from the respective example document,
• wherein the feature vector associates with the identifier for the withheld document in the feature vector index by using the document feature vectors generated for each example document of the set of example documents;
• and
• determining the identifiers of withheld documents of interest based on the feature vector index by:
• obtaining labels associated with identifiers of withheld documents;
• obtaining a supervised machine learning model trained at a first time based on obtained labels for documents;
• further training, at a second time after the first time, the supervised machine learning model using newly obtained labels for withheld documents identified in the content-by-example log, wherein the further training at the second time of the supervised machine learning model utilizes features determined from the example documents and provided by the feature vector index, wherein the features are associated with the withheld documents;
• ranking identifiers for withheld documents of the content-by-example log based on the feature vector index using the further trained supervised machine learning model;
• and
• selecting a number of top ranked identifiers of withheld documents as identifiers of the set of withheld documents of interest;
• and
• generating requests for a producing party having the set of withheld documents, wherein the generated requests correspond to at least some of the set of withheld documents of interest by specifying a subset of identifiers of the at least some of the set of withheld documents of interest.