Document analysis using machine learning and neural networks

A system, comprising:

• a plurality of machine learning models, each of the plurality of machine learning models trained with initial training data generated by a training method including, for each machine learning model:
• obtaining a selection from equipment of a specific user responsive presentation of proposed words according to an A/B test;
• identifying, by application of the selection to a dedicated neural network, a pre-trained, baseline machine learning model;
• displaying a plurality of electronic documents for evaluation on a display of a computing device to obtain a displayed plurality of electronic documents;
• tracking, by an eye movement tracking device, a gaze of the specific user as the specific user reads each of the displayed plurality of electronic documents on the display to determine indications of pauses indicative of the gaze of the specific user in reading each of the displayed plurality of electronic documents for greater than a specified period of time by the specific user;
• correlating objects on each of the displayed plurality of electronic documents to the pauses in reading by the specific user;
• identifying features for the machine learning model to obtain identified features based on the objects and textual analysis of each of the plurality of electronic documents;
• presenting information related to each identified feature to the specific user;
• obtaining from the specific user a descriptor defining each of the identified features and a value for each of the identified features indicating a relative importance or applicability of each of the identified features;
• associating obtained descriptors and values with each of the identified features;
• obtaining from the specific user an overall value for each of the plurality of electronic documents indicating an overall applicability of each of the plurality of electronic documents with respect to specific requirements of the specific user;
• associating obtained overall values with each of the plurality of electronic documents;
• and
• combining the pre-trained, baseline machine learning model, identified features, associated descriptors, associated values, and associated overall values as training data for the machine learning model associated with the specific user;
• and
• a processing system including a processor configured to accept outputs from each of a plurality of neural networks executing the plurality of machine learning models and perform specified activities based on the outputs of each of the plurality of neural networks,
• wherein each machine learning model in a subset of the plurality of machine learning models is trained using training data based on requirements of a different user,
• wherein at least one of the subset of the plurality of machine learning models evaluates a particular electronic document, and based on an evaluation identifies at least one aspect of the particular electronic document requiring additional information, and
• wherein an output of the at least one of the plurality of neural networks causes the processing system to perform an activity to obtain the additional information related to the identified at least one aspect of the particular electronic document.

A method for evaluating electronic documents using a plurality of machine learning models, the method comprising:

• training the plurality of machine learning models for a plurality of different specific users by a training method including, for each machine learning model:
• obtaining a selection from equipment of a specific user responsive presentation of proposed words according to an A/B test;
• identifying, by application of the selection to a neural network, a pre-trained, baseline machine learning model;
• displaying a plurality of electronic documents for evaluation on a display of a computing device to obtain displayed plurality of electronic documents;
• tracking, by an eye movement tracking device, a gaze of the specific user as the specific user reads each of the displayed plurality of electronic documents on the display to determine indications of pauses in reading each of the displayed plurality of electronic documents for greater than a specified period of time by the specific user, wherein the pauses are indicative of the gaze of the specific user;
• correlating objects on each of the displayed plurality of electronic documents to the pauses in reading by the specific user;
• identifying features for the machine learning model to obtain identified features based on the objects and textual analysis of each of the plurality of electronic documents;
• presenting information related to each identified feature to the specific user;
• obtaining from the specific user a descriptor defining each of the identified features and a value for each of the identified features indicating a relative importance or applicability of each of the identified features;
• associating obtained descriptors and values with each of the identified features;
• obtaining from the specific user an overall value for each of the plurality of electronic documents indicating an overall applicability of each of the plurality of electronic documents with respect to specific requirements of the specific user;
• associating obtained overall values with each of the plurality of electronic documents;
• and
• combining the pre-trained, baseline machine learning model, identified features, associated descriptors, associated values, and associated overall values as training data for the machine learning model associated with the specific user;
• evaluating the electronic documents by each machine learning model of the plurality of machine learning models based on requirements of the specific user of the plurality of different specific users used to train each of the plurality of machine learning models;
• outputting a value or an indication for the electronic documents by each neural network executing one of the plurality of machine learning models;
• combining outputs of each neural network with a combiner;
• and
• generating a final a score or indication indicating a degree of compliance with combined requirements of the plurality of different specific users based on the combined outputs of the plurality of neural networks.

A method for providing training data for a machine learning model, the method comprising:

• obtaining a selection from equipment of a specific user responsive presentation of proposed words according to an A/B test;
• identifying, by application of the selection to a dedicated neural network, a pre-trained, baseline machine learning model;
• displaying a plurality of electronic documents for evaluation on a display of a computing device to obtain a displayed plurality of electronic documents;
• tracking, by an eye movement tracking device, a gaze of the specific user as the specific user reads each of the displayed plurality of electronic documents on the display to determine indications of pauses in reading each of the displayed plurality of electronic documents, wherein the pauses are indicative of the gaze of the specific user;
• annotating electronic documents displayed on a display of a computing device with one or more annotations;
• correlating objects on the electronic documents to the gaze of the specific user and at least one of the one or more annotations;
• identifying features for the machine learning model to obtain identified features based on the gaze of the specific user and annotated objects and textual analysis of each of the electronic documents;
• presenting information related to each identified feature to the specific user;
• obtaining from the specific user a descriptor defining each of the identified features and a value for each of the identified features indicating a relative importance or applicability of each of the identified features;
• associating obtained descriptors and values with each of the identified features;
• obtaining from the specific user an overall value for each of the plurality of electronic documents indicating an overall applicability of each of the plurality of electronic documents with respect to specific requirements of the specific user;
• associating obtained overall values with each of the plurality of electronic documents;
• and
• combining the pre-trained, baseline machine learning model, identified features, associated descriptors, associated values, and associated overall values as training data for the machine learning model associated with the specific user.