Machine-learning-based systems and methods for quality detection of digital input
AssigneePrime Research Solutions LLC
InventorsJonathan Robinson, Leonid Litman, Miriam Kerstein
Life of this patent
Estimating the term…
Abstract
Systems and methods for assessing and maintaining the quality, validity, and/or accuracy of digital participant input are provided. Methods may include capturing session data via a computing device, building a set of metrics based on the session data, deriving a set of characteristic scores, generating a set of classifications based on the set of characteristic scores, and displaying the set of classifications on a user interface (“UP”) that is associated with a session administrator. The set of classifications may include a flag for rejecting a session and/or a participant when the set of characteristic scores includes a score that is outside a predetermined threshold.
Claims22
A method for assessing one or more characteristics of digital participant input, said participant input that is generated by a participant in response to a survey during a session on a computing device, the method executed by computer code stored in non-transitory memory and running on a processor, the method comprising:
- capturing session data via the computing device, said session data comprising participant activity data;
- building a set of metrics based on the session data;
- deriving a set of characteristic scores by feeding the set of metrics to one or more machine learning (“ML”) ensembles;
- generating a set of classifications based on the set of characteristic scores;
- and
- displaying the set of classifications on a user interface (“UI”) that is associated with a session administrator;
- wherein the set of classifications comprises a flag for rejecting the session and/or the participant when the set of characteristic scores includes a score that is outside a predetermined threshold;
- and
- wherein, when the participant input is in response to one or more questions presented on one or more pages displayed on the computing device during the session, the session data further comprises page data of each page, and capturing the session data comprises:
- scraping a layout of each page to obtain the page data;
- and
- logging cursor movements and/or keystrokes on the computing device to obtain the participant activity data.
A method for assessing one or more characteristics of digital participant input, said participant input that is generated by a participant in response to a survey during a session on a computing device, the method executed by computer code stored in non-transitory memory and running on a processor, the method comprising:
- capturing session data via the computing device, said session data comprising participant activity data;
- building a set of metrics based on the session data;
- deriving a set of characteristic scores by feeding the set of metrics to one or more machine learning (“ML”) ensembles;
- generating a set of classifications based on the set of characteristic score;
- and
- displaying the set of classifications on a user interface (“UI”) that is associated with a session administrator;
- wherein:
- the set of classifications comprises a flag for rejecting the session and/or the participant when the set of characteristic scores includes a score that is outside a predetermined threshold;
- when the participant input is an open-ended input, capturing the session data comprises capturing session content on the computing device, and building the set of metrics comprises performing natural language processing (“NLP”) on the session data;
- and
- building the set of metrics comprises:
- calculating, via a word similarity tool, a relevance of the participant input to a question to which the participant input is responsive, said relevance comprising:
- coverage, said coverage being a ratio of words in the question that have a similar word in the participant input;
- and
- overlap, said overlap being a ratio of words in the participant input that have a similar word in the question;
- and
- calculating the relevance of the participant input to a common corpus, said common corpus that is built from inputs of other participants to the question;
- and
- calculating, via an NLP function, one or more lexical qualities of the participant input.
A method for assessing one or more characteristics of digital participant input, said participant input that is generated by a participant in response to a survey during a session on a computing device, the method executed by computer code stored in non-transitory memory and running on a processor, the method comprising:
- capturing session data via the computing device, said session data comprising participant activity data;
- building a set of metrics based on the session data;
- deriving a set of characteristic scores by feeding the set of metrics to one or more machine learning (“ML”) ensembles;
- generating a set of classifications based on the set of characteristic scores;
- and
- displaying the set of classifications on a user interface (“UI”) that is associated with a session administrator;
- wherein:
- the set of classifications comprises a flag for rejecting the session and/or the participant when the set of characteristic scores includes a score that is outside a predetermined threshold;
- when the participant input is in response to one or more questions presented on one or more pages displayed on the computing device during the session, the session data further comprises page data of each page, and capturing the session data comprises:
- scraping a layout of each page to obtain the page data;
- and
- logging cursor movements and/or keystrokes on the computing device to obtain the participant activity data;
- and
- building the set of metrics comprises:
- calculating spatial metrics based on trajectories of the cursor movements and on the page data, said spatial metrics comprising an absolute deviation from an ideal trajectory of the cursor movement and a maximum deviation from the ideal trajectory of the cursor movement;
- calculating temporal metrics based on time measurements of participant interaction with a page;
- and
- when an option to exclude keystroke metrics has not been selected, calculating keystroke metrics based on flight time, dwell time, and/or pause time of the keystrokes.
Dependencies are inferred from the claim text.