US 12,118,311 B1Granted

Research replicability assessment

AssigneeTwo Six Labs, LLC

InventorsJames Gentile, Chae A. Clark, Ben Gelman

Life of this patent

Estimating the term…

Loading patent…

Abstract

A machine learning approach for evaluating the replicability of research receives a scholarly paper or publication about a research effort, and applies learning models and natural language processing (NLP) to identify replicability of the asserted results. Rule based and machine learning approaches employ models based on preexisting research papers having defined replicability, and compares language features, research parameters and factual assertions extracted from a research paper for evaluation. A pipeline of evaluations and feature extraction analyzes the content of the research paper and renders a prediction value that the described research can be repeated with similar results.

Claims19

Dependent claims

A method for evaluating replicability of a research paper, comprising:

receiving a narrative representation of a research effort, the narrative representation including a natural language description of the research effort in the form of paragraphs and sentences;
computing, for each paragraph, a metric of readability, subjectivity and sentiment;
and
associating the computed metrics for readability, subjectivity and sentiment with a set of features for each respective paragraph;
analyzing the narrative representation for identifying assertions made by one or more sentences in the narrative representation, the analysis based on extracted natural language features from the sentences, identifying the assertions further comprising:
generating a model of research claims made in a previous narrative representation by training the model of research claims based on features defined by the plurality of sentences;
applying the trained model of research claims to the narrative representation for determining, for each sentence of a plurality of sentences, if the sentence defines an assertion, and if so, computing a score for each of a summary claim, results claim, concrete hypothesis and test specification;
and
associating the computed scores with the features for each respective sentence of the plurality of sentences;
combining the features for each respective paragraph with the features and the computed scores for each respective sentence in a hierarchical script;
comparing the assertions to established assertions represented by a replicability model of previous narrative representations based on features of both the paragraphs and the plurality of sentences;
and
based on the comparison, predicting a repeatability score indicative of a likelihood of replicating the identified assertions.

10 more elements

A computing device for evaluating replicability of a research paper, comprising:

an interface to receive a narrative representation of a research effort, the narrative representation including a natural language description of the research effort in the form of paragraphs and sentences;
a natural language processor (NLP) configured to analyze the narrative representation for identifying assertions made by a plurality of sentences in the narrative representation, the analysis based on extracted natural language features from the sentences;
a machine learning (ML) processor responsive to the interface to compare the assertions to established assertions represented by a replicability model of previous narrative representations based on features of both the paragraphs and the plurality of sentences, the MP processor configured to:
compute, for each paragraph, a metric of readability, subjectivity and sentiment;
and
associate the computed metrics for readability, subjectivity and sentiment with a set of features for each respective paragraph, identifying the assertions further comprising:
generating a model of research claims made in a previous narrative representation by training the model of research claims based on features defined by the plurality of sentences;
applying the trained model of research claims to the narrative representation for determining, for each sentence of a plurality of sentences, if the sentence defines an assertion, and if so, computing a score for each of a summary claim, results claim, concrete hypothesis and test specification;
and
associating the computed score with the features for each respective sentence of the plurality of sentences;
combining the features for each respective paragraph with the features and the computed score for each respective sentence in a hierarchical script;
and
a UI rendering to, based on the comparison, render a prediction of a repeatability score indicative of a likelihood of replicating the identified assertions.

10 more elements

A computer program embodying program code on a non-transitory medium that, when executed by a processor, performs steps for implementing a method of evaluating document research efforts for replicability, the method comprising:

parsing a narrative representation of a research effort, the narrative representation including a plurality of sentences arranged in a plurality of paragraphs of the sentences;
computing, for each paragraph of the plurality of paragraphs, a metric of readability, subjectivity and sentiment;
and
associating the computed metrics for readability, subjectivity and sentiment with a set of features for each respective paragraph;
classifying each of the sentences as belonging to one of a plurality of contexts;
identifying natural language features indicative of a language quality of each paragraph of the plurality of paragraphs;
identifying, based on the natural language features, sentences containing assertions of the narrative representation, the assertions designating an outcome result of the narrative representation, identification of the assertions further comprising:
generating a model of research claims made in a previous narrative representation by training the model of research claims based on features defined by the plurality of sentences;
applying the trained model of research claims to the narrative representation for determining, for each sentence of a plurality of sentences, if the sentence defines an assertion, and if so, computing a score for each of a summary claim, results claim, concrete hypothesis and test specification;
and
associating the computed score with the features for each respective sentence of the plurality of sentences;
and
combining the features for each respective paragraph with the features and the computed score for each respective sentence in a hierarchical script;
comparing the assertions to established assertions represented by a model of replicability of previous narrative representations based on features of both the paragraphs and the plurality of sentences;
evaluating, for each of the assertions, a probability indicative of a random effect causing the outcome result;
and
aggregating, based on the comparisons, each of the evaluated assertions based on a sample size and an effect size on which the outcome was based, for computing a replication probability of the respective assertions in the narrative representation.

14 more elements

Dependencies are inferred from the claim text.