Content moderation system and indication of reliability of documents

A system comprising:

• one or more computers;
• and
• one or more computer-readable media coupled to the one or more computers having instructions stored thereon which, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
• receiving multiple data inputs in relation to respective electronic documents;
• for each electronic document:
• generating, based on the received input data, data representing one or more features of the electronic document, wherein the one or more features comprise:
• electronic document title, and/or authors of the electronic document, and/or date the electronic document was released, and/or summary of the electronic document, and/or hosted domain, and/or multimedia content included in the electronic document, and/or link sources of the electronic document, and/or references included in the electronic document, and/or style of the electronic document;
• analyzing the generated data representing one or more features of the electronic document to determine one or more reliability scores indicating respective measures of reliability and accuracy of the electronic document, wherein the reliability scores comprise (i) a content reliability score representing a likelihood that the content of the electronic document is accurate and fact-based, and one or more of (ii), an author reliability score representing a likelihood that the author publishes accurate and fact-based documents, and (iii) a domain reliability score representing a likelihood that the domain hosts accurate and fact-based documents, and
• wherein analyzing the generated data to determine a content reliability score for the electronic document comprises:
• analyzing the electronic document to identify (i) one or more entities referenced in the electronic document, and (ii) a summary of the electronic document;
• searching one or more databases using the identified one or more entities referenced in the electronic document and the summary of the electronic document to identify one or more relevant electronic documents;
• determining, a similarity score between the electronic document and each identified one or more relevant electronic documents;
• constructing a set of relevant electronic documents with similarity scores that exceed a first predetermined threshold;
• obtaining, for each of the relevant electronic documents in the set, a respective domain reliability score for the relevant electronic document;
• filtering the set of relevant electronic documents to include relevant electronic documents whose domain reliability score exceeds a second predetermined threshold;
• and
• defining the content reliability score to equal the number of electronic documents in the filtered set, and
• wherein analyzing the generated data to determine an author reliability score comprises:
• identifying an author of the electronic document;
• identifying, from a database of stored electronic documents, one or more similar electronic documents and, for each similar electronic document, respective authors of the similar electronic document;
• identifying, for each similar electronic document, a respective content reliability score for the similar electronic document;
• identifying, for each identified author of a similar electronic document, a current respective author reliability score;
• and
• computing an author reliability score for author of the electronic document based at least on the identified content reliability scores for the similar electronic documents and the identified author reliability scores for the authors of the similar electronic documents, and
• wherein analyzing the generated data to determine a domain reliability score comprises:
• identifying a domain that is hosting the electronic document;
• identifying, from a database, a current reliability score for the identified domain;
• receiving data in relation to crowdsourced information about the identified domain;
• determining a crowdsource reliability score based on the received data;
• and
• determining the domain reliability score based on the current reliability score and the crowdsourced score;
• indicating, based on the content reliability score and one or more of the author reliability score and domain reliability score, whether the electronic document is reliable and accurate or not;
• and
• in response to indicating that the electronic document is neither reliable nor accurate, flagging the content of the electronic document as fake news;
• identifying a subset of the electronic documents that each reference a same entity and are indicated as unreliable and inaccurate;
• and
• analyzing other features of electronic documents included in the subset of electronic documents, the other features comprising domains hosting the electronic documents, dates in which the electronic documents were published, and authors of the electronic documents, to model the propagation of the unreliable and inaccurate electronic documents, wherein the model provides i) a domain in which the unreliable and inaccurate documents originated and ii) a measure of how quickly the unreliable and inaccurate documents propagated to other locations.

One or more non-transitory computer-readable media having instructions stored thereon that, when executed by one or more hardware processors, cause performance of operations comprising:

• receiving multiple data inputs in relation to respective electronic documents;
• for each electronic document:
• generating, based on the received input data, data representing one or more features of the electronic document, wherein the one or more features comprise:
• electronic document title, and/or authors of the electronic document, and/or date the electronic document was released, and/or summary of the electronic document, and/or hosted domain, and/or multimedia content included in the electronic document, and/or link sources of the electronic document, and/or references included in the electronic document, and/or style of the electronic document;
• analyzing the generated data representing one or more features of the electronic document to determine one or more reliability scores indicating respective measures of reliability and accuracy of the electronic document, wherein the reliability scores comprise (i) a content reliability score representing a likelihood that the content of the electronic document is accurate and fact-based, and one or more of (ii), an author reliability score representing a likelihood that the author publishes accurate and fact-based documents, and (iii) a domain reliability score representing a likelihood that the domain hosts accurate and fact-based documents, and
• wherein analyzing the generated data to determine a content reliability score for the electronic document comprises:
• analyzing the electronic document to identify (i) one or more entities referenced in the electronic document, and (ii) a summary of the electronic document;
• searching one or more databases using the identified one or more entities referenced in the electronic document and the summary of the electronic document to identify one or more relevant electronic documents;
• determining, a similarity score between the electronic document and each identified one or more relevant electronic documents;
• constructing a set of relevant electronic documents with similarity scores that exceed a first predetermined threshold;
• obtaining, for each of the relevant electronic documents in the set, a respective domain reliability score for the relevant electronic document;
• filtering the set of relevant electronic documents to include relevant electronic documents whose domain reliability score exceeds a second predetermined threshold;
• and
• defining the content reliability score to equal the number of electronic documents in the filtered set, and
• wherein analyzing the generated data to determine an author reliability score comprises:
• identifying an author of the electronic document;
• identifying, from a database of stored electronic documents, one or more similar electronic documents and, for each similar electronic document, respective authors of the similar electronic document;
• identifying, for each similar electronic document, a respective content reliability score for the similar electronic document;
• identifying, for each identified author of a similar electronic document, a current respective author reliability score;
• and
• computing an author reliability score for author of the electronic document based at least on the identified content reliability scores for the similar electronic documents and the identified author reliability scores for the authors of the similar electronic documents, and
• wherein analyzing the generated data to determine a domain reliability score comprises:
• identifying a domain that is hosting the electronic document;
• identifying, from a database, a current reliability score for the identified domain;
• receiving data in relation to crowdsourced information about the identified domain;
• determining a crowdsource reliability score based on the received data;
• and
• determining the domain reliability score based on the current reliability score and the crowdsourced score;
• indicating, based on the content reliability score and one or more of the author reliability score and domain reliability score, whether the electronic document is reliable and accurate or not;
• and
• in response to indicating that the electronic document is neither reliable nor accurate, flagging the content of the electronic document as fake news;
• identifying a subset of the electronic documents that each reference a same entity and are indicated as unreliable and inaccurate;
• and
• analyzing other features of electronic documents included in the subset of electronic documents, the other features comprising domains hosting the electronic documents, dates in which the electronic documents were published, and authors of the electronic documents, to model the propagation of the unreliable and inaccurate electronic documents, wherein the model provides i) a domain in which the unreliable and inaccurate documents originated and ii) a measure of how quickly the unreliable and inaccurate documents propagated to other locations.