Eyes-on analysis results for improving search quality

A computing system that trains a learn-to-rank type model to improve quality and ranking of search results generated by the learn-to-rank type model and to perform certain actions when generating the search results to improve relevance of the search results for corresponding queries, the computing system comprising:

• one or more processors;
• and
• one or more storage devices that store instructions that are executable by the one or more processors to cause the computing system to:
• access client data, which includes a plurality of user queries;
• extract features and patterns of the client data while keeping values of the user queries private, wherein the extracted features and patterns of the client data include:
• a first pattern of types of grammar used in the plurality of user queries and a second pattern of formatting of terms used in the plurality of user queries;
• infer a search result quality of search result content based on user behavior data included in the client data, wherein the inferred search result quality is further based on analyzing click signals made by a user at a user system, and wherein the click signals are further used to determine at least one of:
• a frequency of interacting with the search result content,
• whether the user navigated to another resource that is referenced in the search result content,
• whether the user copied or saved any of the content,
• which one or more search results the user interacted with the most out of the different search results in the search result content, or
• which search results the user did not interact with at all from the search result content;
• generate labelled training data for the learn-to-rank type model by labelling features and patterns of the search result content for correspondingly paired search query features and pattern with corresponding relevance scores based on the inferred search result quality and, additionally, by at least pairing or labelling actions associated with generating the search result content;
• identify conflicting relevance scores for common features and patterns of the search result content included in the labelled training data;
• generate filtered and labelled training data by filtering the labelled training data based on omitting or resolving the identified conflicting relevance scores to reduce noise in the labelled training data that is caused by conflicting relevance scores for common features and patterns of the search result content included in the labelled training data;
• apply the filtered and labelled training data to the learn-to-rank type model to train the learn-to-rank type model to improve search quality of search results provided by the learn-to-rank type model when new user queries are provided to the learn-to-rank type model that have features and patterns corresponding to the filtered and labelled training data and to further train the learn-to-rank type model to perform different actions for different types of queries to improve quality and relevance of the search results generated by the learn-to-rank type model for the new and different queries;
• and
• access analysis data that identifies analyst assessments of quality for the different search result content based on the plurality of user queries, wherein the analysis data includes labels and annotations associated with the values of the user queries.