# US 11,907,315 B1 · Managing search engines based on search perform metrics > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,315 B1 - Document ID: US11907315B1 - Status: Granted - Kind: B1 - Application number: 17/937,180 - Filed: 2022-09-30 - Granted: 2024-02-20 - Earliest claimed: 2022-09-30 - Assignees: Intuit Inc. - Applicants: Intuit Inc. - Inventors: Wendi Cui; Damien Lopez; Colin Ryan - Web page: https://patentgrep.com/patent/US11907315B1 ## CPC - G06F 16/9535: Search customisation based on user profiles and personalisation - H04L 67/535: {Tracking the activity of the user (network monitoring arrangements H04L43/00; recording of computer activity G06F11/34)} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract Certain aspects of the present disclosure provide techniques for managing a search engine based on search performance metrics. An example method generally includes dividing a set of search history data into a first subset of search history data and a second subset of search history data. The first subset of data is associated with interaction with search results, and the second subset of data is associated with non-interaction with search results. A first quality score is generated for searches in the first subset of data. A second quality score is generated for searches in the second subset of data based on different search intents identified for each temporally related group in the second subset of data. An overall quality score is generated for a search engine, and one or more actions with respect to the search engine are taken based on the overall quality score. ## Claims ### Claim 1 A method, comprising: dividing a set of search history data into a first subset of search history data and a second subset of search history data, wherein the first subset of search history data comprises search history data associated with interaction with one or more results of a first corresponding search and the second subset of search history data comprises search history data associated with one or more results of a second corresponding search other than being clicked on by respective users; generating a first quality score for searches in the first subset of search history data; generating a second quality score for searches in the second subset of search history data based on different search intents identified for each temporally related group in the second subset of search history data; generating an overall quality score for a search engine based on the first quality score, the second quality score, and a total number of search events in the set of search history data; and taking one or more actions based on the overall quality score. ### Claim 2 (depends on 1) The method of claim 1, wherein the one or more temporally related groups comprise groups of search events occurring within a threshold amount of time after an initial search event associated with one or more results other than being clicked on by respective users of the initial search event was performed. ### Claim 3 (depends on 1) The method of claim 1, wherein generating the second quality score for searches in the second subset of search history data comprises, for each temporally related group: generating a plurality of search term pairings, each respective search term pairing corresponding to consecutive queries input into the search engine; mapping queries in each respective search term pairing of the plurality of search term pairings into embedding representations of the queries; identifying one or more search intents within the temporally related group based on a similarity score between the queries in each respective search term pairing and a threshold similarity score; calculating a respective quality score for each respective search intent of the one or more search intents; and generating the second quality score based on the respective quality score for each respective search intent of the one or more search intents. ### Claim 4 (depends on 3) The method of claim 3, wherein the similarity score comprises a cosine similarity score. ### Claim 5 (depends on 4) The method of claim 4, wherein the cosine similarity score comprises a score generated by a machine learning model trained to generate the similarity score based on a semantic similarity between the queries in each search pairing. ### Claim 6 (depends on 3) The method of claim 3, wherein calculating the respective quality score for each respective search intent comprises calculating a click rank score for the respective search intent across queries associated with the respective search intent such that interaction with a search result for a second query in a group of queries associated with the respective search intent also represents interaction with a search result for a first query in the group of queries. ### Claim 7 (depends on 6) The method of claim 6, wherein the respective quality score is calculated based on an average click rank score calculated based on a rank of each search result with which a user interacts across search results associated with each search query in the temporally related group. ### Claim 8 (depends on 1) The method of claim 1, wherein the first quality score comprises a mean reciprocal rank score calculated for searches in the first subset of search history data. ### Claim 9 (depends on 1) The method of claim 1, wherein the overall quality score for the search engine comprises a sum of the first quality score and the second quality score, divided by the total number of search events in the set of search history data. ### Claim 10 (depends on 1) The method of claim 1, wherein the one or more actions comprise triggering re-training of one or more machine learning models used by the search engine to deliver relevant data in response to a received query. ### Claim 11 A system, comprising: a memory having executable instructions stored thereon; and a processor configured to execute the executable instructions in order to cause the system to: divide a set of search history data into a first subset of search history data and a second subset of search history data, wherein the first subset of search history data comprises search history data associated with interaction with one or more results of a first corresponding search and the second subset of search history data comprises search history data associated with one or more results of a second corresponding search not other than being clicked on by respective users; generate a first quality score for searches in the first subset of search history data; generate a second quality score for searches in the second subset of search history data based on different search intents identified for each temporally related group in the second subset of search history data; generate an overall quality score for a search engine based on the first quality score, the second quality score, and a total number of search events in the set of search history data; and take one or more actions based on the overall quality score. ### Claim 12 (depends on 11) The system of claim 11, wherein the one or more temporally related groups comprise groups of search events occurring within a threshold amount of time after an initial search event associated with one or more results of the initial search event other than being clicked on by respective users was performed. ### Claim 13 (depends on 11) The system of claim 11, wherein in order to generate the second quality score for searches in the second subset of search history data, the processor is configured to cause the system to, for each temporally related group: generate a plurality of search term pairings, each respective search term pairing corresponding to consecutive queries input into the search engine; map queries in each respective search term pairing of the plurality of search term pairings into embedding representations of the queries; identify one or more search intents within the temporally related group based on a similarity score between the queries in each respective search term pairing and a threshold similarity score; calculate a respective quality score for each respective search intent of the one or more search intents; and generate the second quality score based on the respective quality score for each respective search intent of the one or more search intents. ### Claim 14 (depends on 13) The system of claim 13, wherein the similarity score comprises a cosine similarity score. ### Claim 15 (depends on 14) The system of claim 14, wherein the cosine similarity score comprises a score generated by a machine learning model trained to generate the similarity score based on a semantic similarity between the queries in each search pairing. ### Claim 16 (depends on 13) The system of claim 13, wherein in order to calculate the respective quality score for each respective search intent, the processor is configured to cause the system to calculate a click rank score for the respective search intent across queries associated with the respective search intent such that interaction with a search result for a second query in a group of queries associated with the respective search intent also represents interaction with a search result for a first query in the group of queries. ### Claim 17 (depends on 16) The system of claim 16, wherein the respective quality score is calculated based on an average click rank score calculated based on a rank of each search result with which a user interacts across search results associated with each search query in the temporally related group. ### Claim 18 (depends on 11) The system of claim 11, wherein the overall quality score for the search engine comprises a sum of the first quality score and the second quality score, divided by the total number of search events in the set of search history data. ### Claim 19 (depends on 11) The system of claim 11, wherein the one or more actions comprises triggering re-training of one or more machine learning models used by the search engine to deliver relevant data in response to a received query. ### Claim 20 A method, comprising: dividing a set of search history data into a first subset of search history data and a second subset of search history data, wherein the first subset of search history data comprises search history data associated with interaction with one or more results of a first corresponding search and the second subset of search history data comprises search history data associated with one or more results of a second corresponding search other than being clicked on by respective users; generating a first quality score for searches in the first subset of search history data; generating a plurality of search term pairings from the second subset of search history data, each word pairing corresponding to consecutive queries input into a search engine; mapping queries in each respective search term pairing of the plurality of search term pairings into embedding representations of the queries; identifying one or more search intents within the generated plurality of search term pairings based on a similarity score between the queries in each respective search term pairing and a threshold similarity score; calculating a respective quality score for each respective search intent of the one or more search intents; and generating a second quality score for the second subset of search history data based on the respective quality score for each respective search intent of the one or more search intents; generating an overall quality score for a search engine based on the first quality score, the second quality score, and a total number of search events in the set of search history data; and taking one or more actions based on the overall quality score.