# US 2020/0401644 A1 · Two-stage training with non-randomized and randomized data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401644 A1 - Document ID: US20200401644A1 - Status: Published application - Kind: A1 - Application number: 16/449,149 - Filed: 2019-06-21 - Published: 2020-12-24 - Earliest claimed: 2019-06-21 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Daniel Sairom Krishnan Hewlett; Dan Liu; Qi Guo; Wenxiang Chen; Xiaoyi Zhang; Lester Gilbert Cottle; Xuebin Yan; Yu Gong; Haitong Tian; Siyao Sun; Pei-Lun Liao - Web page: https://patentgrep.com/patent/US20200401644A1 ## CPC - G06F 16/9538: Presentation of query results - G06N 3/084: Backpropagation, e.g. using gradient descent - G06F 16/2237: {Vectors, bitmaps or matrices} - G06F 40/205: Parsing - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06N 5/022: Knowledge engineering; Knowledge acquisition - G06N 20/00: Machine learning - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract In an example embodiment, position bias and other types of bias may be compensated for by using two-phase training of a machine-learned model. In a first phase, the machine-learned model is trained using non-randomized training data. Since certain types of machine-learned models, such as those involving deep learning (e.g., neural networks) require a lot of training data, this allows the bulk of the training to be devoted to training using non-randomized training data. However, since this non-randomized training data may be biased, a second training phase is then used to revise the machine-learned model based on randomized training data to remove the bias from the machine-learned model. Since this randomized training data may be less plentiful, this allows the deep learning machine-learned model to be trained to operate in an unbiased manner without the need to generate additional randomized training data. ## Claims ### Claim 1 A system for returning search results in an online computer system, the system comprising: a computer readable medium having instructions stored there on, which, when executed by a processor, cause the system to: retrieve training data, the training data comprising a plurality of search results, a plurality of sample labels, and activity and usage information pertaining to actions taken by users on the search results on an online service, the training data comprising randomized and non-randomized training data, the randomized training data derived from historical presentation of a first subset of the plurality of search results using a randomized ranking and the non-randomized training data derived from historical presentation of a second subset of the plurality of search results using a non-randomized ranking; for each sample search result in a plurality of sample search results in the non-randomized training data: parse the sample search result and the activity and usage information pertaining to the sample search result in the non-randomized training data to extract a first set of one or more features; feed a sample member label corresponding to the sample search result and the extracted first set of one or more features into a first machine learning algorithm to train a machine-learned model to output a score indicative of a probability that an action will be taken, in the online service, pertaining to the sample search result; for each sample search result in a plurality of sample search results in the randomized training data: parse the sample search result and the activity and usage information pertaining to the sample search result in the randomized training data to extract a second set of one or more features; and feed a sample member label corresponding to the sample search result in the randomized training data and the extracted second set of one or more features into the first machine learning algorithm to revise the machine-learned model. ### Claim 2 (depends on 1) The system of claim 1, wherein the instructions further cause the system to: obtain a plurality of candidate search results in response to a query corresponding to a searcher; for each candidate search result from the plurality of candidate search results: obtain activity and usage information for the candidate search result; parse the candidate search result and the activity and usage information corresponding to the candidate search result to extract a third set of one or more features; feed the extracted third set of one or more features into the machine-learned model, outputting a score for the candidate search result; rank the plurality of candidate search results by their corresponding scores; and return one or more of the plurality of candidate search results based on the ranking. ### Claim 3 (depends on 1) The system of claim 1, wherein the non-randomized training data is more plentiful than the randomized training data. ### Claim 4 (depends on 1) The system of claim 1, wherein the machine-learned model is a neural network. ### Claim 5 (depends on 1) The system of claim 1, wherein the machine-learned model is a deep learning model. ### Claim 6 (depends on 1) The system of claim 1, wherein the sample search results and candidate search results are member profiles in the online service. ### Claim 7 (depends on 6) The system of claim 6, wherein the actions taken by users on the search results are responses to communication requests from searchers. ### Claim 8 A computer-implemented method for returning search results in an online computer system, the method comprising: retrieving training data, the training data comprising a plurality of search results, a plurality of sample labels, and activity and usage information pertaining to actions taken by users on the search results on an online service, the training data comprising randomized and non-randomized training data, the randomized training data derived from historical presentation of a first subset of the plurality of search results using a randomized ranking and the non-randomized training data derived from historical presentation of a second subset of the plurality of search results using a non-randomized ranking; for each sample search result in a plurality of sample search results in the non-randomized training data: parsing the sample search result and the activity and usage information pertaining to the sample search result in the non-randomized training data to extract a first set of one or more features; feeding a sample member label corresponding to the sample search result and the extracted first set of one or more features into a first machine learning algorithm to train a machine-learned model to output a score indicative of a probability that an action will be taken, in the online service, pertaining to the sample search result; for each sample search result in a plurality of sample search results in the randomized training data: parsing the sample search result and the activity and usage information pertaining to the sample search result in the randomized training data to extract a second set of one or more features; and feeding a sample member label corresponding to the sample search result in the randomized training data and the extracted second set of one or more features into the first machine learning algorithm to revise the machine-learned model. ### Claim 9 (depends on 8) The method of claim 8, further comprising: obtaining a plurality of candidate search results in response to a query corresponding to a searcher; for each candidate search result from the plurality of candidate search results: obtaining activity and usage information for the candidate search result; parsing the candidate search result and the activity and usage information corresponding to the candidate search result to extract a third set of one or more features; feeding the extracted third set of one or more features into the machine-learned model, outputting a score for the candidate search result; ranking the plurality of candidate search results by their corresponding scores; and returning one or more of the plurality of candidate search results based on the ranking. ### Claim 10 (depends on 8) The method of claim 8, wherein the non-randomized training data is more plentiful than the randomized training data. ### Claim 11 (depends on 8) The method of claim 8, wherein the machine-learned model is a neural network. ### Claim 12 (depends on 8) The method of claim 8, wherein the machine-learned model is a deep learning model. ### Claim 13 (depends on 8) The method of claim 8, wherein the sample search results and candidate search results are member profiles in the online service. ### Claim 14 (depends on 13) The method of claim 13, wherein the actions taken by users on the search results are responses to communication requests from searchers. ### Claim 15 A non-transitory machine-readable storage medium comprising instructions, which when implemented by one or more machines, cause the one or more machines to perform operations comprising: retrieving training data, the training data comprising a plurality of search results, a plurality of sample labels, and activity and usage information pertaining to actions taken by users on the search results on an online service, the training data comprising randomized and non-randomized training data, the randomized training data derived from historical presentation of a first subset of the plurality of search results using a randomized ranking and the non-randomized training data derived from historical presentation of a second subset of the plurality of search results using a non-randomized ranking; for each sample search result in a plurality of sample search results in the non-randomized training data: parsing the sample search result and the activity and usage information pertaining to the sample search result in the non-randomized training data to extract a first set of one or more features; feeding a sample member label corresponding to the sample search result and the extracted first set of one or more features into a first machine learning algorithm to train a machine-learned model to output a score indicative of a probability that an action will be taken, in the online service, pertaining to the sample search result; for each sample search result in a plurality of sample search results in the randomized training data: parsing the sample search result and the activity and usage information pertaining to the sample search result in the randomized training data to extract a second set of one or more features; and feeding a sample member label corresponding to the sample search result in the randomized training data and the extracted second set of one or more features into the first machine learning algorithm to revise the machine-learned model. ### Claim 16 (depends on 15) The non-transitory machine-readable storage medium of claim 15, wherein the instructions further comprise: obtaining a plurality of candidate search results in response to a query corresponding to a searcher; for each candidate search result from the plurality of candidate search results: obtaining activity and usage information for the candidate search result; parsing the candidate search result and the activity and usage information corresponding to the candidate search result to extract a third set of one or more features; feeding the extracted third set of one or more features into the machine-learned model, outputting a score for the candidate search result; ranking the plurality of candidate search results by their corresponding scores; and returning one or more of the plurality of candidate search results based on the ranking. ### Claim 17 (depends on 15) The non-transitory machine-readable storage medium of claim 15, wherein the non-randomized training data is more plentiful than the randomized training data. ### Claim 18 (depends on 15) The non-transitory machine-readable storage medium of claim 15, wherein the machine-learned model is a neural network. ### Claim 19 (depends on 15) The non-transitory machine-readable storage medium of claim 15, wherein the machine-learned model is a deep learning model. ### Claim 20 (depends on 15) The non-transitory machine-readable storage medium of claim 15, wherein the sample search results and candidate search results are member profiles in the online service.