# US 2021/0004379 A1 · Utilize high performing trained machine learning models for information retrieval in a web store

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0004379 A1
- Document ID: US20210004379A1
- Status: Published application
- Kind: A1
- Application number: 16/918,928
- Filed: 2020-07-01
- Published: 2021-01-07
- Earliest claimed: 2019-07-01
- Assignees: Kohl's, Inc.
- Applicants: Kohl's, Inc.
- Inventors: Alan Lee; Ru Wang; Ritaja Sur; Arjun Manimaran; Suraj Nayak Mithbail; Xiaoyu Jin; Jinge Zhang; Xiaobing Luo; Zhiyu Liang; Milan Kumar Behera; Thrinath Babu Kathula
- Web page: https://patentgrep.com/patent/US20210004379A1

## CPC

- G06F 16/24578: {using ranking}
- G06F 16/248: Presentation of query results
- G06F 16/958: Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 5/04: Inference or reasoning models
- G06N 20/00: Machine learning
- G06N 20/20: Ensemble learning
- G06Q 30/0282: Rating or review of business operators or products
- G06Q 30/0625: {by formulating product or service queries, e.g. using keywords or predefined options}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

The present technology is directed to high performing trained machine learning models for information retrieval in a web store. In some embodiments, for example, when a search query is received from a user of an online retailer, a computer system associated with the online retailer accesses measurements of performance of sets of search results returned in response to previous user search queries. Each of the previous search results set is a set that was ranked by a machine learning model selected from a store of machine learning models that are each trained to rank search results. Based on the measurements of performance, the computer system selects a machine learning model to rank search results for a response to the received search query. The ranked search results are provided for output to the user.

## Claims

### Claim 1

A method comprising:
receiving at a computer system associated with an online retailer, a search query from a user of the online retailer;
accessing by the computer system, measurements of performance of sets of search results returned in response to previous user search queries submitted at the online retailer, each set of search results ranked by a machine learning model selected from a store of machine learning models that are each trained to rank search results;
selecting by the computer system, based on the measurements of performance, a machine learning model to rank search results for a response to the received search query; and
providing the ranked search results for output to the user.

### Claim 2 (depends on 1)

The method of claim 1, further comprising measuring the performance of the sets of search results, wherein measuring the performance comprises, for each of the previous user search queries:
applying one of the machine learning models selected from the store to rank search results that match the previous user search query;
returning the ranked search results that match the previous user search query for output to a corresponding user; and
capturing a metric indicating performance of the output search results.

### Claim 3 (depends on 2)

The method of claim 2, wherein measuring the performance of the sets of search results comprises using each of the machine learning models in the store to rank search results for approximately equivalent numbers of received user queries.

### Claim 4 (depends on 2)

The method of claim 2, wherein measuring the performance of the sets of search results comprises measuring at least one of a click-through rate, a user rating of relevancy of the search results, or implicit user feedback.

### Claim 5 (depends on 1)

The method of claim 1, wherein selecting the machine learning model based on the measurements of performance comprises selecting a machine learning model from the store that has a highest probability of achieving a specified performance criterion.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
detecting, by the computer system, a triggering condition for a scaling cycle;
responsive to detecting the triggering condition, applying, by the computer system, a scaling mechanism to increase an exploration of additional machine learning models selected from the store;
receiving, at the computer system, a subsequent search query; and
selecting, by the computer system, a different machine learning model from the store based on the exploration to rank search results for a response to the subsequent search query.

### Claim 7 (depends on 6)

The method of claim 6, wherein applying the scaling mechanism comprises:
measuring, by the computer system, performance of the sets of search results ranked using a first machine learning model selected from the store;
initializing, by the computer system, a parameter of a first probability distribution to the measured performed of the recommendations generated using the first machine learning model; and
responsive to detecting the triggering condition, applying a scaling value to the parameter of the first probability distribution to generate a second probability distribution.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
receiving user feedback indicating a relevancy of the ranked search results; and
storing the received user feedback as a performance metric associated with the ranked search results.

### Claim 9 (depends on 1)

The method of claim 1, wherein the ranked search results include one or more products offered for sale by the online retailer that are determined to match the received search query.

### Claim 10 (depends on 1)

The method of claim 1, wherein selecting the machine learning model to rank the search results for the response to the received search query comprises selecting a first machine learning model to rank the search results for the received search query and selecting a second machine learning model that is different from the first machine learning model to rank the search results for another search query.

### Claim 11

A method comprising:
accessing by a computer system associated with an online retailer, a store including multiple machine learning models that are each trained to generate a ranking of search results in response to search queries received at the online retailer;
for each of a plurality of search queries input at the online retailer:
applying, by the computer system, one of the machine learning models selected from the store to generate a ranking of search results that match the received search query;
returning the ranked search results for output to a user who input the received search query; and
measuring a performance of the returned search results;
receiving a subsequent search query from a subsequent user of the online retailer;
responsive to receiving the subsequent search query, selecting, by the computer system, one of the machine learning models based on the measured performance of the returned search results; and
providing search results ranked using the selected machine learning model for output to the subsequent user.

### Claim 12 (depends on 11)

The method of claim 11, wherein measuring the performance of the returned search results comprises, for a respective search query in the plurality of search queries input at the online retailer:
applying one of the machine learning models selected from the store to rank search results that match the respective search query;
returning the ranked search results that match the respective search query for output to a corresponding user; and
capturing a metric indicating performance of the output search results.

### Claim 13 (depends on 12)

The method of claim 12, wherein measuring the performance of the returned search results comprises using each of the machine learning models in the store to rank search results for approximately equivalent numbers of the plurality of search queries input at the online retailer.

### Claim 14 (depends on 12)

The method of claim 12, wherein measuring the performance of the returned search results comprises measuring at least one of a click-through rate, a user rating of relevancy of the search results, or implicit user feedback.

### Claim 15 (depends on 11)

The method of claim 11, wherein selecting the machine learning model based on the measured performance of the returned search results comprises selecting a machine learning model from the store that has a highest probability of achieving a specified performance criterion.

### Claim 16 (depends on 11)

The method of claim 11, further comprising:
detecting, by the computer system, a triggering condition for a scaling cycle;
responsive to detecting the triggering condition, applying, by the computer system, a scaling mechanism to increase an exploration of additional machine learning models selected from the store;
receiving, at the computer system, a second subsequent search query; and
selecting, by the computer system, a different machine learning model from the store based on the exploration to rank search results for a response to the second subsequent search query.

### Claim 17 (depends on 16)

The method of claim 16, wherein applying the scaling mechanism comprises:
measuring, by the computer system, performance of the sets of search results ranked using a first machine learning model selected from the store;
initializing, by the computer system, a parameter of a first probability distribution to the measured performed of the recommendations generated using the first machine learning model; and
responsive to detecting the triggering condition, applying a scaling value to the parameter of the first probability distribution to generate a second probability distribution.

### Claim 18 (depends on 11)

The method of claim 11, further comprising:
receiving user feedback indicating a relevancy of the search results ranked using the selected machine learning model; and
storing the received user feedback as a performance metric associated with the search results ranked using the selected machine learning model.

### Claim 19 (depends on 11)

The method of claim 11, wherein the ranked search results include one or more products offered for sale by the online retailer that are determined to match the received search query.

### Claim 20

A system comprising:
a trained machine learning model storage comprising a non-transitory computer readable storage medium, the trained machine learning model storage storing multiple machine learning models that are each trained to generate a ranking of search results in response to search queries received at an online retailer;
a performance monitoring system including a processor configured to, for each of a plurality of search queries input at the online retailer:
apply one of the machine learning models selected from the store to generate a ranking of search results that match the received search query;
return the ranked search results for output to the user; and
measure a performance of the returned search results;
a decision engine including a processor configured to select, for a given search query received at the online retailer, one of the machine learning models based on measured performance of previously returned search results; and
a search platform including a processor configured to:
process the search queries input at the online retailer to identify sets of products that match each of the search queries;
apply for each of the processed search queries, a machine learning model selected by the decision engine to rank the identified set of products that match the processed search query; and
provide the ranked set of products for output.
