# US 2022/0036138 A1 · System and Method for Ensemble Expert Diversification via Bidding and Control Thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036138 A1
- Document ID: US20220036138A1
- Status: Published application
- Kind: A1
- Application number: 16/944,503
- Filed: 2020-07-31
- Published: 2022-02-03
- Earliest claimed: 2020-07-31
- Assignees: Oath Inc.
- Applicants: Oath Inc.
- Inventors: Gal Lalouche; Ran Wolff
- Web page: https://patentgrep.com/patent/US20220036138A1

## CPC

- G06F 18/2185: {the supervisor being an automated module, e.g. intelligent oracle}
- G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling}
- G06N 20/20: Ensemble learning
- G06Q 30/0241: Advertisements
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

The present teaching relates to method, system, medium, and implementations for machine learning. A bid is received, from an expert during training, for a training sample with an amount within a level of available bidding currency associated with the expert. The training sample is used for training a model associated with the expert. It is determined whether the expert is among at least one winner selected based on bids from one or more experts. If the expert is among the at least one winner, the training sample is sent to the expert. The at least one winner is selected based on one or more criteria aiming at expert diversification.

## Claims

### Claim 1

A method implemented on at least one machine including at least one processor, memory, and communication platform capable of connecting to a network for machine learning, the method comprising:
receiving, from an expert during training, a bid for a training sample with an amount within a level of available bidding currency associated with the expert, wherein the training sample is used for training a model associated with the expert;
determining whether the expert is among at least one winner selected based on bids from one or more experts; and
sending the training sample to the expert if the expert is among the at least one winner, wherein
the at least one winner is selected based on one or more criteria aiming at expert diversification.

### Claim 2 (depends on 1)

The method of claim 1, further comprising initializing the level of available bidding currency associated with the expert.

### Claim 3 (depends on 1)

The method of claim 1, further comprising receiving, from the expert, a prediction generated in accordance with the training sample and based on one or more parameters associated with the model associated with the expert.

### Claim 4 (depends on 3)

The method of claim 3, further comprising receiving, from the expert, a metric characterizing the prediction, wherein the metric is used to determine whether a ground truth label for the training sample is to be provided to the expert to facilitate the training.

### Claim 5 (depends on 4)

The method of claim 4, wherein the metric includes a confidence score indicative of a level of confidence the expert has in the prediction.

### Claim 6 (depends on 1)

The method of claim 1, further comprising sending, to the expert, a ground truth label corresponding to the training sample to enable the expert to update, during the learning, the one or more parameters associated with the model based on the prediction and the ground truth label.

### Claim 7 (depends on 4)

The method of claim 4, further comprising updating the level of available bidding currency based on the metric and the amount.

### Claim 8

Machine readable and non-transitory medium having information recorded thereon for machine learning, wherein the information, once read by the machine, causes the machine to perform:
receiving, from an expert during training, a bid for a training sample with an amount within a level of available bidding currency associated with the expert, wherein the training sample is used for training a model associated with the expert;
determining whether the expert is among at least one winner selected based on bids from one or more experts; and
sending the training sample to the expert if the expert is among the at least one winner, wherein the at least one winner is selected based on one or more criteria aiming at expert diversification.

### Claim 9 (depends on 8)

The medium of claim 8, wherein the information, once read by the machine, further causes the machine to perform initializing the level of available bidding currency associated with the expert.

### Claim 10 (depends on 8)

The medium of claim 8, wherein the information, once read by the machine, further causes the machine to perform receiving, from the expert, a prediction generated in accordance with the training sample and based on one or more parameters associated with the model associated with the expert.

### Claim 11 (depends on 10)

The medium of claim 10, wherein the information, once read by the machine, further causes the machine to perform receiving, from the expert, a metric characterizing the prediction, wherein the metric is used to determine whether a ground truth label for the training sample is to be provided to the expert to facilitate the training.

### Claim 12 (depends on 11)

The medium of claim 11, wherein the metric includes a confidence score indicative of a level of confidence the expert has in the prediction.

### Claim 13 (depends on 8)

The medium of claim 8, wherein the information, once read by the machine, further causes the machine to perform sending, to the expert, a ground truth label corresponding to the training sample to enable the expert to update, during the learning, the one or more parameters associated with the model based on the prediction and the ground truth label.

### Claim 14 (depends on 11)

The medium of claim 11, wherein the information, once read by the machine, further causes the machine to perform updating the level of available bidding currency based on the metric and the amount.

### Claim 15

A system for machine learning, comprising:
a bidding winner selector configured for
receiving, from an expert during training, a bid for a training sample with an amount within a level of available bidding currency associated with the expert, wherein the training sample is used for training a model associated with the expert, and
determining whether the expert is among at least one winner selected based on bids from one or more experts; and
a training data distribution unit configured for sending the training sample to the expert if the expert is among the at least one winner, wherein
the at least one winner is selected based on one or more criteria aiming at expert diversification.

### Claim 16 (depends on 15)

The system of claim 15, further comprising an initialization unit configured for initializing the level of available bidding currency associated with the expert.

### Claim 17 (depends on 1)

The system of claim 1, further comprising a ground truth allocation unit configured for receiving, from the expert, a prediction generated in accordance with the training sample and based on one or more parameters associated with the model associated with the expert.

### Claim 18 (depends on 17)

The system of claim 17, wherein the ground truth allocation unit is further configured for receiving, from the expert, a metric characterizing the prediction, wherein the metric is used to determine whether a ground truth label for the training sample is to be provided to the expert to facilitate the training, wherein the metric includes a confidence score indicative of a level of confidence that the expert has in the prediction.

### Claim 19 (depends on 15)

The system of claim 15, wherein the ground truth allocation unit is further configured for sending, to the expert, a ground truth label corresponding to the training sample to enable the expert to update, during the learning, the one or more parameters associated with the model based on the prediction and the ground truth label.

### Claim 20 (depends on 18)

The system of claim 18, further comprising a currency allocation updater configured for updating the level of available bidding currency based on the metric and the amount.
