# US 2021/0334585 A1 · Methods and apparatus for machine learning engine optimization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334585 A1
- Document ID: US20210334585A1
- Status: Published application
- Kind: A1
- Application number: 16/860,930
- Filed: 2020-04-28
- Published: 2021-10-28
- Earliest claimed: 2020-04-28
- Assignees: The Nielsen Company (US), LLC
- Applicants: The Nielsen Company (US), LLC
- Inventors: Rachel Anne Szabo
- Web page: https://patentgrep.com/patent/US20210334585A1

## CPC

- G06N 20/20: Ensemble learning
- G06F 18/2113: by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/285: {Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system}
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods, apparatus, systems and articles of manufacture are disclosed for machine learning engine optimization. An example apparatus includes a selection metric analyzer to compute a first selection metric based on a first set of ordered output values from a first candidate machine learning engine and a set of reference data values; compute a second selection metric based on a second set of ordered output values from a second candidate machine learning engine and the set of reference data values; and a machine learning engine replacer to determine whether to replace an active machine learning engine with at least one of the first candidate machine learning engine or the second candidate machine learning engine based on the first selection metric and the second selection metric.

## Claims

### Claim 1

An apparatus comprising:
a selection metric analyzer to:
compute a first selection metric based on a first set of ordered output values from a first candidate machine learning engine and a set of reference data values;
compute a second selection metric based on a second set of ordered output values from a second candidate machine learning engine and the set of reference data values; and
a machine learning engine replacer to determine whether to replace an active machine learning engine with at least one of the first candidate machine learning engine or the second candidate machine learning engine based on the first selection metric and the second selection metric.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the first set of ordered output values are ranked by order of importance based on a configuration of the first candidate machine learning engine, and the second set of ordered output values are ranked by order of importance based on a configuration of the second candidate machine learning engine.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein the set of reference data values includes unordered reference data values that are utilized to determine consecutive highest rankings in the first set of ordered output values from the first machine learning engine and the second set of ordered output values from second first machine learning engine.

### Claim 4 (depends on 3)

The apparatus of claim 3, wherein the determining of the first selection metric further includes the selection metric analyzer to determine a first evaluation metric by:
determining a first difference between 1) a number of the unordered reference data values in the set of reference data values and 2) a number of the unordered reference data values included in beginning positions of the first set of ordered output values from the first candidate machine learning engine; and
dividing the first difference by the number of the unordered reference data values from the set of reference data values.

### Claim 5 (depends on 4)

The apparatus of claim 4, wherein the determining of the first selection metric further includes the selection metric analyzer to determine a second evaluation metric by:
determining a second difference between 1) a number corresponding to how many of the unordered reference data values from the set of reference data values are present in the first set of ordered output values and 2) the number of the unordered reference data values from the set of reference data values;
adding a penalty to the second difference for each of the unordered reference data values from the set of reference data values that is not present in the first set of ordered output values;
dividing the second difference by an addition of 1) the number of values in the first set of ordered output values and 2) the number of the unordered reference data values from the set of reference data values.

### Claim 6 (depends on 5)

The apparatus of claim 5, wherein the penalty has a value of one-fourth.

### Claim 7 (depends on 5)

The apparatus of claim 5, wherein the determining the first selection metric further includes the selection metric analyzer to:
add the first evaluation metric and the second evaluation metric to determine a result; and
divide the result by two to determine the first selection metric.

### Claim 8 (depends on 1)

The apparatus of claim 1, wherein determining whether to replace the active machine learning engine with at least one of the first candidate machine learning engine or the second candidate machine learning engine further includes the selection metric analyzer to:
compare the first selection metric to the second selection metric to identify the lowest selection metric;
compare the lowest selection metric to a threshold to determine if the at least one of the first candidate machine learning engine or the second candidate machine learning should replace the active machine learning engine; and
replace the active machine learning engine with the at least one of the first candidate machine learning engine or the second candidate machine learning when the lowest selection metric satisfies the threshold.

### Claim 9

A non-transitory computer readable storage medium comprising instructions which, when executed, cause one or more processors to at least:
compute a first selection metric based on a first set of ordered output values from a first candidate machine learning engine and a set of reference data values;
compute a second selection metric based on a second set of ordered output values from a second candidate machine learning engine and the set of reference data values; and
determine whether to replace an active machine learning engine with at least one of the first candidate machine learning engine or the second candidate machine learning engine based on the first selection metric and the second selection metric.

### Claim 10 (depends on 9)

The computer readable storage medium of claim 9, wherein the first set of ordered output values are ranked by order of importance based on a configuration of the first candidate machine learning engine, and the second set of ordered output values are ranked by order of importance based on a configuration of the second candidate machine learning engine.

### Claim 11 (depends on 10)

The computer readable storage medium of claim 10, wherein the set of reference data values includes unordered reference data values that are utilized to determine consecutive highest rankings in the first set of ordered output values from the first machine learning engine and the second set of ordered output values from second first machine learning engine.

### Claim 12 (depends on 11)

The computer readable storage medium of claim 11, wherein the instructions, when executed, cause the one or more processors to determine a first evaluation metric by:
determining a first difference between 1) a number of the unordered reference data values in the set of reference data values and 2) a number of the unordered reference data values included in beginning positions of the first set of ordered output values from the first candidate machine learning engine; and
dividing the first difference by the number of the unordered reference data values from the set of reference data values.

### Claim 13 (depends on 12)

The computer readable storage medium of claim 12, wherein the instructions, when executed, cause the one or more processors to determine a second evaluation metric by:
determining a second difference between 1) a number corresponding to how many of the unordered reference data values from the set of reference data values are present in the first set of ordered output values and 2) the number of the unordered reference data values from the set of reference data values;
adding a penalty to the second difference for each of the unordered reference data values from the set of reference data values that is not present in the first set of ordered output values;
dividing the second difference by an addition of 1) the number of values in the first set of ordered output values and 2) the number of the unordered reference data values from the set of reference data values.

### Claim 14 (depends on 13)

The computer readable storage medium of claim 13, wherein the instructions, when executed, cause the one or more processors to:
add the first evaluation metric and the second evaluation metric to determine a result; and
divide the result by two to determine the first selection metric.

### Claim 15

A method comprising:
computing, by executing an instruction with a processor, a first selection metric based on a first set of ordered output values from a first candidate machine learning engine and a set of reference data values;
computing, by executing an instruction with the processor, a second selection metric based on a second set of ordered output values from a second candidate machine learning engine and the set of reference data values; and
determining, by executing an instruction with the processor, whether to replace an active machine learning engine with at least one of the first candidate machine learning engine or the second candidate machine learning engine based on the first selection metric and the second selection metric.

### Claim 16 (depends on 15)

The method of claim 15, wherein the first set of ordered output values are ranked by order of importance based on a configuration of the first candidate machine learning engine, and the second set of ordered output values are ranked by order of importance based on a configuration of the second candidate machine learning engine.

### Claim 17 (depends on 16)

The method of claim 16, wherein the set of reference data values includes unordered reference data values that are utilized to determine consecutive highest rankings in the first set of ordered output values from the first machine learning engine and the second set of ordered output values from second first machine learning engine.

### Claim 18 (depends on 17)

The method of claim 17, wherein the determining of the first selection metric includes determining a first evaluation metric by:
determining a first difference between 1) a number of the unordered reference data values in the set of reference data values and 2) a number of the unordered reference data values included in beginning positions of the first set of ordered output values from the first candidate machine learning engine; and
dividing the first difference by the number of the unordered reference data values from the set of reference data values.

### Claim 19 (depends on 17)

The method of claim 17, wherein the determining of the first selection metric further includes determining a second evaluation metric by:
determining a second difference between 1) a number corresponding to how many of the unordered reference data values from the set of reference data values are present in the first set of ordered output values and 2) the number of the unordered reference data values from the set of reference data values;
adding a penalty to the second difference for each of the unordered reference data values from the set of reference data values that is not present in the first set of ordered output values;
dividing the second difference by an addition of 1) the number of values in the first set of ordered output values and 2) the number of the unordered reference data values from the set of reference data values.

### Claim 20 (depends on 19)

The method of claim 19, wherein the determining the first selection metric further includes:
adding the first evaluation metric and the second evaluation metric to determine a result; and
dividing the result by two to determine the first selection metric.
