# US 8,280,891 B1 · System and method for the calibration of a scoring function

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 8,280,891 B1
- Document ID: US8280891B1
- Status: Granted
- Kind: B1
- Application number: 13/163,014
- Filed: 2011-06-17
- Granted: 2012-10-02
- Earliest claimed: 2011-06-17
- Assignees: Google Inc.
- Inventors: Shlomo Urbach; Yuval Netzer
- Web page: https://patentgrep.com/patent/US8280891B1

## CPC

- G06F 16/35: Clustering; Classification
- G06V 2201/10: Recognition assisted with metadata
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A system and method for calibrating a scoring function. The scoring function S(input, classification) provides a score based on the amount of evidence a particular input has in connection with a particular classification. For example, a street level image may be OCR'ed so as to indicate the names of establishments contained within the image, and the scoring function indicates how much evidence exists within the image for a particular establishment. Some establishments (i.e. classifications) may produce higher scores based on the nature of the establishment rather than the nature of the image (i.e. input) causing any ranking of establishments done on the basis of the scoring function to be biased. Accordingly, the scoring function is calibrated by determining the probability distribution of scores for an establishment over a false set of images that do not display the establishment. The scoring function is calibrated so as to adjust the score to overcome such bias.

## Claims

### Claim 1

A method of calibrating a scoring function, the method comprising:
accessing a scoring function, wherein the scoring function provides a score indicating a degree to which an input relates to a classification;
receiving a first classification;
accessing a set of background inputs that have no more than an incidental relation to the first classification;
applying the scoring function to each of the background inputs in connection with the first classification so as to create a first distribution of background scores;
receiving a first input that is not included in the set of background inputs;
altering the scoring function in accordance with the first distribution of background scores to create a first calibrated scoring function; and
determining a first calibrated score for the first input in connection with the first classification using the first calibrated scoring function.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first input and the set of background inputs are images, the classification is an establishment, and the score from the scoring function relates to the amount of evidence that exists as to whether the establishment appears within a particular image.

### Claim 3 (depends on 1)

The method of claim 1, wherein the first input, the set of background inputs, and classification are websites, and the score from the scoring function relates to the number of users that visited both a first input website and one or more classification websites.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first input and the set of background inputs include text, the classification is a phrase, and the score from the scoring function relates to number of times the phrase appears within a particular input.

### Claim 5 (depends on 4)

The method of claim 4, wherein the first input and the set of background inputs are selected from the group consisting of: documents, websites, or books.

### Claim 6 (depends on 1)

The method of claim 1, wherein determining the calibrated score includes creating a cumulative distribution function (CDF), the CDF indicating, for a subset of potential scores, the probability of the distribution of background scores being higher than each of the potential scores.

### Claim 7 (depends on 6)

The method of claim 6, wherein determining the calibrated score includes:
smoothing the CDF to create a smoothed CDF (SCDF); and
calculating −log2(SCDF).

### Claim 8 (depends on 7)

The method of claim 7, wherein the CDF for any potential score is represented as CDFi=CDFi+1+Hi/n, where n is the number of potential scores, Hi represents the number of background inputs receiving a score i, and CDFN=HN/n.

### Claim 9 (depends on 7)

The method of claim 7, wherein determining the calibrated score includes:
calculating Mi=min(K*Miall, U, −log2(SCDFi)), where Miall represents a default model for all background scores relating to a set of classifications, K represents an optional limiting factor, and U represents an upper limit on calibrated scores.

### Claim 10 (depends on 1)

The method of claim 1 further comprising:
accessing a second classification, wherein the scoring function provides a second score indicating a degree to which an input relates to the second classification;
applying the scoring function to each of the background inputs in connection with the second classification to create a second distribution of background scores;
altering the scoring function in accordance with the second distribution of background scores to create a second calibrated scoring function;
determining a second calibrated score for the first input in connection with the second classification using the second calibrated scoring function; and
comparing the first calibrated score and the second calibrated score.

### Claim 11 (depends on 10)

The method of claim 10, wherein the second calibrated score is for a second input in connection with the second classification.

### Claim 12 (depends on 10)

The method of claim 10, further comprising transmitting data over a network based on the comparison of the first calibrated score and the second calibrated score.

### Claim 13 (depends on 1)

The method of claim 1, further comprising:
accessing a set of default classifications;
applying the scoring function to each combination of background inputs and default classifications, to create a default background distribution; and
altering the first calibrated scoring function in accordance with the default background distribution.

### Claim 14 (depends on 13)

The method of claim 13 further comprising:
comparing the default background distribution to the first distribution of background scores;
wherein the altering of the first calibrated scoring function occurs only if the first distribution of background scores indicates a higher score than the default background distribution.

### Claim 15

A device for calibrating a scoring function:
a memory for storing classifications, inputs, and scoring functions, wherein at least a subset of the inputs are background inputs; and
a processor coupled to the memory, the processor configured to execute the instructions to:
access a first classification and a scoring function, wherein the scoring function provides a score indicating a degree to which an input relates to the first classification;
access background inputs that have no more than incidental relation to the first classification;
apply the scoring function to each of the background inputs in connection with the first classification so as to create a first distribution of background scores;
access a first input that is not included in the set of background inputs; and
determine a first calibrated score for the first input in connection with the first classification, the first calibrated score being determined by altering the scoring function in accordance with the distribution of background scores.

### Claim 16 (depends on 15)

The device of claim 15, wherein the first input and the set of background inputs are images, the first classification is an establishment, and the first score from the scoring function relates to whether the establishment appears within a particular image.

### Claim 17 (depends on 15)

The device of claim 15, wherein the first input, the set of background inputs, and first classification are websites, and the first score from the scoring function relates to the number of users that visited both a first input website and one or more classification websites.

### Claim 18 (depends on 15)

The device of claim 15, wherein the first input and the set of background inputs include text, the classification is a phrase, and the first score from the scoring function relates to number of times the phrase appears within a particular input.

### Claim 19 (depends on 15)

The device of claim 15 wherein the instructions further cause the processor to:
receive a second classification, wherein the scoring function provides a second score indicating a degree to which an input relates to the classification;
apply the scoring function to each of the background inputs in connection with the second classification so as to create a second distribution of background scores;
determine a second calibrated score for the first input in connection with the second classification, the second calibrated score being determined using the second distribution of background scores to alter the second score of the scoring function; and
compare the first calibrated score and second calibrated score.

### Claim 20 (depends on 15)

The device of claim 15 wherein the processor is further configured to:
transmit data relating to the classification having a higher calibrated score.
