# US 9,519,870 B2 · Weighting dictionary entities for language understanding models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,519,870 B2
- Document ID: US9519870B2
- Status: Granted
- Kind: B2
- Application number: 14/207,986
- Filed: 2014-03-13
- Published: 2015-09-17
- Granted: 2016-12-13
- Earliest claimed: 2014-03-13
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Corporation
- Inventors: Ruhi Sarikaya; Xiaohu Liu
- Web page: https://patentgrep.com/patent/US9519870B2

## CPC

- G06F 16/313: {Selection or weighting of terms for indexing}
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A dictionary used by a spoken language understanding (SLU) system is improved by providing weightings for entities in the dictionary that represent the likelihood each entity belongs to an entity class represented by the dictionary. A classifier model may be trained using a seed list containing sample entities that belong in the entity class and a background entity list containing samples that do not belong in the entity class. Clicked URLs from search logs, search result URLs, and attributes from an entity graph may be used as features of the sample entities to train the classifier model. The classifier model may be used to weight entities from a candidate dictionary. The entity weightings are used to generate an improved dictionary for use in the SLU system.

## Claims

### Claim 1

One or more computer storage media storing computer-executable instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising:
accessing a seed list containing positive sample entities that belong to an entity class;
accessing a background entity list containing negative sample entities that do not belong to the entity class;
identifying clicked URLs from search click logs for at least a portion of the positive sample entities and negative sample entities;
identifying search result URLs for at least a portion of the positive sample entities and negative sample entities;
identifying attributes from an entity graph for at least a portion of the positive sample entities and negative sample entities;
training a classifier model using the clicked URLs, search result URLs, and attributes from the entity graph as features of the positive sample entities and negative sample entities; and
using the classifier model to weight entities in a candidate dictionary to provide weightings for the entities from the candidate dictionary.

### Claim 2 (depends on 1)

The one or more computer storage media of claim 1, wherein the seed list is generated based on information from at least one selected from the following: an existing entity graph and training data from a spoken language understanding system.

### Claim 3 (depends on 1)

The one or more computer storage media of claim 1, wherein the background entity list is generated based on information from at least one selected from the following: an existing entity graph and training data from a spoken language understanding system.

### Claim 4 (depends on 1)

The one or more computer storage media of claim 1, wherein URL search results are obtained for positive sample entities and negative sample entities for which no clicked URLs are available from the search click logs.

### Claim 5 (depends on 1)

The one or more computer storage media of claim 1, wherein the classifier model is trained using at least one selected from the following: logistic regression and support vector machines.

### Claim 6 (depends on 1)

The one or more computer storage media of claim 1, wherein the operations further comprise:
using the weightings to generate an improved dictionary that does not include a subset of entities that do not satisfy a weighting threshold; and
using the improved dictionary in a spoken language understanding system to process a user input.

### Claim 7 (depends on 1)

The one or more computer storage media of claim 1, wherein the operations further comprise:
creating a weighted dictionary using the weightings; and
employing the weighted dictionary in a spoken language understanding system to process a user input.

### Claim 8 (depends on 1)

The one or more computer storage media of claim 1, wherein the operations further comprise:
clustering entities based on the weightings; and
generating a set of clustered dictionaries, each clustered dictionary including a cluster of entities.

### Claim 9 (depends on 8)

The one or more computer storage media of claim 8, wherein the operations further comprise:
employing the set of clustered dictionaries in a spoken language understanding system to process a user input.

### Claim 10

A computer-implemented method comprising:
accessing positive sample entities that belong to an entity class and negative sample entities that do not belong to the entity class;
identifying clicked URLs from search click logs for at least a portion of the positive sample entities and negative sample entities;
identifying search result URLs for at least a portion of the positive sample entities and negative sample entities;
identifying attributes from an entity graph for at least a portion of the positive sample entities and negative sample entities;
training, using a computing device, a classifier model using the clicked URLs, search result URLs, and attributes from the entity graph as features of the positive sample entities and negative sample entities; and
employing the classifier model to weight entities in a candidate dictionary to provide weightings for the entities from the candidate dictionary.

### Claim 11 (depends on 10)

The method of claim 10, wherein the positive sample entities and the negative sample entities are identified based on information from at least one selected from the following: an existing entity graph and training data from a spoken language understanding system.

### Claim 12 (depends on 10)

The method of claim 10, wherein URL search results are obtained for positive sample entities and negative sample entities for which no clicked URLs are available from the search click logs.

### Claim 13 (depends on 10)

The method of claim 10, wherein the method further comprises:
employing the weightings to generate at least one improved dictionary for use in a spoken language understanding system to process a user input.

### Claim 14 (depends on 10)

The method of claim 10, wherein the method further comprises:
clustering entities based on weighting; and
generating a set of clustered dictionaries, each clustered dictionary including a cluster of entities.

### Claim 15 (depends on 14)

The method of claim 14, wherein the method further comprises:
employing the set of clustered dictionaries in a spoken language understanding system to process a user input.

### Claim 16

A computerized system comprising:
one or more processors; and
a plurality of components that include computer-executable instructions that are executed by the one or more processors, the components including:
a model building component that trains a classifier model for an entity class using positive sample entities that belong to the entity class and negative sample entities that do not belong to the entity class, the model building component also using clicked URLs, search result URLs, and attributes from an entity graph as features of the positive sample entities and negative sample entities to train the classifier model; and
a weighting component that employs the classifier model to weight entities in a candidate dictionary to provide weightings for the entities from the candidate dictionary.

### Claim 17 (depends on 16)

The system of claim 16, wherein the positive sample entities are from a seed list and the negative sample entities are from a background entity list, and wherein the seed list and background entity list are generated based on information from at least one selected from the following: an existing entity graph and training data from a spoken language understanding system.

### Claim 18 (depends on 16)

The system of claim 16, wherein the weightings for the entities from the candidate dictionary are used to provide at least one improved dictionary.

### Claim 19 (depends on 16)

The computerized system of claim 16, wherein the components further comprise:
a clustering component that clusters entities based on the weightings and generates a set of clustered dictionaries, each clustered dictionary including a cluster of entities.