# US 2014/0207716 A1 · Natural language processing method and system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2014/0207716 A1
- Document ID: US20140207716A1
- Status: Published application
- Kind: A1
- Application number: 14/159,975
- Filed: 2014-01-21
- Published: 2014-07-24
- Earliest claimed: 2013-01-22
- Assignees: Maluuba Inc.
- Applicants: Maluuba Inc.
- Inventors: Wilson Hsu; Joshua Pantony; Kaheer Suleman
- Web page: https://patentgrep.com/patent/US20140207716A1

## CPC

- G06F 16/35: Clustering; Classification
- G06F 16/353: into predefined classes
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method, system and non-transitory computer-readable medium are provided for improving a statistical classification system, such as a statistical classification system that accepts natural language voice queries as inputs. A clustering engine may create one or more clusters of queries where the queries in each cluster are related in some way. A reviewing module may be employed to determine whether each cluster relates to an existing category supported by the classification system, a new category that can be supported by the classification system by training statistical models with the data from the cluster, is ambiguous, or is not useful to improve the classification system. For clusters determined to be useful for improving the system, the data in the clusters may be added to an existing training set or used as a training set to train new statistical models.

## Claims

### Claim 1

A computer-implemented method for improving a statistical classification system comprising one or more statistical classifiers, the one or more statistical classifiers configured to classify an input query into one category of a set of one or more categories, the method comprising:
storing an input query dataset comprising a plurality of input queries;
performing one or more iterations of clustering operations on the input query dataset to create clusters of input queries related by category, wherein each of the one or more input queries is assigned to one of the clusters;
for a respective one of the clusters, training a statistical classifier to classify the one or more input queries into the respective related category; and
providing the statistical classifier for implementing in the statistical classification system.

### Claim 2 (depends on 1)

The method of claim 1 wherein the clustering operations utilize one or more of K-means, Lloyd's algorithm, other distance measures, and Naïve Bayes clustering techniques.

### Claim 3 (depends on 1)

The method of claim 1 comprising automatically filtering the clusters using a probability threshold to at least one of: eliminate a particular cluster and eliminate a particular input query from a particular cluster.

### Claim 4 (depends on 1)

The method of claim 1 wherein the training comprises one of retraining one of the statistical classifiers from the statistical classification system; and training a new statistical classifier for a new category for the statistical classification system.

### Claim 5 (depends on 4)

The method of claim 4 comprising providing a user interface for manually identifying a respective cluster as one of: useful for adding to an existing training set for retraining one of the statistical classifiers from the statistical classification system; useful for training the new statistical classifier for the new category for the statistical classification system; a candidate for manual curating; and not currently useful for improving the statistical classification system.

### Claim 6 (depends on 5)

The method of claim 5 comprising providing a user interface for initiating training in accordance with said identifying.

### Claim 7 (depends on 1)

The method of claim 1 wherein the statistical classification system comprises a natural language processing system and the input queries comprise audio queries or text-based queries derived from audio queries.

### Claim 8 (depends on 7)

The method of claim 7 wherein the audio queries are voice commands.

### Claim 9 (depends on 1)

The method of claim 1 wherein the input query dataset comprises input queries related to one or more categories which are additional to the categories in the set of one or more categories.

### Claim 10

A computer system for improving a statistical classification system comprising one or more statistical classifiers, the one or more statistical classifiers configured to classify an input query into one category of a set of one or more categories, the system comprising one or more processors coupled to memory storing instructions and data for configuring the computer system to:
store an input query dataset comprising a plurality of input queries;
perform one or more iterations of clustering operations on the input query dataset to create clusters of input queries related by category, wherein each of the one or more input queries is assigned to one of the clusters;
for a respective one of the clusters, train a statistical classifier to classify the one or more input queries into the respective related category; and
provide the statistical classifier for implementing in the statistical classification system.

### Claim 11 (depends on 10)

The computer system of claim 10 wherein the clustering operations utilize one or more of K-means, Lloyd's algorithm, other distance measures, and Naïve Bayes clustering techniques.

### Claim 12 (depends on 10)

The computer system of claim 10 configured to automatically filter the clusters using a probability threshold to at least one of: eliminate a particular cluster and eliminate a particular input query from a particular cluster.

### Claim 13 (depends on 10)

The computer system of claim 10 wherein the training of a statistical classifier comprises one of: retraining one of the statistical classifiers from the statistical classification system; and training a new statistical classifier for a new category for the statistical classification system.

### Claim 14 (depends on 13)

The computer system of claim 13 configured to provide a user interface for manually identifying a respective cluster as one of: useful for adding to an existing training set for retraining one of the statistical classifiers from the statistical classification system; useful for training the new statistical classifier for the new category for the statistical classification system; a candidate for manual curating; and not currently useful for improving the statistical classification system.

### Claim 15 (depends on 14)

The computer system of claim 14 configured to provide a user interface for initiating training in accordance with said identifying.

### Claim 16 (depends on 1)

The computer system of claim 1 wherein the statistical classification system comprises a natural language processing system and the input queries comprise audio queries or text-based queries derived from audio queries.

### Claim 17 (depends on 16)

The computer system of claim 16 wherein the audio queries are voice commands.

### Claim 18 (depends on 10)

The computer system of claim 10 wherein the input query dataset comprises input queries related to one or more categories which are additional to the categories in the set of one or more categories.

### Claim 19

A non-transitory computer-readable medium for improving a statistical classification system comprising one or more statistical classifiers, the one or more statistical classifiers configured to classify an input query into one category of a set of one or more categories, the non-transitory computer-readable medium comprising instructions that, when executed, cause a computer to perform operations comprising:
storing an input query dataset comprising a plurality of input queries;
performing one or more iterations of clustering operations on the input query dataset to create clusters of input queries related by category, wherein each of the one or more input queries is assigned to one of the clusters;
for a respective one of the clusters, training a statistical classifier to classify the one or more input queries into the respective related category; and
providing the statistical classifier for implementing in the statistical classification system.

### Claim 20 (depends on 19)

The computer-readable medium of claim 19 wherein the clustering operations utilize one or more of K-means, Lloyd's algorithm, other distance measures, and Naïve Bayes clustering techniques.

### Claim 21 (depends on 19)

The computer-readable medium of claim 19 wherein the operations further comprise automatically filtering the clusters using a probability threshold to at least one of: eliminate a particular cluster and eliminate a particular input query from a particular cluster.

### Claim 22 (depends on 19)

The computer-readable medium of claim 19 wherein training a statistical classifier comprises one of retraining one of the statistical classifiers from the statistical classification system; and training a new statistical classifier for a new category for the statistical classification system.

### Claim 23 (depends on 22)

The computer-readable medium of claim 22 wherein the operations further comprise providing a user interface for manually identifying a respective cluster as one of: useful for adding to an existing training set for retraining one of the statistical classifiers from the statistical classification system; useful for training the new statistical classifier for the new category for the statistical classification system; a candidate for manual curating; and not currently useful for improving the statistical classification system.

### Claim 24 (depends on 23)

The computer-readable medium of claim 23 wherein the operations further comprise providing a user interface for initiating training in accordance with said identifying.

### Claim 25 (depends on 1)

The computer-readable medium of claim 1 wherein the statistical classification system comprises a natural language processing system and the input queries comprise audio queries or text-based queries derived from audio queries.

### Claim 26 (depends on 25)

The computer-readable medium of claim 25 wherein the audio queries are voice commands.

### Claim 27 (depends on 1)

The computer-readable medium of claim 1 wherein the input query dataset comprises input queries related to one or more categories which are additional to the categories in the set of one or more categories.
