# US 2017/0286869 A1 · Self-Service Classification System

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0286869 A1
- Document ID: US20170286869A1
- Status: Published application
- Kind: A1
- Application number: 15/478,772
- Filed: 2017-04-04
- Published: 2017-10-05
- Earliest claimed: 2016-04-05
- Assignees: Refinitiv US Organization LLC
- Inventors: Hila Zarosim; Oren Hazai; Ofri Rom; Ehud Azikri; Lior Weintraub; Yael Lindman; Enav Weinreb; Savva Khalaman; Yossi Ben-Shlomo; Dmitry Levinson; Evyatar Sharabi; Alexandra Rabinovich Goldshlager; Shai Hertz
- Web page: https://patentgrep.com/patent/US20170286869A1

## CPC

- G06N 20/00: Machine learning
- G06F 16/35: Clustering; Classification
- G06F 30/20: Design optimisation, verification or simulation (optimisation, verification or simulation of circuit designs G06F30/30)
- G06F 40/30: Semantic analysis
- G06N 5/025: Extracting rules from data
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems, technologies and techniques for generating a customized classification model are disclosed. The system and technologies, such as THOMSON REUTERS SELF-SERVICE CLASSIFICATION™, employ part machine learning and part an user interactive approach to generate a customized classification model. The system combines a novel approach for text classification using a smaller initial set of data to initiate training, with a unique workflow and user interaction for customization.

## Claims

### Claim 1

A computer-implemented method comprising:
receiving a set of documents comprising a set of positive labeled documents and a set of unlabeled documents;
processing the set of documents to remove duplicate documents and noise within the documents;
generating a preliminary model for each topic of a set of topics of the set of documents;
training the preliminary model to determine an additional set of positive and negative documents based on the set of documents, wherein training comprises in part identifying negative documents from the set of documents; and
generating a customized classification model using the trained preliminary model and the additional set of positive and negative documents.

### Claim 2 (depends on 1)

The method of claim 1 wherein training the preliminary model further comprises applying a nearest neighbor linking algorithm.

### Claim 3 (depends on 1)

The method of claim 1 further comprising:
testing the customized classification model on a predetermined test set of documents and presenting to the user the precision and recall values associated with the testing.

### Claim 4 (depends on 1)

The method of claim 1 further comprising:
providing the customized classification model along with at least a precision and recall value.

### Claim 5 (depends on 1)

The method of claim 1 further comprising:
tuning the preliminary model and the customized classification model using one of triage rules, features selections, user feedback or combinations thereof;
generating a tuned preliminary model and a tuned customized classification model; and
providing the tuned customized classification model along with at least an updated precision or recall value.

### Claim 6 (depends on 5)

The method of claim 5 wherein tuning using triage rules comprises:
providing a list of phrases to a user interface; and
applying a selected list of phrases to the set of documents in response to a selection of inclusion or exclusion, wherein a document containing one or more phrases from the list of phrases are indicative of the selected topic.

### Claim 7 (depends on 5)

The method of claim 5 wherein tuning using features selection comprises providing a features list to a user interface, the features list further comprising a list of phrases which characterize documents associated with topic, applying a selected list of features to the set of documents in response to a selection of inclusion or exclusion.

### Claim 8 (depends on 5)

The method of claim 5 wherein tuning using user feedback comprises providing a selectable dialog box on a user interface to indicate confirmation of a document relating to a topic.

### Claim 9 (depends on 7)

The method of claim 7 further comprising weighting the features list based on a ranking.

### Claim 10 (depends on 5)

The method of claim 5 further comprising publishing the tuned customized classification model.

### Claim 11 (depends on 1)

The method of claim 1 further comprising:
validating the set of topics using the set of positive labeled documents and the set of negative documents.

### Claim 12 (depends on 11)

The method of claim 11 further comprising determining by a user interface that a topic has sufficient documents for validating.

### Claim 13

A system comprising:
a classification server including a processor and memory storing instructions that, in response to receiving a set of documents comprising a set of positive labeled documents and a set of unlabeled documents, cause the processor to:
process the set of documents to remove duplicate documents and noise within the documents;
generate a preliminary model for each topic of a set of topics of the set of documents;
train the preliminary model to determine an additional set of positive and negative documents based on the set of documents, wherein training comprises in part identifying negative documents from the set of document; and
generate a customized classification model using the preliminary model and the additional set of positive and negative documents.

### Claim 14 (depends on 13)

The system of claim 13, wherein the preliminary model applies a nearest neighbor linking algorithm.

### Claim 15 (depends on 13)

The system of claim 13, wherein the memory storing instructions that, in response to receiving the set of documents comprising the set of positive labeled documents and the set of unlabeled documents, cause the processor to:
test the customized classification model on a predetermined test set of documents and presenting to the user the precision and recall values associated with the testing.

### Claim 16 (depends on 13)

The system of claim 13, wherein the memory storing instructions that, in response to receiving the set of documents comprising the set of positive labeled documents and the set of unlabeled documents, cause the processor to:
provide the customized classification model along with at least a precision or recall value.

### Claim 17 (depends on 13)

The system of claim 13, wherein the memory storing instructions that, in response to receiving the set of documents comprising the set of positive labeled documents and the set of unlabeled documents, cause the processor to:
tune the preliminary model and the customized classification model using one of triage rules, features selections, user feedback or combinations thereof;
generate a tuned preliminary model and a tuned customized classification model; and
provide the tuned customized classification model along with at least the precision and recall value.

### Claim 18 (depends on 17)

The system of claim 17, wherein triage rules provides a list of phrases to a user interface and applies a selected list of phrases to the set of documents in response to a selection of inclusion or exclusion, wherein a document containing one or more phrases from the list of phrases are indicative of the selected topic.

### Claim 19 (depends on 17)

The system of claim 17, wherein features selections provides a features list to a user interface, the features list further comprises a list of phrases which characterize documents associated with topic, and applies a selected list of features to the set of documents in response to a selection of inclusion or exclusion.

### Claim 20 (depends on 17)

The system of claim 17 wherein user feedback provides a selectable dialog box on a user interface to indicate confirmation of a document relating to a topic.

### Claim 21 (depends on 19)

The system of claim 19 wherein the features list is weighted based on a ranking.

### Claim 22 (depends on 17)

The system of claim 17 wherein the memory storing instructions that, in response to receiving the set of documents comprising the set of positive labeled documents and the set of unlabeled documents, cause the processor to:
publish the tuned customized classification model.

### Claim 23 (depends on 13)

The system of claim 13 wherein the memory storing instructions that, in response to receiving the set of documents comprising the set of positive labeled documents and the set of unlabeled documents, cause the processor to:
validate the set of topics using the set of positive labeled documents and the set of negative documents.

### Claim 24 (depends on 23)

The system of claim 23 wherein the memory storing instructions that, in response to receiving the set of documents comprising the set of positive labeled documents and the set of unlabeled documents, cause the processor to:
determine by a user interface that a topic has sufficient documents for validating.
