# US 2021/0232976 A1 · Composite machine learning system for label prediction and training data collection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0232976 A1
- Document ID: US20210232976A1
- Status: Published application
- Kind: A1
- Application number: 17/232,455
- Filed: 2021-04-16
- Published: 2021-07-29
- Earliest claimed: 2017-03-31
- Assignees: Intuit Inc.
- Applicants: Intuit Inc.
- Inventors: Yu-Chung HSIAO; Lei PEI; Meng CHEN; Nhung HO
- Web page: https://patentgrep.com/patent/US20210232976A1

## CPC

- G06N 20/00: Machine learning
- G06N 5/04: Inference or reasoning models
- G06N 20/20: Ensemble learning
- G06Q 30/04: Billing or invoicing
- G06Q 40/123: {Tax preparation or submission}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

The present disclosure provides a composite machine learning system for a transaction labeling service. A transaction labeling service receives at least one descriptive string describing a transaction associated with a user. The service identifies a preliminary grouping from a generalized scheme. The service extracts a set of N-grams from the descriptive string and converts the N-grams and the preliminary grouping into a set of features. A machine learning model determines a label from a labeling scheme for the transaction based on the features. User input related to the label includes an accuracy indicator and a reliability indicator. If the reliability indicator satisfies a reliability condition, a set of training data for the machine learning model is updated based on the descriptive string and the label. The machine learning model is then trained against the updated set of training data.

## Claims

### Claim 1

A method, comprising:
receiving an electronic request to label a transaction comprising a transaction amount and a descriptive string associated with the transaction;
generating a set of features associated with the descriptive string;
determining, via a machine learning model, a label for the transaction based on the set of features;
providing the label in response to the electronic request;
receiving user input comprising a reliability indicator related to the label;
adding a training instance representing the transaction to a set of training data for the machine learning model based on the reliability indicator satisfying a reliability condition; and
updating an aggregate value associated with the label based on the transaction amount.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
identifying a preliminary grouping that is bound to at least one substring of the descriptive string in an associative array; and
determining a feature in the set of features based on the preliminary grouping.

### Claim 3 (depends on 2)

The method of claim 2, wherein generating the set of features for the transaction comprises:
applying a hashing vectorizer to a set of N-grams extracted from the descriptive string; and
applying a one-hot encoder to the preliminary grouping.

### Claim 4 (depends on 3)

The method of claim 3, wherein each N-gram in the set of N-grams has a length less than or equal to three tokens.

### Claim 5 (depends on 1)

The method of claim 1, wherein adding a training instance representing the transaction to a set of training data for the machine learning model based on the reliability indicator satisfying the reliability condition comprises evaluating an amount of time a user took to review information related to the transaction.

### Claim 6 (depends on 1)

The method of claim 1, wherein adding the training instance representing the transaction to the set of training data for the machine learning model based on the reliability indicator satisfying the reliability condition comprises determining that a user providing the user input reviewed the label on a non-mobile device.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
identifying a geographical region or industry associated with the user input; and
selecting the machine learning model from a set of available machine learning models based on the geographical region or industry.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
receiving user input indicating a corrected label for the transaction;
storing the descriptive string in a transaction record labeled with the corrected label in a relabeling history associated with a user that provided the user input indicating the corrected label;
identifying a set of similar transaction records in the relabeling history, wherein each respective record in the set of similar transaction records includes a respective descriptive string that meets a threshold similarity level relative to the descriptive string;
determining the set of similar transaction records has a cardinality that meets a threshold number; and
generating an override rule for the user, wherein the override rule specifies that the corrected label be assigned to transactions that meet the threshold similarity level relative to the descriptive string.

### Claim 9 (depends on 8)

The method of claim 8, wherein receiving user input indicating a corrected label for the transaction comprises:
displaying a plurality of categories; and
receiving a user selection of a category of the plurality of categories, wherein the corrected label is associated with the category.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
displaying the label and information describing the transaction to a user in a graphical user interface; and
receiving user input via the graphical user interface.

### Claim 11

A system, comprising:
one or more processors; and
memory storing one or more instructions that, when executed on the one or more processors, perform a method comprising:
receiving an electronic request to label a transaction comprising a transaction amount and a descriptive string associated with the transaction;
generating a set of features associated with the descriptive string;
determining, via a machine learning model, a label for the transaction based on the set of features;
providing the label in response to the electronic request;
receiving user input comprising a reliability indicator related to the label;
adding a training instance representing the transaction to a set of training data for the machine learning model based on the reliability indicator satisfying a reliability condition; and
updating an aggregate value associated with the label based on the transaction amount.

### Claim 12 (depends on 11)

The system of claim 11, wherein the method further comprises:
identifying a preliminary grouping that is bound to at least one substring of the descriptive string in an associative array; and
determining a feature in the set of features based on the preliminary grouping.

### Claim 13 (depends on 12)

The system of claim 12, wherein generating the set of features for the transaction comprises:
applying a hashing vectorizer to a set of N-grams extracted from the descriptive string; and
applying a one-hot encoder to the preliminary grouping.

### Claim 14 (depends on 13)

The system of claim 13, wherein each N-gram in the set of N-grams has a length less than or equal to three tokens.

### Claim 15 (depends on 11)

The system of claim 11, wherein adding a training instance representing the transaction to a set of training data for the machine learning model based on the reliability indicator satisfying the reliability condition comprises evaluating an amount of time a user took to review information related to the transaction.

### Claim 16 (depends on 11)

The system of claim 11, wherein adding the training instance representing the transaction to the set of training data for the machine learning model based on the reliability indicator satisfying the reliability condition comprises determining that a user providing the user input reviewed the label on a non-mobile device.

### Claim 17 (depends on 11)

The system of claim 11, wherein the method further comprises:
identifying a geographical region or industry associated with the user input; and
selecting the machine learning model from a set of available machine learning models based on the geographical region or industry.

### Claim 18 (depends on 11)

The system of claim 11, wherein the method further comprises:
receiving user input indicating a corrected label for the transaction;
storing the descriptive string in a transaction record labeled with the corrected label in a relabeling history associated with a user that provided the user input indicating the corrected label;
identifying a set of similar transaction records in the relabeling history, wherein each respective record in the set of similar transaction records includes a respective descriptive string that meets a threshold similarity level relative to the descriptive string;
determining the set of similar transaction records has a cardinality that meets a threshold number; and
generating an override rule for the user, wherein the override rule specifies that the corrected label be assigned to transactions that meet the threshold similarity level relative to the descriptive string.

### Claim 19 (depends on 18)

The system of claim 18, wherein receiving user input indicating a corrected label for the transaction comprises:
displaying a plurality of categories; and
receiving a user selection of a category of the plurality of categories, wherein the corrected label is associated with the category.

### Claim 20

A method, comprising:
receiving an electronic request to label a transaction comprising a transaction amount and a descriptive string associated with the transaction;
identifying a preliminary grouping that is bound to at least one substring of the descriptive string in an associative array;
determining an accuracy condition is met based on the preliminary grouping;
comparing the preliminary grouping to a mapping based on the accuracy condition being met;
using the mapping to determine a label for the transaction;
providing the label in response to the electronic request;
receiving user input comprising a reliability indicator related to the label;
adding a training instance representing the transaction to a set of training data for a machine learning model based on the reliability indicator satisfying a reliability condition; and
updating an aggregate value associated with the label based on the transaction amount.
