# US 12,118,439 B2 · Mixed-initiative machine learning systems and methods for determining segmentations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,439 B2
- Document ID: US12118439B2
- Status: Granted
- Kind: B2
- Application number: 17/326,688
- Filed: 2021-05-21
- Published: 2021-09-09
- Granted: 2024-10-15
- Earliest claimed: 2017-03-17
- Assignees: Visa International Service Association
- Applicants: Visa International Service Association
- Inventors: Liang Gou; Hao Yang
- Web page: https://patentgrep.com/patent/US12118439B2

## CPC

- G06N 20/00: Machine learning
- G06N 5/04: Inference or reasoning models
- G06Q 30/0204: Market segmentation
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

A computer system can perform a semi-supervised machine learning processes to cluster a plurality of entities within a population based on their features and associated labels. The computer system can generate visualization data representing the clusters of entities and associated labels for displaying on a user interface. A user can review the clustering of entities and use the user interface to add or modify the labels associated with a particular entity or set of entities. The computer system can use the user's feedback to update the labels and then re-determine the clustering of entities using the semi-supervised machine learning process with the updated labels as input. As such, the computer system can use the user's feedback to improve the accuracy of the machine learning model without requiring a larger amount of labeled input data.

## Claims

### Claim 1

A method comprising:
obtaining, by a computer, input data associated with a plurality of individuals;
obtaining, by the computer, label data associated with a subset of the plurality of individuals, the label data assigning one or more labels to each individual of the subset;
performing, by the computer, a semi-supervised machine learning process using the input data and the label data;
generating, by the computer, visualization data that is based on the semi-supervised machine learning process and is displayed to a user, the visualization data representing one or more segmentations of the plurality of individuals;
receiving, by the computer, user feedback from the user corresponding to the visualization data; and
performing, by the computer, the semi-supervised machine learning process, using the user feedback, the input data, and the label data, and wherein the method further comprises:
determining, by the computer, sets of initial features based on the input data, the sets of initial features including a set of initial features for each individual of the plurality of individuals,
wherein the semi-supervised machine learning process includes determining sets of updated features based on the sets of initial features and the label data, the sets of updated features including a set updated features for each individual of the plurality of individuals,
wherein the generating of the visualization data is based on the sets of updated features and the label data, and
wherein the sets of updated features are different than the sets of initial features, the sets of updated features more consistently characterizing individuals to specific labels than the sets of initial features.

### Claim 2 (depends on 1)

The method of claim 1, further comprising segmenting, by the computer, the plurality of individuals into one or more segments of individuals based on the label data, each individual of each segment of individuals being associated with a particular label in the label data.

### Claim 3 (depends on 2)

The method of claim 2, wherein the determining of the sets of initial features is based on a deep embedding model or an auto-encoder process.

### Claim 4 (depends on 2)

The method of claim 2, further comprising performing, by the computer, one or more of a t-distributed stochastic neighbor embedding clustering process based on the sets of updated features, a bi-clustering process based on the sets of updated features, and a statistical summarization process based on the sets of updated features, and a profiling process on meta-data associated with the input data, wherein the generating of the visualization data is based on one or more of the t-distributed stochastic neighbor embedding clustering process, the statistical summarization process, and the profiling process.

### Claim 5 (depends on 2)

The method of claim 2, wherein the semi-supervised machine learning process includes a Ladder network process using an unsupervised machine learning process based one of the sets of initial features and the sets of updated features and a supervised learning process based one of the sets of initial features and the sets of updated features and one or more of the label data and the user feedback.

### Claim 6 (depends on 1)

The method of claim 1, further comprising modifying, by the computer, the label data based on the user feedback, wherein the semi-supervised machine learning process performed in response to the receiving of the user feedback is based on the label data modified based on the user feedback.

### Claim 7 (depends on 6)

The method of claim 6, wherein the user feedback comprises a first command to apply a first label to a first individual represented in the visualization data or a second command to remove the first label from the first individual.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
performing, by the computer, a clustering algorithm based on the semi-supervised machine learning process to determine clusters of individuals of the plurality of individuals; and
modifying, by the computer, the label data associated with a first individual based on a distance between the first individual and a first cluster of individuals.

### Claim 9 (depends on 1)

The method of claim 1, wherein the input data comprises image recognition data or speech recognition data.

### Claim 10 (depends on 1)

The method of claim 1, wherein the visualization data is displayed on a display.

### Claim 11

A machine learning computer system, comprising:
a non-transitory computer readable storage medium storing a plurality of instructions; and
one or more processors for executing the instructions stored on the non-transitory computer readable storage medium to:
obtain input data associated with a plurality of individuals;
obtain label data associated with a subset of the plurality of individuals, the label data assigning one or more labels to each individual of the subset;
perform a semi-supervised machine learning process based on the input data and the label data;
generate visualization data that is based on the semi-supervised machine learning process and is displayed to a user, the visualization data representing one or more segmentations of the plurality of individuals;
receive user feedback from the user corresponding to the visualization data; and
perform the semi-supervised machine learning process, in response to the receiving of the user feedback, based on the user feedback, the input data, and the label data, and wherein the instructions stored on the non-transitory computer readable storage medium cause the one or more processors to:
determine sets of initial features based on the input data, the sets of initial features including a set of initial features for each individual of the plurality of individuals,
wherein the semi-supervised machine learning process includes determining sets of updated features based on the sets of initial features and the label data, the sets of updated features including a set updated features for each individual of the plurality of individuals,
wherein the generating of the visualization data is based on the sets of updated features and the label data, and
wherein the sets of updated features are different than the sets of initial features, the sets of updated features more consistently characterizing individuals to specific labels than the sets of initial features.

### Claim 12 (depends on 11)

The computer system of claim 11, wherein the non-transitory computer readable storage medium further stores instructions to segment the plurality of individuals into one or more segments of individuals based on the label data, each individual of each segment of individuals being associated with a particular label in the label data.

### Claim 13 (depends on 11)

The computer system of claim 11, wherein the determining of the sets of initial features is based on a deep embedding model or an auto-encoder process.

### Claim 14 (depends on 11)

The computer system of claim 11, wherein the computer readable storage medium further stores instructions to perform one or more of a t-distributed stochastic neighbor embedding clustering process based on the sets of updated features, a bi-clustering process based on the sets of updated features, and a statistical summarization process based on the sets of updated features, and a profiling process on meta-data associated with the input data, wherein the generating of the visualization data is based on one or more of the t-distributed stochastic neighbor embedding clustering process, the statistical summarization process, and the profiling process.

### Claim 15 (depends on 11)

The computer system of claim 11, wherein the semi-supervised machine learning process includes a Ladder network process using an unsupervised machine learning process based one of the sets of initial features and the sets of updated features and a supervised learning process based one of the sets of initial features and the sets of updated features and one or more of the label data and the user feedback.

### Claim 16 (depends on 11)

The computer system of claim 11, wherein the computer readable storage medium further stores instructions to modify the label data based on the user feedback, wherein the semi-supervised machine learning process performed in response to the receiving of the user feedback is based on the label data modified based on the user feedback.

### Claim 17 (depends on 16)

The computer system of claim 16, wherein the user feedback comprises a first command to apply a first label to a first individual represented in the visualization data or a second command to remove the first label from the first individual.

### Claim 18 (depends on 11)

The computer system of claim 11, wherein the computer readable storage medium further stores instructions to:
perform a clustering algorithm based on the semi-supervised machine learning process to determine clusters of individuals of the plurality of individuals; and
modify the label data associated with a first individual based on a distance between the first individual and a first cluster of individuals.

### Claim 19 (depends on 11)

The computer system of claim 11, wherein the input data comprises image recognition data or speech recognition data.

### Claim 20 (depends on 11)

The computer system of claim 11, further comprising a display coupled to the one or more processors, the display configured to display the visualization data.