# US 2021/0279636 A1 · Efficient ground truth annotation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279636 A1
- Document ID: US20210279636A1
- Status: Published application
- Kind: A1
- Application number: 16/811,421
- Filed: 2020-03-06
- Published: 2021-09-09
- Earliest claimed: 2020-03-06
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Peter Willem Jan Staar; Michele Dolfi; Christoph Auer; Leonidas Georgopoulos; Ralf Kaestner; Alexander Velizhev; Dal Noguer Hidalgo; Rita Kuznetsova; Konstantinos Bekas
- Web page: https://patentgrep.com/patent/US20210279636A1

## CPC

- G06N 20/00: Machine learning
- G06F 16/285: {Clustering or classification}
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/091: Active learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A computer-implemented method for determining a set of target items to be annotated for training a machine learning application. The method comprises providing a training data set with a set of data samples and an auto-encoder with a classifier. The auto-encoder comprises an embedding model that maps the set of data samples to a set of compressed feature vectors. The set of compressed feature vectors define a compressed feature matrix. Further provided are: a definition of a graph associated to the compressed feature matrix, applying a clustering-algorithm to identify node clusters of the graph and applying a centrality algorithm to identify central nodes of the node clusters, retrieving from an annotator node labels for the central nodes, propagating the annotated node labels to other nodes of the graph and performing a training of the embedding model and the classifier with the annotated and the propagated node labels.

## Claims

### Claim 1

A computer-implemented method comprising:
providing a training data set comprising a set of data samples;
providing an auto-encoder, the auto-encoder encompassing a classifier;
performing an initial training of an embedding model of the auto-encoder, the embedding model being configured to map the set of data samples to a set of compressed feature vectors comprising feature elements, the set of compressed feature vectors defining a compressed feature matrix of the set of data samples;
providing a definition of a graph associated to the compressed feature matrix;
applying a clustering-algorithm to identify one or more node clusters of the graph;
applying a centrality algorithm to identify one or more central nodes of the one or more node clusters;
retrieving from an annotator one or more node labels for the one or more central nodes of the one or more node clusters, thereby creating annotated node labels;
propagating the annotated node labels of the one or more central nodes to other nodes of the graph, thereby creating propagated node labels; and
performing a further training of the auto-encoder, the further training comprising
training the embedding model of the auto-encoder and the classifier of the auto-encoder with the annotated and the propagated node labels, wherein the classifier is configured to predict the one or more node labels for the elements of the compressed feature vectors.

### Claim 2 (depends on 1)

The method according to claim 1, the method further comprising:
repeating, at one or more iteration rounds, the applying of the clustering algorithm, the applying of the centrality algorithm, the propagating of the annotated labels and the performing of the further training of the auto-encoder until convergence.

### Claim 3 (depends on 2)

The method according to claim 2, further comprising outputting the one or more node labels of the last iteration round before convergence as set of target items to be annotated.

### Claim 4 (depends on 1)

The method according to claim 1, the method further comprising:
repeating retrieving from one or more annotators node labels for the one or more central nodes of the one or more node clusters.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the graph is defined by an adjacency matrix, the adjacency matrix being the product of the compressed feature matrix and the transpose of the compressed feature matrix.

### Claim 6 (depends on 1)

The method according to claim 1, wherein applying the clustering algorithm comprises applying a graph-based clustering algorithm.

### Claim 7 (depends on 1)

The method according to claim 1, wherein applying the centrality algorithm comprises applying a graph-based centrality algorithm.

### Claim 8 (depends on 6)

The method according to claim 6, wherein the graph-based clustering algorithm is selected from the group consisting of:
k-spanning tree or minimum spanning tree algorithms;
shared nearest neighbor algorithms;
algorithms based on betweenness centrality; and
spectral clustering algorithms.

### Claim 9 (depends on 1)

The method according to claim 1, wherein the graph-based centrality algorithm comprises:
approximating a product of a matrix exponential and a random probe vector of the adjacency matrix;
computing a diagonal of the adjacency matrix based on the product of the matrix exponential and the random probe vector; and
calculating node centralities based on the computed diagonal until a predefined number of the one or more central nodes has been detected.

### Claim 10 (depends on 1)

The method according to claim 1, the method further comprising:
identifying one or more border nodes, the border nodes being defined as nodes which are situated at a border between two of the clusters.

### Claim 11 (depends on 10)

The method according to claim 10, the method further comprising:
retrieving from one or more annotators node labels for the one or more border nodes.

### Claim 12 (depends on 1)

The method according to claim 1, the method further comprising:
identifying one or more farthest nodes of the one or more clusters.

### Claim 13 (depends on 12)

The method according to claim 12, the method further comprising:
retrieving from one or more annotators node labels for one or more farthest nodes of the one or more clusters.

### Claim 14 (depends on 1)

The method according to claim 1, wherein the step of propagating the annotated node labels comprises applying a propagation function to the annotated node labels, wherein the propagation function is a function of the adjacency matrix.

### Claim 15 (depends on 14)

The method as claimed in claim 14, wherein the propagation function is defined as a Chebyshev expansion ƒ(X)=Σj=0 cj Pj(x) where Pj is the j-th Chebyshev polynomial.

### Claim 16 (depends on 1)

The method according to claim 1, wherein the machine learning application is a deep learning application.

### Claim 17

A method comprising:
providing a training data set comprising a set of data samples and a set of target items to be annotated to an annotator;
receiving the training data set with the annotated target items; and
training a cognitive model of the machine learning application with the training data comprising the annotated target items.

### Claim 18

A training data set comprising:
a set of data samples and a set of target items to be annotated to an annotator; and
annotated target items as annotated by the annotator.

### Claim 19 (depends on 18)

The training data set of claim 18, further comprising:
receiving the annotated training data set from the annotator; and
training a cognitive model of the machine learning application with the training data comprising the annotated target items.

### Claim 20

A system comprising one or more processors for executing computer-readable instructions, the computer-readable instructions controlling the one or more processors to perform operations comprising:
providing a training data set comprising a set of data samples;
providing an auto-encoder, the auto-encoder encompassing a classifier;
performing an initial training of an embedding model of the auto-encoder, the embedding model being configured to map the set of data samples to a set of compressed feature vectors comprising feature elements, the set of compressed feature vectors defining a compressed feature matrix of the set of data samples;
providing a definition of a graph associated to the compressed feature matrix;
applying a clustering-algorithm to identify one or more node clusters of the graph;
applying a centrality algorithm to identify one or more central nodes of the one or more node clusters;
retrieving from an annotator one or more node labels for the one or more central nodes of the one or more node clusters, thereby creating annotated node labels;
propagating the annotated node labels of the one or more central nodes to other nodes of the graph, thereby creating propagated node labels; and
performing a further training of the auto-encoder, the further training comprising:
training the embedding model of the auto-encoder and the classifier of the auto-encoder with the annotated and the propagated node labels, wherein the classifier is configured to predict the one or more nodes labels for the elements of the compressed feature vectors.

### Claim 21 (depends on 20)

The system according to claim 20 the system further comprising:
repeating, at one or more iteration rounds, the applying of the clustering algorithm, the applying of the centrality algorithm, the propagating of the annotated labels and the performing of the further training of the auto-encoder until convergence.

### Claim 22 (depends on 20)

The system according to claim 20, the system further comprising:
outputting the one or more node labels of the last iteration round before convergence as set of target items to be annotated.

### Claim 23 (depends on 21)

The system according to claim 21, the system being further configured to
repeat the step of retrieving from one or more annotators node labels for the one or more central nodes of the one or more clusters.

### Claim 24

A computer program product comprising a computer-readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform operations comprising:
providing a training data set comprising a set of data samples;
providing an auto-encoder, the auto-encoder encompassing a classifier;
performing an initial training of an embedding model of the auto-encoder, the embedding model being configured to map the set of data samples to a set of compressed feature vectors comprising feature elements, the set of compressed feature vectors defining a compressed feature matrix of the set of data samples;
providing a definition of a graph associated to the compressed feature matrix;
applying a clustering-algorithm to identify one or more node clusters of the graph;
applying a centrality algorithm to identify one or more central nodes of the one or more node clusters;
retrieving from an annotator one or more node labels for the one or more central nodes of the one or more node clusters, thereby creating annotated node labels;
propagating the annotated node labels of the one or more central nodes to other nodes of the graph, thereby creating propagated node labels; and
performing a further training of the auto-encoder, the further training comprising:
training the embedding model of the auto-encoder and the classifier of the auto-encoder with the annotated and the propagated node labels, wherein the classifier is configured to predict the one or more nodes labels for the elements of the compressed feature vectors.

### Claim 25 (depends on 24)

The computer program product of claim 24, further comprising:
repeating, at one or more iteration rounds, the applying of the clustering algorithm, the applying of the centrality algorithm, the propagating of the annotated labels and the performing of the further training of the auto-encoder until convergence.
