Transformer-based clustering
AssigneeInternational Business Machines Corporation
InventorsCharlotte Isabella Sinclair PARK, Parikshit RAM, Kenneth Lee CLARKSON, Lior HORESH
Life of this patent
Estimating the term…
Abstract
In some implementations, a device may input data points and cluster centers as token embeddings into a transformer architecture configured with k-means clustering for an encoder and a decoder. The device may perform a forward pass through the transformer architecture that uses a self-attention mechanism and a cross-attention mechanism to update cluster assignments for the data points and to update the cluster centers. The device may output the updated cluster assignments for the data points and the updated cluster centers from the transformer architecture.
Claims20
A method, comprising:
- inputting data points and cluster centers as token embeddings into a transformer architecture configured with k-means clustering for an encoder and a decoder;
- performing a forward pass through the transformer architecture that uses a self-attention mechanism and a cross-attention mechanism to update cluster assignments for the data points and to update the cluster centers;
- and
- outputting the updated cluster assignments for the data points and the updated cluster centers from the transformer architecture.
A computer system, comprising:
- a processor set;
- one or more computer-readable storage media;
- and
- program instructions stored on the one or more computer-readable storage media to cause the processor set to perform operations comprising:
- generating data token embeddings for data points with cluster assignments and center token embeddings for cluster centers;
- setting up an encoder-decoder transformer with the data token embeddings and the center token embeddings as input;
- performing a forward pass through the encoder-decoder transformer to update the cluster assignments and the cluster centers;
- extracting final cluster assignments and final cluster centers after the forward pass;
- and
- outputting the final cluster assignments and the final cluster centers.
A computer program product, comprising:
- one or more computer-readable storage media;
- and
- program instructions stored on the one or more computer-readable storage media to perform operations comprising:
- configuring a transformer architecture to perform discrete clustering of input data points;
- setting weights of the transformer architecture to mimic a Lloyd's algorithm for k-means clustering;
- performing a forward pass through the transformer architecture using the input data points and cluster centers as input;
- updating cluster assignments for the input data points based on an output of the transformer architecture;
- and
- outputting final cluster assignments and final cluster centers for the input data points from the output of the transformer architecture.
Dependencies are inferred from the claim text.