Method, system and storage medium implementing cross-modal contrastive learning to improve item categorization BERT model
AssigneeRakuten Group, Inc.
InventorsLei Chen, Hou Wei Chou
Life of this patent
Estimating the term…
Abstract
A method of contrastive learning (CL) performed by at least one processor includes receiving, by at least one processor implementing a text-based transformer, a text title corresponding to an anchor, generating, by the at least one processor implementing the text-based transformer, a text representation of the text title, receiving, by at least one processor implementing an image-based transformer, a first image corresponding to the anchor, generating, by the at least one processor implementing the image-based transformer, a first image representation of the first image corresponding to the anchor, and constructing a positive pair between the text representation and the first image representation.
Claims20
A method of contrastive learning (CL) performed by at least one processor and utilizing an anchor item having a text title, comprising:
- receiving, by at least one processor implementing a text-based transformer, the text title corresponding to the anchor;
- generating, by the at least one processor implementing the text-based transformer, a text representation of the text title that is different from the text title;
- receiving, by at least one processor implementing an image-based transformer, a first image corresponding to the anchor;
- generating, by the at least one processor implementing the image-based transformer, a first image representation of the first image corresponding to the anchor, the first image representation being a text representation corresponding to the first image;
- and
- constructing a positive pair between the text representation and the first image representation.
A system for contrastive learning (CL) utilizing an anchor item having a text title, comprising:
- at least one memory storing program code;
- and
- at least one processor configured to operate as instructed by the program code, the program code including:
- first receiving code configured to cause at least one of the at least one processor to receive, by a text-based transformer, the text title corresponding to the anchor;
- first generating code configured to cause at least one of the at least one processor to generate, by the text-based transformer, a text representation of the text title that is different from the text title;
- second receiving code configured to cause at least one of the at least one processor to receive, by an image-based transformer, a first image corresponding to the anchor;
- second generating code configured to cause at least one of the at least one processor to generate, by the image-based transformer, a first image representation of the first image corresponding to the anchor, the first image representation being a text representation corresponding to the first image;
- and
- first constructing code configured to cause at least one of the at least one processor to construct a positive pair between the text representation and the first image representation.
A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to:
- receive, by a text-based transformer, a text title corresponding to an anchor item;
- generate, by the text-based transformer, a text representation of the text title that is different from the text title;
- receive, by an image-based transformer, a first image corresponding to the anchor;
- generate, by the image-based transformer, a first image representation of the first image corresponding to the anchor, the first image representation being a text representation corresponding to the first image;
- and
- construct a positive pair between the text representation and the first image representation.
Dependencies are inferred from the claim text.