US 2022/0012425 A1 · TOTAL CORRELATION VARIATIONAL AUTOENCODER STRENGTHENED WITH ATTENTIONS FOR SEGMENTING SYNTAX AND SEMANTICS

Abstract

Described herein are embodiments of a framework named as total correlation variational autoencoder (TC_VAE) to disentangle syntax and semantics by making use of total correlation penalties of KL divergences. One or more Kullback-Leibler (KL) divergence terms in a loss for a variational autoencoder are discomposed so that generated hidden variables may be separated. Embodiments of the TC_VAE framework were examined on semantic similarity tasks and syntactic similarity tasks. Experimental results show that better disentanglement between syntactic and semantic representations have been achieved compared with state-of-the-art (SOTA) results on the same data sets in similar settings.

Claims20

Dependent claims

A computer-implemented method for segmenting latent representations comprising:

generating, using an embedding layer, a sequence of embeddings for a sequence of tokens;
generating, using an attention layer, a sequence of attention masks based on the sequence of embeddings;
generating a sequence of hidden variables based on the sequence of embeddings and the sequence of attention masks;
generating, using a first encoder and a second encoder respectively, a first sequence of latent variables and a second sequence of latent variables based on the sequence of hidden variables;
and
inferring, using a decoder, a sequence of reconstructed tokens and a sequence of reconstructed attention masks based on at least information of the first and second sequences of latent variables.

3 more elements

A system for segmenting latent representations comprising:

one or more processors;
and
a non-transitory computer-readable medium or media comprising one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:
generating a sequence of embeddings for a sequence of tokens;
generating a sequence of attention masks based on the sequence of embeddings;
generating a sequence of hidden variables based on the sequence of embeddings with the sequence of attention masks;
generating respectively a first sequence of latent variables and a second sequence of latent variables based on the sequence of hidden variables;
and
inferring a sequence of reconstructed tokens and a sequence of reconstructed attention masks based on at least information of the first and second sequences of latent variables.

6 more elements

A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by at least one processor, causes steps for segmenting latent representations comprising:

generating, using an embedding layer, a sequence of embeddings for a sequence of tokens;
generating, using an attention layer, a sequence of attention masks based on the sequence of embeddings;
generating a sequence of hidden variables based on the sequence of embeddings with the sequence of attention masks;
generating, using a first encoder and a second encoder respectively, a first sequence of latent variables and a second sequence of latent variables based on the sequence of hidden variables;
and
inferring, using a decoder, a sequence of reconstructed tokens and a sequence of reconstructed attention masks based on at least information of at least the first and second sequences of latent variables.

3 more elements

Dependencies are inferred from the claim text.

Total correlation variational autoencoder strengthened with attentions for segmenting syntax and semantics

Life of this patent

Total correlation variational autoencoder strengthened with attentions for segmenting syntax and semantics

Life of this patent

Abstract

Claims20