Systems and methods for mutual information based self-supervised learning
Assigneesalesforce.com, inc.
InventorsJunwen Bai, Weiran Wang, Yingbo Zhou, Caiming Xiong
Life of this patent
Estimating the term…
Abstract
Embodiments described herein combine both masked reconstruction and predictive coding. Specifically, unlike contrastive learning, the mutual information between past states and future states are directly estimated. The context information can also be directly captured via shifted masked reconstruction—unlike standard masked reconstruction, the target reconstructed observations are shifted slightly towards the future to incorporate more predictability. The estimated mutual information and shifted masked reconstruction loss can then be combined as the loss function to update the neural model.
Claims20
A system for mutual information based self-supervised learning, the system comprising:
- a memory configured to store a machine learning model;
- an input interface configured to receive a time sequence of input data instances over a time window corresponding to a window size;
- a processor configured to:
- encode, via an encoder of the machine learning model, the time sequence of input data instances to a sequence of latent states;
- generate a past vector in a latent space and a future vector in the latent space separated by a first timestamp based on the sequence of latent states;
- compute mutual information corresponding to the first timestamp between the past vector and the future vector in the latent space;
- compute a mutual information objective for the time window by averaging the computed mutual information across all timestamps within the time window;
- and
- update the machine learning model by maximizing the mutual information objective.
A method for mutual information based self-supervised learning, the method comprising:
- receiving, via a data interface, a time sequence of input data instances over a time window corresponding to a window size;
- encoding, via an encoder of the machine learning model, the time sequence of input data instances to a sequence of latent states;
- generating a past vector in a latent space and a future vector in the latent space separated by a first timestamp based on the sequence of latent states;
- computing mutual information corresponding to the first timestamp between the past vector and the future vector in the latent space;
- computing a mutual information objective for the time window by averaging the computed mutual information across all timestamps within the time window;
- and
- updating the machine learning model by maximizing the mutual information objective.
A non-transitory processor-readable storage medium storing processor-executable instructions for mutual information based self-supervised learning, the instructions being executed by a processor to perform operations comprising:
- receiving, via a data interface, a time sequence of input data instances over a time window corresponding to a window size;
- encoding, via an encoder of the machine learning model, the time sequence of input data instances to a sequence of latent states;
- generating a past vector in a latent space and a future vector in the latent space separated by a first timestamp based on the sequence of latent states;
- computing mutual information corresponding to the first timestamp between the past vector and the future vector in the latent space;
- computing a mutual information objective for the time window by averaging the computed mutual information across all timestamps within the time window;
- and
- updating the machine learning model by maximizing the mutual information objective.
Dependencies are inferred from the claim text.