Back to results
US 12,587,675 B2Granted
Pixel-level video prediction with improved performance and efficiency
AssigneeGoogle Inc.
InventorsMohammad Babaeizadeh, Chelsea Breanna Finn, Dumitru Erhan, Mohammad Taghi Saffar, Sergey Vladimir Levine, Suraj Nair
Life of this patent
Estimating the term…
Abstract
One aspect provides a machine-learned video prediction model configured to receive and process one or more previous video frames to generate one or more predicted subsequent video frames, wherein the machine-learned video prediction model comprises a convolutional variational auto encoder, and wherein the convolutional variational auto encoder comprises an encoder portion comprising one or more encoding cells and a decoder portion comprising one or more decoding cells.
Claims23
A computing system for video prediction with improved efficiency, the computing system comprising:
- one or more processors;
- and
- one or more non-transitory computer-readable media that collectively store:
- a machine-learned video prediction model configured to receive and process one or more previous video frames to generate one or more predicted subsequent video frames, wherein the machine-learned video prediction model comprises a convolutional variational auto encoder, and wherein the convolutional variational auto encoder comprises:
- an encoder portion comprising one or more encoding cells configured to process the one or more previous video frames to generate an encoded representation of the one or more previous video frames,
- a dynamic portion comprising a machine-learned model configured to process the encoded representation of the one or more previous video frames to predict, based on the encoded representation of the one or more previous video frames, an encoded representation of the one or more predicted subsequent video frames, and
- a decoder portion comprising one or more decoding cells configured to process the encoded representation of the one or more predicted subsequent video frames to generate, based on the encoded representation of the one or more predicted subsequent video frames, the one or more predicted subsequent video frames,
- wherein the machine-learned video prediction model has been trained on training videos, and wherein the training videos were augmented prior to their use to train the machine-learned video prediction model;
- and
- instructions that, when executed by the one or more processors, cause the computing system to process the one or more previous video frames with the machine-learned video prediction model to generate the one or more predicted subsequent video frames.
7 more elements
A computing system for datastream prediction with improved efficiency, the computing system comprising:
- one or more processors;
- and
- one or more non-transitory computer-readable media that collectively store:
- a machine-learned datastream prediction model configured to receive and process one or more previous sets of data from a datastream to generate one or more predicted subsequent sets of data for the datastream,
- wherein the machine-learned datastream prediction model comprises a convolutional variational auto encoder,
- wherein the convolutional variational auto encoder comprises:
- an encoder portion comprising one or more encoding cells configured to process the one or more previous sets of data to generate an encoded representation of the one or more previous sets of data,
- a dynamic portion comprising a machine-learned model configured to process the encoded representation of the one or more previous sets of data to predict, based on the encoded representation of the one or more previous sets of data, an encoded representation of the one or more predicted subsequent sets of data, and
- a decoder portion comprising one or more decoding cells configured to process the encoded representation of the one or more predicted subsequent sets of data to generate, based on the encoded representation of the one or more predicted subsequent sets of data, the one or more predicted subsequent sets of data;
- the machine-learned datastream prediction model trained on training data modified according to one or more augmentation operations;
- and
- instructions that, when executed by the one or more processors, cause the computing system to process the one or more previous sets of data with the machine-learned datastream prediction model to generate the one or more predicted subsequent sets of data.
9 more elements
One or more non-transitory computer-readable media that collectively store:
- a machine-learned video prediction model configured to receive and process one or more previous video frames to generate one or more predicted subsequent video frames, wherein the machine-learned video prediction model comprises a convolutional variational auto encoder, and wherein the convolutional variational auto encoder comprises:
- an encoder portion comprising one or more encoding cells configured to process the one or more previous video frames to generate an encoded representation of the one or more previous video frames,
- a dynamic portion comprising a machine-learned model configured to process the encoded representation of the one or more previous video frames to predict, based on the encoded representation of the one or more previous video frames, an encoded representation of the one or more predicted subsequent video frames, and
- a decoder portion comprising one or more decoding cells configured to process the encoded representation of the one or more predicted subsequent video frames to generate, based on the encoded representation of the one or more predicted subsequent video frames, the one or more predicted subsequent video frames,
- wherein the machine-learned video prediction model has been trained on training videos, and wherein the training videos were augmented prior to their use to train the machine-learned video prediction model.
2 more elements
Dependencies are inferred from the claim text.