# US 2019/0377974 A1 · Deep Multi-Scale Video Prediction

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0377974 A1
- Document ID: US20190377974A1
- Status: Published application
- Kind: A1
- Application number: 16/550,774
- Filed: 2019-08-26
- Published: 2019-12-12
- Earliest claimed: 2016-11-16
- Assignees: Facebook, Inc.
- Applicants: Facebook, Inc.
- Inventors: Michael Fabien Mathieu; Camille Couprie; Yann Andre Le Cun
- Web page: https://patentgrep.com/patent/US20190377974A1

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F 18/21: Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/094: Adversarial learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/758: {Involving statistics of pixels or of feature values, e.g. histogram matching}
- G06V 10/82: using neural networks
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06N 3/048: Activation functions
- G06Q 10/40: Business processes related to social networking or social networking services
- G06V 20/44: {Event detection}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

In one embodiment, a method includes receiving a plurality of input frames of a video sequence associated with a time t, predicting, using a convolutional network, one or more future frames of the video sequence from the plurality of input frames, wherein the convolutional network is trained with randomly selected temporal sequences of a n×m grid of pixels from the plurality of input frames exhibiting a threshold of optical flow. In addition, the training may comprise randomly selecting temporal sequences of a n×m grid of pixels from the plurality of input frames exhibiting a threshold of optical flow.

## Claims

### Claim 1

A method comprising, by one or more computing devices:
receiving a plurality of input frames of a video sequence associated with a time t;
predicting, using a convolutional network, one or more future frames of the video sequence from the plurality of input frames, wherein the convolutional network is trained with randomly selected temporal sequences of a n×m grid of pixels from the plurality of input frames exhibiting a threshold of optical flow; and
outputting a first future frame of the video sequence associated with a time t+1 as predicted by the convolutional network.

### Claim 2 (depends on 1)

The method of claim 1, further comprising by the one or more computing devices:
predicting, using the convolutional network, a second future frame of the video sequence associated with time t+2 by using the first future frame as an input frame of the video sequence;
recursively applying the convolutional network based on the first future frame; and
outputting the second future frame of the video sequence associated with the time t+2 as predicted by the convolutional network.

### Claim 3 (depends on 1)

The method of claim 1, wherein the threshold of optical flow is determined based on a comparison of movement in the n×m grid of pixels between a time t−1 and the time t.

### Claim 4 (depends on 3)

The method of claim 3, wherein the comparison of movement comprises comparing pixel intensity associated with each pixel of the n×m grid of pixels between the time t−1 and the time t and calculating a similarity measure based on compared pixel intensities.

### Claim 5 (depends on 1)

The method of claim 1, wherein the convolutional network is further trained by evaluating a quality of the predicted one or more future frames of the video sequence by calculating a peak-signal-to-noise ratio between the one or more future frames of the video sequence and a ground truth frame of the video sequence.

### Claim 6 (depends on 1)

The method of claim 1, wherein the convolutional network is further trained by evaluating a quality of the predicted one or more future frames of the video sequence by calculating a structural similarity index measure between the one or more future frames of the video sequence and a ground truth frame of the video sequence.

### Claim 7 (depends on 1)

The method of claim 1, wherein the convolutional network employs an L1 loss to train the network on the input frames.

### Claim 8 (depends on 1)

The method of claim 1, wherein the convolutional network employs an L2 loss to train the network on the input frames.

### Claim 9 (depends on 1)

The method of claim 1, wherein the convolutional network employs a gradient difference loss to train the network on the input frames.

### Claim 10

One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
receive a plurality of input frames of a video sequence associated with a time t;
predict, using a convolutional network, one or more future frames of the video sequence from the plurality of input frames, wherein the convolutional network is trained with randomly selected temporal sequences of a n×m grid of pixels from the plurality of input frames exhibiting a threshold of optical flow; and
output a first future frame of the video sequence associated with a time t+1 as predicted by the convolutional network.

### Claim 11 (depends on 10)

The media of claim 10, wherein threshold of optical flow is determined based on a comparison of movement in the n×m grid of pixels between a time t−1 and the time t.

### Claim 12 (depends on 11)

The media of claim 11, wherein the comparison of movement comprises comparing pixel intensity associated with each pixel of the n×m grid of pixels between the time t−1 and the time t and calculating a similarity measure based on compared pixel intensities.

### Claim 13 (depends on 10)

The media of claim 10, wherein the training of the convolutional network may further comprise evaluating a quality of the predicted one or more future frames of the video sequence by calculating a peak-signal-to-noise ratio between the one or more future frames of the video sequence and a ground truth frame of the video sequence.

### Claim 14 (depends on 10)

The media of claim 10, wherein the training of the convolutional network may further comprise evaluating a quality of the predicted one or more future frames of the video sequence by calculating a structural similarity index measure between the one or more future frames of the video sequence and a ground truth frame of the video sequence.

### Claim 15

A system comprising: one or more processors; and one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:
receive a plurality of input frames of a video sequence associated with a time t;
predict, using a convolutional network, one or more future frames of the video sequence from the plurality of input frames, wherein the convolutional network is trained with randomly selected temporal sequences of a n×m grid of pixels from the plurality of input frames exhibiting a threshold of optical flow; and
output a first future frame of the video sequence associated with a time t+1 as predicted by the convolutional network.

### Claim 16 (depends on 15)

The system of claim 15, wherein the processors are further operable when executing the instructions to:
predict a second future frame of the video sequence associated with time t+2 by using the first future frame as an input frame of the video sequence,
recursively apply the generative model based on the first future frame, and
outputting the second future frame of the video sequence associated with the time t+2 as predicted by the generative model.

### Claim 17 (depends on 15)

The system of claim 15, wherein threshold of optical flow is determined based on a comparison of movement in the n×m grid of pixels between a time t−1 and the time t.

### Claim 18 (depends on 17)

The system of claim 17, wherein the comparison of movement comprises comparing pixel intensity associated with each pixel of the n×m grid of pixels between the time t−1 and the time t and calculating a similarity measure based on compared pixel intensities.

### Claim 19 (depends on 15)

The system of claim 15, wherein the training of the convolutional network may further comprise evaluating a quality of the predicted one or more future frames of the video sequence by calculating a peak-signal-to-noise ratio between the one or more future frames of the video sequence and a ground truth frame of the video sequence.

### Claim 20 (depends on 15)

The system of claim 15, wherein the training of the convolutional network may further comprise evaluating a quality of the predicted one or more future frames of the video sequence by calculating a structural similarity index measure between the one or more future frames of the video sequence and a ground truth frame of the video sequence.
