# US 12,223,719 B2 · Apparatus and method for prediction of video frame based on deep learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,223,719 B2
- Document ID: US12223719B2
- Status: Granted
- Kind: B2
- Application number: 17/548,824
- Filed: 2021-12-13
- Published: 2022-06-16
- Granted: 2025-02-11
- Earliest claimed: 2020-12-11
- Assignees: Korea University Research and Business Foundation
- Applicants: Korea University Research and Business Foundation
- Inventors: Kun Fan; Chung-In Joung; Seungjun Baek; Seunghwan Byun
- Web page: https://patentgrep.com/patent/US12223719B2

## CPC

- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/82: using neural networks
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

An apparatus and a method of predicting a video frame are provided. The apparatus includes a level encoder configured to extract and learn at least one feature from a video frame, a feature learning unit configured to learn based on the at least one feature or transmit predicted feature data corresponding to the at least one feature, and a level decoder configured to obtain and learn a predicted video frame based on the predicted feature data.

## Claims

### Claim 1

An apparatus for predicting a video frame, the apparatus comprising:
a level encoder configured to extract and learn at least one feature from a video frame;
a feature learning unit configured to learn based on the at least one feature or transmit predicted feature data corresponding to the at least one feature; and
a level decoder configured to obtain and learn a predicted video frame based on the predicted feature data,
wherein the level encoder receives first to (T−1)th video frames, respectively, and extracts at least one feature from each of the first to (T−1)th video frames, where “T” includes a natural number equal to or greater than 2,
wherein the feature learning unit is trained based on at least one feature extracted from each of the first to (T−1)th video frames,
wherein the level encoder receives the T-th video frame,
wherein the level decoder obtains a (T+1)th predicted video frame corresponding to the T th video frame,
wherein the level encoder receives the (T+1)th predicted video frame, and
wherein the level decoder obtains a (T+2)th predicted video frame corresponding to the (T+1)th predicted video frame.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the level encoder includes a first-level encoder to an N-th level encoder, and
wherein each of the first-level encoder to the N-th level encoder extracts features of different levels from the video frame where “N” is a natural number equal to or greater than 2.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein the feature learning unit includes a first feature learning unit to an N-th feature learning unit corresponding to each of the first-level encoder to the N-th level encoder, and
wherein each of the first feature learning unit to the N-th feature learning unit receives each feature of the different levels, obtains and transmit predicted feature data corresponding to each feature of the different levels.

### Claim 4 (depends on 3)

The apparatus of claim 3, wherein the level decoder includes a first-level decoder to an N-th level decoder corresponding to each of the first-level encoder to the N-th level encoder or corresponding to each of the first feature learning unit to the N-th feature learning unit, and
wherein the first-level decoder to the N-th level decoder receive each of the predicted feature data, respectively, and generate a predicted video frame by using the predicted feature data.

### Claim 5 (depends on 1)

The apparatus of claim 1, wherein at least one of the level encoder and the level decoder is based on at least one of a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), and a deep belief neural network (DBN).

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein the feature learning unit is based on a long short term memory (LSTM).

### Claim 7

A method of predicting a video frame, the method comprising:
extracting at least one feature from a video frame;
obtaining predicted feature data trained based on the at least one feature or corresponding to the at least one feature; and
obtaining a predicted video frame based on the predicted feature data,
wherein the extracting of the at least one feature from the video frame includes:
receiving first to (T−1)th video frames, respectively, and extracting at least one feature from each of the first to (T−1)th video frames, where “T” includes a natural number of 2 or more,
wherein the training based on the at least one feature includes:
training based on at least one feature extracted from each of the first to (T−1)th video frames,
wherein the extracting of the at least one feature from the video frame includes:
receiving a T-th video frame and extracting at least one feature from the T-th video frame,
wherein the obtaining of the predicted video frame based on the predicted feature data includes:
obtaining a (T+1) th predicted video frame corresponding to the T th video frame based on the predicted feature data.

### Claim 8 (depends on 7)

The method of claim 7, wherein the extracting of the at least one feature from the video frame includes:
extracting, by the first-level encoder to the N-th level encoder, features of different levels from each other from the video frame, where “N” includes a natural number equal to or greater than 2.

### Claim 9 (depends on 8)

The method of claim 8, wherein the obtaining of the predicted feature data corresponding to the at least one feature includes:
receiving, by a first feature learning unit to an N-th feature learning unit corresponding to the first-level encoder to the N-th level encoder, the features of the different levels, respectively; and
obtaining, by the first feature learning unit to the N-th feature learning unit, the predicted feature data corresponding to each of the features of the different levels, respectively, and transmitting the predicted feature data to a next frame processing process.

### Claim 10 (depends on 9)

The method of claim 9, wherein the obtaining of the predicted video frame based on the predicted feature data includes:
receiving each of the predicted feature data by a first-level decoder to an N-th level decoder, respectively; and
generating, by the first-level decoder to the N-th level decoder, the predicted video frame by using the predicted feature data, and
wherein the first-level decoder to the N-th level decoder correspond to each of the first-level encoder to the N-th level encoder, or corresponding to each of the first feature learning unit to the N-th feature learning unit.

### Claim 11 (depends on 7)

The method of claim 7, further comprising:
receiving the (T+1)th predicted video frame and obtaining a (T+2)th predicted video frame corresponding to the (T+1)th predicted video frame.

### Claim 12

A method of predicting a video frame, the method comprising:
extracting at least one feature from a video frame;
obtaining predicted feature data trained based on the at least one feature or corresponding to the at least one feature; and
obtaining a predicted video frame based on the predicted feature data,
wherein the extracting of the at least one feature from the video frame includes:
receiving first to (T−1)th video frames, respectively, and extracting at least one feature from each of the first to (T−1)th video frames, where “T” includes a natural number of 2 or more, and
wherein the training based on the at least one feature includes:
training based on at least one feature extracted from each of the first to (T−1)th video frames,
wherein the extracting of the at least one feature from the video frame includes:
extracting, by the first-level encoder to the N-th level encoder, features of different levels from each other from the video frame, where “N” includes a natural number equal to or greater than 2,
wherein the obtaining of the predicted feature data corresponding to the at least one feature includes:
receiving, by a first feature learning unit to an N-th feature learning unit corresponding to the first-level encoder to the N-th level encoder, the features of the different levels, respectively; and
obtaining, by the first feature learning unit to the N-th feature learning unit, the predicted feature data corresponding to each of the features of the different levels, respectively, and transmitting the predicted feature data to a next frame processing process, and
wherein the obtaining of the predicted video frame based on the predicted feature data includes:
receiving each of the predicted feature data by a first-level decoder to an N-th level decoder, respectively; and
generating, by the first-level decoder to the N-th level decoder, the predicted video frame by using the predicted feature data, and
wherein the first-level decoder to the N-th level decoder correspond to each of the first-level encoder to the N-th level encoder, or corresponding to each of the first feature learning unit to the N-th feature learning unit.