# US 10,771,807 B1 · System and method for compressing video using deep learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,771,807 B1
- Document ID: US10771807B1
- Status: Granted
- Kind: B1
- Application number: 16/369,377
- Filed: 2019-03-29
- Granted: 2020-09-08
- Earliest claimed: 2019-03-28
- Assignees: Wipro Limited
- Applicants: Wipro Limited
- Inventors: Sethuraman Ulaganathan; Manjunath Ramachandra
- Web page: https://patentgrep.com/patent/US10771807B1

## CPC

- H04N 19/50: using predictive coding (H04N19/61 takes precedence)
- H04N 19/52: by predictive encoding
- G06F 17/15: Correlation function computation {including computation of convolution operations (arithmetic circuits for sum of products per se, e.g. multiply-accumulators G06F7/5443; digital filters, e.g. FIR, IIR, adaptive filters H03H17/00)}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 3/091: Active learning
- H04N 19/159: Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
- H04N 19/184: the unit being bits, e.g. of the compressed video stream
- H04N 19/186: the unit being a colour or a chrominance component
- G06N 3/08: Learning methods
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method and system for compressing videos using deep learning is disclosed. The method includes segmenting each of a plurality of frames associated with a video into a plurality of super blocks. The method further includes determining a block size for partition of each of the plurality of super blocks into a plurality of sub blocks, based on a feature of each of the plurality of super blocks using a Convolutional Neural Network (CNN). The method further includes generating a prediction data for each of the plurality of sub blocks based on a motion vector predicted and learned by the CNN. The method further includes determining a residual data for each of the plurality of sub blocks by subtracting the prediction data from an associated original data. The method includes generating a transformed quantized residual data using each of a transformation algorithm and a quantization algorithm.

## Claims

### Claim 1

A method of compressing videos using deep learning, the method comprising:
segmenting, by a video compressing device, each of a plurality of frames associated with a video into a plurality of super blocks based on an element present in each of the plurality of frames and a motion associated with the element, wherein the segmentation of the plurality frames into the plurality of super blocks is of variable shape and size;
determining, by the video compressing device, a block size for partition of each of the plurality of super blocks into a plurality of sub blocks, based on a feature of each of the plurality of super blocks using a Convolutional Neural Network (CNN), wherein the feature comprises at least one of a size of the super block and a motion related information;
generating, by the video compression device, a prediction data for each of the plurality of sub blocks based on a motion vector predicted and learned by the CNN, wherein the CNN predicts the motion vector based on a co-located frames;
determining, by the video compression device, a residual data for each of the plurality of sub blocks by subtracting the prediction data from an associated original data, wherein the associated original data is a bit stream of each of the plurality of sub blocks; and
generating, by the video compressing device, a transformed quantized residual data using each of a transformation algorithm and a quantization algorithm based on a plurality of parameters associated with the residual data, wherein the plurality of parameters comprises the compression rate and signal to noise ratio.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
receiving the video from an external computing device through an interface; and
performing pre-processing analytics on the video, wherein the pre-processing analytics comprises at least one of removal of noise or converting of Red Green Blue (RGB) color space to YCbCr color space.

### Claim 3 (depends on 1)

The method of claim 1, further comprising training the CNN for each of the plurality of super block based on the feature of a plurality of set of frames associated with a plurality of video compression techniques and a user feedback to the CNN.

### Claim 4 (depends on 3)

The method of claim 3, further comprising predicting, by the trained CNN, for each of the plurality of super blocks, at least one of a prediction data, the block size, or a motion related information.

### Claim 5 (depends on 1)

The method of claim 1, further comprising selecting, by the CNN, a suitable prediction mode, wherein the suitable prediction mode is at least one of an inter mode or an intra mode.

### Claim 6 (depends on 5)

The method of claim 5, wherein the inter mode comprises prediction between a frame and at least one adjacent frame within the plurality of frames, and wherein the intra mode comprises prediction within the frame.

### Claim 7 (depends on 1)

The method of claim 1, wherein the transformation algorithm and the quantization algorithm is applied to compress the residual data.

### Claim 8 (depends on 7)

The method of claim 7, wherein the transformation algorithm is based on at least one of the CNN or a gaussian pulse wavelet.

### Claim 9 (depends on 1)

The method of claim 1, further comprising generating a plurality of compressed bit streams for the transformed quantized residual data based on an entropy coding.

### Claim 10 (depends on 1)

The method of claim 1, wherein the element comprises at least one of an object present in a frame of the plurality of frames and texture associated with the object.

### Claim 11

A video compressing device using deep learning, the video compressing device comprising:
a processor; and
a memory communicatively coupled to the processor, wherein the memory stores processor instructions, which, on execution, causes the processor to:
segment each of a plurality of frames associated with a video into a plurality of super blocks based on an element present in each of the plurality of frames and a motion associated with the element, wherein the segmentation of the plurality of frames into the plurality of super blocks is of variable shape and size;
determine a block size for partition of each of the plurality of super blocks into a plurality of sub blocks, based on a feature of each of the plurality of super blocks using a Convolutional Neural Network (CNN), wherein the feature comprises at least one of a size of the super block and a motion related information;
generate a prediction data for each of the plurality of sub blocks based on a motion vector predicted and learned by the CNN, wherein the CNN predicts the motion vector based on a co-located frames;
determine a residual data for each of the plurality of sub blocks by subtracting the prediction data from an associated original data, wherein the associated original data is a bit stream of each of the plurality of sub blocks; and
generate a transformed quantized residual data using each of a transformation algorithm and a quantization algorithm based on a plurality of parameters associated with the residual data, wherein the plurality of parameters comprises the compression rate and Signal to noise ratio.

### Claim 12 (depends on 11)

The video compressing device of claim 11, wherein the processor instructions further cause the processor to:
receive the video from an external computing device through an interface; and
perform pre-processing analytics on the video, wherein the pre-processing analytics comprises at least one of removal of noise or converting of Red Green Blue (RGB) color space to YCbCr color space.

### Claim 13 (depends on 11)

The video compressing device of claim 11, further comprising training the CNN for each of the plurality of super block based on the feature of a plurality of set of frames associated with a plurality of video compression techniques and a user feedback to the CNN.

### Claim 14 (depends on 13)

The video compressing device of claim 13, further comprising predicting, by the trained CNN, for each of the plurality of super blocks, at least one of a prediction data, the block size, or a motion related information.

### Claim 15 (depends on 11)

The video compressing device of claim 11, further comprising selecting, by the CNN, a suitable prediction mode, wherein the suitable prediction mode is at least one of an inter mode or an intra mode.

### Claim 16 (depends on 15)

The video compressing device of claim 15, wherein the inter mode comprises prediction between a frame and at least one adjacent frame within the plurality of frames, and wherein the intra mode comprises prediction within the frame.

### Claim 17 (depends on 11)

The video compressing device of claim 11, wherein the transformation algorithm and the quantization algorithm is applied to compress the residual data.

### Claim 18 (depends on 17)

The video compressing device of claim 17, wherein the transformation algorithm is based on at least one of the CNN or a gaussian pulse wavelet.

### Claim 19 (depends on 11)

The video compressing device of claim 11, further comprising generating a plurality of compressed bit streams for the transformed quantized residual data based on an entropy coding.

### Claim 20

A non-transitory computer-readable storage medium having stored thereon, a set of computer-executable instructions causing a computer comprising one or more processors to perform steps comprising:
segmenting each of a plurality of frames associated with a video into a plurality of super blocks based on an element present in each of the plurality of frames and a motion associated with the element, wherein the segmentation of the plurality of frames into the plurality super blocks is of variable shape and size;
determining a block size for partition of each of the plurality of super blocks into a plurality of sub blocks, based on a feature of each of the plurality of super blocks using a Convolutional Neural Network (CNN), wherein the feature comprises at least one of a size of the super block and a motion related information;
generating a prediction data for each of the plurality of sub blocks based on a motion vector predicted and learned by the CNN, wherein the CNN predicts the motion vector based on a co-located frames;
determining a residual data for each of the plurality of sub blocks by subtracting the prediction data from an associated original data, wherein the associated original data is a bit stream of each of the plurality of sub blocks; and
generating a transformed quantized residual data using each of a transformation algorithm and a quantization algorithm based on a plurality of parameters associated with the residual data, wherein the plurality of parameters comprises the compression rate and Signal to noise ratio.