# US 2021/0233224 A1 · Artificial intelligence based perceptual video quality assessment system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0233224 A1
- Document ID: US20210233224A1
- Status: Published application
- Kind: A1
- Application number: 17/156,588
- Filed: 2021-01-24
- Published: 2021-07-29
- Earliest claimed: 2020-01-23
- Assignees: Modaviti Emarketing Pvt Ltd
- Applicants: Modaviti Emarketing Pvt Ltd
- Inventors: MANOJ SHINDE; DANISH JAMIL; ABHIT SINHA
- Web page: https://patentgrep.com/patent/US20210233224A1

## CPC

- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- H04L 65/611: for multicast or broadcast (systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06)
- H04L 65/80: Responding to QoS
- H04N 17/02: for colour television signals
- H04N 21/238: Interfacing the downstream path of the transmission network, e.g. adapting the transmission rate of a video stream to network bandwidth; Processing of multiplex streams {(hybrid fiber coaxial [HFC] networks for downstream channel allocation for video distribution H04L12/2801; flow control in packet networks H04L47/10; real-time communication protocols in data switching networks H04L65/00; scheduling or organising the servicing of application requests H04L67/60)}
- H04N 21/2402: {Monitoring of the downstream path of the transmission network, e.g. bandwidth available (traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876)}
- H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/4666: {using neural networks, e.g. processing the feedback provided by the user}
- H04N 21/6106: {specially adapted to the downstream path of the transmission network}
- H04N 21/637: Control signals issued by the client directed to the server or network components
- H04N 21/658: Transmission by the client directed to the server
- H04N 21/84: Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)}
- H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30168: Image quality inspection
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A video quality evaluation system comprises a training module, a calculation module, an analytical module, a designing module, an optimization module, and an estimation module. The training module collects training videos and trains labels of perceptual quality generation that is associated with the collected videos. The calculation module determines objective metrics based on the trained labels that are associated with the collected videos. The analytical module analyses scenes of the training video, and correlates objective metrics associated with the analysed scenes using perceptual parameters. The designing module designs a Convolutional Neural Network (CNN) Architecture based on data associated with the objective metrics and trains a model generated based on the designed CNN architecture. The optimization module optimizes the model and optimizes power after the model optimization. The estimation module estimates perceptual quality scores for incoming video data after the power optimization.

## Claims

### Claim 1

A method for video quality evaluation:
collecting training videos via a training module controlled by a processor;
training, via the training module, labels of perceptual quality generation that is associated with the collected training videos;
determining, via calculation module, objective metrics based on the trained labels that are associated with the collected training videos;
analysing, via an analytical module controlled by the processor, one or more scenes of the training video;
correlating, via the analytical module, objective metrics associated with the analysed scenes using perceptual parameters;
designing, via a designing module, a Convolutional Neural Network (CNN) Architecture based on data associated with the objective metrics retrieved from the correlation;
training, via the training module, a model generated based on the designed CNN architecture;
optimizing, via an optimization module, the model generated based on the designed CNN architecture, wherein a hardware optimised solution is generated for deployment at a router level;
optimizing power via the optimization module, after the model optimization; and
estimating, via an estimation module controlled by the processor, perceptual quality scores for incoming video data after the power optimization.

### Claim 2 (depends on 1)

The method as claimed in claim 1, wherein the training videos comprise multiple video codecs where different video transmission channels use different type of video encoding and decoding, and wherein the training captures degradation introduced by each of the multiple codecs to calculate the perceptual quality scores.

### Claim 3 (depends on 1)

The method as claimed in claim 1, wherein performing scene classification on the collected training videos generate content aware dataset, and wherein the content aware dataset is used for the identification of multiple classes.

### Claim 4 (depends on 1)

The method as claimed in claim 1, further comprising quality parameters associated with the training of the model include one or more of: blocking, blur, freezing, flickering, Peak Signal-To-Noise Ratio (PSNR), Mean-Square Error (MSE), Structural Similarity Index Measure (SSIM), and Natural Image Quality Evaluator (NIQE).

### Claim 5 (depends on 1)

The method as claimed in claim 1, wherein the CNN architecture is designed for higher accuracy while keeping the resource requirement minimum, which facilitates performing the video quality evaluation on low power devices while maintaining accuracy and throughput.

### Claim 6 (depends on 5)

The method as claimed in claim 5, wherein the CNN architecture is designed for higher accuracy while keeping the resource requirement minimum.

### Claim 7

A video quality evaluation system comprising:
a training module that is controlled by a processor collects training videos, wherein the training module trains labels of perceptual quality generation that is associated with the collected training videos;
a calculation module determines objective metrics based on the trained labels that are associated with the collected training videos;
an analytical module controlled by the processor analyses one or more scenes of the training video, wherein the analytical module correlates objective metrics associated with the analysed scenes using perceptual parameters;
a designing module designs a Convolutional Neural Network (CNN) Architecture based on data associated with the objective metrics retrieved from the correlation, wherein the training module trains a model generated based on the designed CNN architecture;
an optimization module optimizes the model generated based on the designed CNN architecture, wherein a hardware optimised solution is generated for deployment at a router level, and wherein the optimization module optimizes power after the model optimization; and
an estimation module controlled by the processor estimates perceptual quality scores for incoming video data after the power optimization.

### Claim 8 (depends on 7)

The video quality evaluation system as claimed in claim 7, wherein the training videos comprise of multiple video codecs where different video transmission channels use different type of video encoding and decoding, and wherein the training captures degradation introduced by each of the multiple codecs to calculate the perceptual quality scores.

### Claim 9 (depends on 7)

The video quality evaluation system as claimed in claim 7, wherein scene classification is performed on the collected training videos to generate content aware dataset, and wherein the content aware dataset is used for the identification of multiple classes.

### Claim 10 (depends on 7)

The video quality evaluation system as claimed in claim 7, further comprising quality parameters associated with the training of the model include one or more of: blocking, blur, freezing, flickering, Peak Signal-To-Noise Ratio (PSNR), Mean-Square Error (MSE), Structural Similarity Index Measure (SSIM), and Natural Image Quality Evaluator (NIQE).

### Claim 11

A non-transitory computer program product to evaluate video quality, when executed by a computer, the computer program product comprising programmed codes to:
collect training videos via a processor;
train, via the processor, labels of perceptual quality generation that is associated with the collected training videos;
determine, via the processor, objective metrics based on the trained labels that are associated with the collected training videos;
analyse, via the processor, one or more scenes of the training video;
correlate, via the processor, objective metrics associated with the analysed scenes using perceptual parameters;
design, via the processor, a Convolutional Neural Network (CNN) Architecture based on data associated with the objective metrics retrieved from the correlation;
train, via the processor, a model generated based on the designed CNN architecture;
optimize, via the processor, the model generated based on the designed CNN architecture, wherein a hardware optimised solution is generated for deployment at a router level;
optimize, via the processor, power via the optimization module, after the model optimization; and
estimate, via the processor, perceptual quality scores for incoming video data after the power optimization.
