# US 2016/0253784 A1 · System for video super resolution using semantic components

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0253784 A1
- Document ID: US20160253784A1
- Status: Published application
- Kind: A1
- Application number: 15/154,058
- Filed: 2016-05-13
- Published: 2016-09-01
- Earliest claimed: 2014-12-11
- Assignees: Sharp Laboratories of America, Inc.
- Applicants: Sharp Laboratories of America, Inc.
- Inventors: Xu CHEN; Anustup Kumar A. CHOUDHURY; Christopher A. SEGALL; Petrus J.L. VAN BEEK
- Web page: https://patentgrep.com/patent/US20160253784A1

## CPC

- G06T 3/10: Selection of transformation methods according to the characteristics of the input images
- G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403)
- G06T 5/73: Deblurring; Sharpening
- G06T 7/33: using feature-based methods
- G06V 40/167: {using comparisons between temporally consecutive images}
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30201: Face
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method for increasing the resolution of a series of low resolution frames of a low resolution video sequence to a series of high resolution frames of a high resolution video sequence includes receiving the series of low resolution frames of the video sequence. The system determines a first plurality of semantically relevant key points of a first low resolution frame of the series of low resolution frames and determines a second plurality of semantically relevant key points of a second low resolution frame of the series of low resolution frames. The system temporally processes the first plurality of key points based upon the second plurality of key points to determine a more temporally consistent set of key points for the first plurality of key points.

## Claims

### Claim 1

A method for modifying the resolution of a series of frames of a video sequence using an image processing system comprising:
(a) receiving said series of frames of said video sequence by said image processing system;
(b) determining a first plurality of key points of a first frame of said series of frames based upon semantic content of said first frame of said series of frames by said image processing system;
(c) based on said first plurality of key points, determining a first set of candidate images in a database that match a semantic region of said first frame;
(d) determining a second plurality of key points of a second frame of said series of frames based upon semantic content of said second frame of said series of frames by said image processing system;
(e) based on said second plurality of key points, determining a second set of candidate images in said database that match a semantic region of said second frame;
(f) selecting a selected candidate image in said database, based on said first set of candidate images and said second set of candidate images;
(g) modifying said resolution of said second frame of said series of frames based on said selected candidate image in said database.

### Claim 2 (depends on 1)

The method of claim 1 wherein said determining said first set of candidate images in said database is based upon a score.

### Claim 3 (depends on 2)

The method of claim 2 wherein score is based upon pixel values.

### Claim 4 (depends on 3)

The method of claim 3 wherein said pixel values includes at least one of difference pixel values and pixel gradient values.

### Claim 5 (depends on 1)

The method of claim 1 wherein said selecting said selected candidate image is based upon temporal consistency.

### Claim 6 (depends on 5)

The method of claim 5 wherein said temporal consistency is based upon a frequency of semantic content over a plurality of said series of said frames.

### Claim 7 (depends on 5)

The method of claim 5 wherein said temporal consistency is based upon a matching strength of semantic content over a plurality of said series of said frames.

### Claim 8 (depends on 1)

The method of claim 1 wherein said first set of candidate images in said database are selected as a first set of best matching candidate images.

### Claim 9 (depends on 1)

The method of claim 1 wherein a plurality of said set of candidate images in said database are ranked over a temporal time period.

### Claim 10 (depends on 9)

The method of claim 9 wherein said ranking is based upon a histogram.

### Claim 11 (depends on 1)

The method of claim 1 wherein a plurality of said set of candidate images is selectively modified based upon individual frames of said series of frames and temporal consistency over a plurality of said frames of said series of frames.

### Claim 12 (depends on 1)

The method of claim 1 further comprising said modifying said resolution of said second frame of said series of frames based upon a selected candidate image from said database.

### Claim 13 (depends on 12)

The method of claim 12 further comprising said modifying based upon said selected candidate image is based upon a blending process.