# US 2010/0166070 A1 · Low-resolution video coding content extraction

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2010/0166070 A1
- Document ID: US20100166070A1
- Status: Published application
- Kind: A1
- Application number: 12/346,936
- Filed: 2008-12-31
- Published: 2010-07-01
- Earliest claimed: 2008-12-31
- Assignees: NXP B.V.
- Inventors: Anurag Goel
- Web page: https://patentgrep.com/patent/US20100166070A1

## CPC

- H04N 19/59: involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
- H04N 19/122: Selection of transform size, e.g. 8x8 or 2x4x8 DCT; Selection of sub-band transforms of varying structure or type
- H04N 19/126: Details of normalisation or weighting functions, e.g. normalisation matrices or variable uniform quantisers
- H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/18: the unit being a set of transform coefficients
- H04N 19/44: Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
- H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence)
- H04N 19/48: using compressed domain processing techniques other than decoding, e.g. modification of transform coefficients, variable length coding [VLC] data or run-length data (motion estimation in a transform domain H04N19/547; processing of decoded motion vectors H04N19/513)
- H04N 19/61: in combination with predictive coding
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Low complexity method embodiments directly decode low-resolution frames from compressed high-resolution videos that were encoded using predictive coding techniques like the H.264 video coding standard. The smaller the decoding resolution, the higher will be the computation and power savings of using the method. Low-frequency coefficients of 2D transformed predictions are added to the low-frequency coefficients of the transformed residual error. Low-frequency coefficients of the reconstructed data are then inverse transformed taking a smaller size transform. Further savings are obtained by reconstructing only those reference pixels that will be needed for accurate decoding of further Intra blocks.

## Claims

### Claim 1

A method for extracting a low-resolution video from a high-resolution compressed video, comprising:
inputting a video encoded with predictive coding techniques;
extracting a low-resolution Intra frame from a high-resolution compressed Intra frame that has been encoded using predictive coding techniques;
extracting a low-resolution predictive/bi-predictive frame from a high-resolution compressed predictive/bi-predictive frame that has been encoded using predictive coding techniques and will not be used as reference to decode any future frame; and
outputting reconstructing frames as thumbnails.

### Claim 2 (depends on 1)

The method of claim 1, wherein said high-resolution compressed predictive/bi-predictive frame is taken immediately before an Intra/Intra Decoder Refresh (IDR) frame or another frame which is not used as a reference frame.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
adding low frequency coefficients of 2D transformed predictions to low frequency coefficients of transformed residual error;
wherein, low frequency coefficients of reconstructed data are inverse transformed by taking an inverse transform of smaller size.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
reconstructing only those reference pixels required for an accurate decoding of further Intra blocks.

### Claim 5 (depends on 1)

The method of claim 1, further comprising extracting low resolution Intra frames from high resolution compressed Intra frames which have been encoded using predictive coding techniques, by:
generating frequency coefficients of a prediction needed, wherein the number of frequency coefficients are no more than N/2×N/2 and a fraction of the number in high resolution compressed Intra frames being sampled;
generating corresponding frequency coefficients of a residual error;
reconstructing only those pixels of a particular block that will be used as reference pixels for decoding future blocks;
adding the frequency coefficients of the prediction needed to corresponding frequency coefficients of the residual error;
taking the inverse transform of the frequency coefficients that are no more than N/2×N/2 and a fraction of the number called for in high resolution compressed Intra frames;
normalizing a resulting down-sampled image; and
if a down-sampled frame generated is to be used as a reference frame, then any reference DC coefficients for a next frame are generated by a weighted average of four reference DC coefficients, so the DC coefficient of the reference block of the next frame are within a region covered by the four reference DC coefficients;
wherein, using down-sampled Intra frames as reference frames for Intra frame reconstruction introduces drift errors which are accepted.

### Claim 6 (depends on 1)

The method of claim 1, further comprising sub-sampling of Inter blocks in a predictive coding video bitstream, by:
using an integer portion of a motion vector to point to a reference block selected in a reference frame;
disregarding the fractional part of the motion vector;
generating frequency coefficients needed from a selected reference block, wherein, the number of these frequency coefficients are no more than N/2×N/2;
dropping a fractional part in sub-sampling to avoid the otherwise considerable computational load;
generating corresponding frequency coefficients of a residual error;
adding frequency coefficients of a prediction needed to corresponding frequency coefficients of a reconstructed residual error;
taking an inverse transform of only the few frequency coefficients needed;
normalizing a resulting down-sampled block to form a down-sampled image.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
if any down-sampled frame generated is to be used as a reference frame later, then a reference DC coefficient for next frame is generated from it using a weighted average of four reference DC coefficients, and the DC coefficients of a reference block of a next frame will be within a region covered by said four reference DC coefficients.

### Claim 8

A method for generating a down-sampled frame directly from a video input that was compressed using predictive coding techniques, comprising:
generating no more than N/2×N/2 number of frequency coefficients of prediction using a larger size transform of no less than N×N obtained from reference frame;
adding said frequency coefficients of prediction to corresponding frequency coefficients of a reconstructed residual error;
taking an inverse transform of said frequency coefficients;
normalizing with an inverse transform; and
forming a down-sampled image from the normalizing.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
if a generated down-sampled frame is to be used as a down-sampled reference frame, then generating a reference DC coefficient for a next frame from a down-sampled frame by weighted average of four reference DC coefficients.

### Claim 10 (depends on 9)

The method of claim 9, wherein a DC coefficient of the reference block of next frame is within a region covered by said four reference DC coefficients.

### Claim 11 (depends on 10)

The method of claim 10, wherein, if any down-sampled frames are used as reference frames, any resulting drift errors are either tolerated or their visual impact is mitigated by post-processing.

### Claim 12

A device for extracting low resolution frames from high resolution compressed frames which have been encoded using predictive coding techniques, comprising:
a device for generating frequency coefficients of a prediction needed, wherein the number of frequency coefficients are no more than N/2×N/2 and a fraction of the number in high resolution compressed Intra frames being sampled;
a device for generating corresponding frequency coefficients of a residual error;
a device for reconstructing only those pixels of a particular block that will be used as reference pixels for decoding future blocks;
a device for adding the frequency coefficients of the prediction needed to corresponding frequency coefficients of the residual error;
a device for taking the inverse transform of the frequency coefficients that are no more than N/2×N/2 and a fraction of the number called for in high resolution compressed Intra frames;
a device for normalizing a resulting down-sampled image; and
a device for deciding if a down-sampled frame generated is to be used as a reference frame, then any reference DC coefficients for a next frame are generated by a weighted average of four reference DC coefficients, so the DC coefficient of the reference block of the next frame are within a region covered by the four reference DC coefficients, wherein, using down-sampled Intra frames as reference frames for Intra frame reconstruction introduces drift errors which are accepted.

### Claim 13 (depends on 12)

The device of claim 12, further comprising:
a device for sub-sampling of Inter blocks in a predictive coding video bitstream;
a device for using an integer portion of a motion vector to point to a reference block selected in a reference frame;
a device for disregarding the fractional part of the motion vector;
a device for generating frequency coefficients needed from a selected reference block, wherein, the number of these frequency coefficients are no more than N/2×N/2;
a device for dropping a fractional part in sub-sampling to avoid the otherwise considerable computational load;
a device for generating corresponding frequency coefficients of a residual error;
a device for adding frequency coefficients of a prediction needed to corresponding frequency coefficients of a reconstructed residual error;
a device for taking an inverse transform of only the few frequency coefficients needed; and
a device for normalizing a resulting down-sampled block to form a down-sampled image.

### Claim 14 (depends on 10)

The device of claim 10, further comprising:
a device for deciding if any down-sampled frame generated is to be used as a reference frame later, then a reference DC coefficient for next frame is generated from it using a weighted average of four reference DC coefficients, and the DC coefficients of a reference block of a next frame will be within a region covered by said four reference DC coefficients.

### Claim 15 (depends on 10)

The device of claim 10, further comprising:
a video thumbnail device for generating a low resolution video bitstream for use as a supplementary video file for a corresponding high-resolution video bitstream;
a device for generating supplementary information files to index locations and frame properties of key frames in video bitstreams;
a device for associating and storing supplementary video files and supplementary information files along with their corresponding video bitstreams in a single container file format; and
a device for extracting selected low-resolution frames from a high-resolution stored compressed video file, wherein frames to be decoded are either chosen on-the-fly from the high-resolution coded bitstream, or are indexed in a high-resolution coded bitstream with the help of a supplementary information file

### Claim 16 (depends on 12)

The device of claim 12, further comprising:
a device for decoding selected frames and displaying only selected decoded low-resolution frames for selected video bitstreams when a user opens a video archive.

### Claim 17 (depends on 12)

The device of claim 12, further comprising:
a device for visually communicating to a user simultaneous, motion picture content of a number of video files.

### Claim 18 (depends on 12)

The device of claim 12, further comprising:
a device for sampling a high-resolution compressed predictive/bi-predictive frame placed immediately before an Intra/Intra Decoder Refresh (IDR) frame, and which will not be used later as a reference frame, and using it as a basis for generating a low resolution video.

### Claim 19 (depends on 18)

The device of claim 18, further comprising:
a supplementary video file associated with a corresponding high-resolution video bitstream and providing an index of locations and frame properties of key frames.

### Claim 20 (depends on 19)

The device of claim 19, further comprising:
a supplementary information file that indexes a high-resolution coded bitstream from a stored compressed video file so frames to be decoded can be chosen on-the-fly by a user.
