# US 5,708,732 A · Fast DCT domain downsampling and inverse motion compensation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 5,708,732 A
- Document ID: US5708732A
- Status: Granted
- Kind: A
- Application number: 08/611,922
- Filed: 1996-03-06
- Granted: 1998-01-13
- Earliest claimed: 1996-03-06
- Assignees: Hewlett-Packard Company
- Inventors: Neri Merhav; Vasudev Bhaskaran
- Web page: https://patentgrep.com/patent/US5708732A

## CPC

- G11B 27/034: on discs (G11B27/036, G11B27/038 take precedence)
- G06T 3/4084: in the transform domain, e.g. fast Fourier transform [FFT] domain scaling
- H04N 19/40: using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
- H04N 19/42: characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation (H04N19/635 takes precedence)
- H04N 19/48: using compressed domain processing techniques other than decoding, e.g. modification of transform coefficients, variable length coding [VLC] data or run-length data (motion estimation in a transform domain H04N19/547; processing of decoded motion vectors H04N19/513)
- H04N 19/51: Motion estimation or motion compensation
- H04N 19/59: involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
- H04N 19/61: in combination with predictive coding
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Downsampling and inverse motion compensation are performed on compressed domain representations for video. By directly manipulating the compressed domain representation instead of the spatial domain representation, computational complexity is significantly reduced. For downsampling, the compressed stream is processed in the compressed (DCT) domain without explicit decompression and spatial domain downsampling so that the resulting compressed stream corresponds to a scaled down image, ensuring that the resulting compressed stream conforms to the standard syntax of 8.times.8 DCT matrices. For typical data sets, this approach of downsampling in the compressed domain results in computation savings around 80% compared with traditional spatial domain methods for downsampling from compressed data. For inverse motion compensation, motion compensated compressed video is converted into a sequence of DCT domain blocks corresponding to the spatial domain blocks in the current picture alone. By performing inverse motion compensation directly in the compressed domain, the reduction in computation complexity is around 68% compared with traditional spatial domain methods for inverse motion compensation from compressed data. The techniques for downsampling and inverse motion compensation can be used in a variety of applications, such as multipoint video conferencing and video editing.

## Claims

### Claim 1

A method for downsampling image information in the compressed domain, where such information has been compressed in accordance with a discrete cosine transform (DCT) based compression scheme, the method comprising the steps of:
Huffman decoding a compressed bitstream to extract a plurality of N.times.N DCT-based data blocks;
processing n of said data blocks in accordance with at least one sampling matrix to extract a single N.times.N DCT-based data block, wherein n is determined by a desired downsampling factor, and wherein said single data block is a determined average of said N data blocks.

### Claim 2 (depends on 1)

The method of claim 1, said processing step further comprising the step of:
using partially factorized operators.

### Claim 3 (depends on 1)

The method of claim 1, said processing step further comprising the step of:
determining X, a DCT of an N.times.N vector x, from at least two N-dimensional vectors X.sub.1 and X.sub.2, where X has a plurality of components, each of which is an average of related components in x.sub.1 and x.sub.2, where x.sub.1 =S.sub.-1 X.sub.1 and x.sub.2 =S.sup.-1 X.sub.2, and where S is an N-point DCT matrix.

### Claim 4 (depends on 3)

The method of claim 3, said determining step further comprising the steps of:
performing a one-dimensional decimation by a factor of two in accordance with the relationship: ##EQU25## where Q.sub.1 and Q.sub.2 are N.times.N DCT-based matrices; where matrices U.sub.1 =SQ.sub.1 S.sup.-1 and U.sub.2 =SQ.sub.2 S.sup.-1 ;
where a factorization S is represented as follows:
S=DPB.sub.1 B.sub.2 MA.sub.1 A.sub.2 A.sub.3,
where D is a fixed diagonal matrix,
where P is a fixed permutation matrix;
where B.sub.1 B.sub.2 MA.sub.1 A.sub.2 A.sub.3 are N.times.N fixed matrices; and
where products:
F.sub.i =MA.sub.1 A.sub.2 A.sub.3 Q.sub.i A.sub.3.sup.-1 A.sub.2.sup.-1 A.sub.1.sup.-1 M.sup.-1 i=1,2
are sparse matrices, said matrix having a sum F.sub.+ =F.sub.1 +F.sub.2 and a difference F.sub.- =F.sub.1 -F.sub.2.

### Claim 5 (depends on 3)

The method of claim 3, said determining step further comprising the steps of:
downsampling by a factor of two in accordance with the relationship: ##EQU26## where Q.sub.1 and Q.sub.2 are N.times.N DCT-based matrices; where matrices U.sub.1 =SQ.sub.1 S.sup.-1 and U.sub.2 =SQ.sub.2 S.sup.-1 ;
where a factorization S is represented as follows:
S=DPB.sub.1 B.sub.2 MA.sub.1 A.sub.2 A.sub.3,
where D is a fixed diagonal matrix,
where P is a fixed permutation matrix;
where B.sub.1 B.sub.2 MA.sub.1 A.sub.2 A.sub.3 are N.times.N fixed matrices; and
where products:
F.sub.i =MA.sub.1 A.sub.2 A.sub.3 Q.sub.i A.sub.3.sup.-1 A.sub.2.sup.-1 A.sub.1.sup.-1 M.sup.-1 i=1,2
are sparse matrices, said matrix having a sum F.sub.+ =F.sub.1 +F.sub.2 and a difference F.sub.- =F.sub.1 -F.sub.2.

### Claim 6 (depends on 3)

The method of claim 3, said determining step further comprising the steps of:
downsampling by a factor of four in accordance with the relationship: ##EQU27## where Q.sub.1 and Q.sub.2 are N.times.N DCT-based matrices; where matrices U.sub.1 =SQ.sub.1 S.sup.-1 and U.sub.2 =SQ.sub.2 S.sup.-1 ;
where a factorization S is represented as follows:
S=DPB.sub.1 B.sub.2 MA.sub.1 A.sub.2 A.sub.3,
where D is a fixed diagonal matrix,
where P is a fixed permutation matrix;
where B.sub.1 B.sub.2 MA.sub.1 A.sub.2 A.sub.3 are N.times.N fixed matrices; and
where products:
F.sub.i =MA.sub.1 A.sub.2 A.sub.3 Q.sub.i A.sub.3.sup.-1 A.sub.2.sup.-1 A.sub.1.sup.-1 M.sup.-1 i=1,2
are sparse matrices, said matrix having a sum F.sub.+ =F.sub.1 +F.sub.2 and a difference F.sub.- =F.sub.1 -F.sub.2.

### Claim 7 (depends on 3)

The method of claim 3, said determining step further comprising the steps of:
downsampling by a factor of three in accordance with the relationship: ##EQU28## based on the fact that the matrices:
T=T.sub.1 +T.sub.2 +T.sub.3,
T.sub.+ =T.sub.1 +T.sub.3,
T.sub.- =(T.sub.1 -T.sub.3)/2
are relatively sparse, and on the identity: ##EQU29## where D is a fixed diagonal matrix, where P is a fixed permutation matrix; and
where B.sub.1 B.sub.2 MA.sub.1 A.sub.2 A.sub.3 are fixed N.times.N matrices.

### Claim 8 (depends on 7)

The method of claim 7, where: ##EQU30##

### Claim 9

A method for achieving inverse motion compensation in the compressed domain, where such information has been compressed in accordance with a discrete cosine transform (DCT) based compression scheme, the method comprising the steps of:
Huffman decoding a compressed bitstream to extract a plurality of N.times.N DCT-based data blocks defining a picture T-1, and to extract a motion vector h,w;
processing n of said data blocks in accordance with at least one sampling matrix and with said motion vector to produce a single N.times.N DCT-based data block corresponding to a picture T, wherein n is used to define a region occupied by said picture T-1, and wherein said single data block is a combination of a data block consisting of a determined average of said n data blocks.

### Claim 10 (depends on 9)

The method of claim 9, said processing step further comprising the step of:
using partially factorized operators.

### Claim 11 (depends on 9)

The method of claim 9, said processing step further comprising the step of:
determining a DCT X of a current block x=x+e from a given DCT E of a prediction error e of the current block, and DCTs X.sub.1, . . . , X.sub.4 of x.sub.1, . . . , x.sub.4 respectively, where X=X+E, and X is a DCT of x, such that X is determined directly from X.sub.1, . . . , X.sub.4.

### Claim 12 (depends on 9)

The method of claim 9, further comprising the step of: precomputing fixed matrices:
J.sub.i .DELTA.U.sub.i (MA.sub.1 A.sub.2 A.sub.3).sup.t, i=1,2, . . . ,8
and
K.sub.i .DELTA.L.sub.i (MA.sub.1 A.sub.2 A.sub.3).sup.t, i=1,2, . . . ,8,
where Q.sub.1 and Q.sub.2 are N.times.N DCT-based matrices;
where matrices U.sub.1 =SQ.sub.1 S.sup.-1 and K.sub.1 =SQ.sub.2 S.sup.-1 ;
where a factorization S is represented as follows:
S=DPB.sub.1 B.sub.2 MA.sub.1 A.sub.2 A.sub.3,
where D is a fixed diagonal matrix,
where P is a fixed permutation matrix; and
where B.sub.1 B.sub.2 MA.sub.1 A.sub.2 A.sub.3 are fixed N.times.N matrices.

### Claim 13 (depends on 12)

The method of claim 12, further comprising the step of:
determining X in accordance with: ##EQU31##

### Claim 14 (depends on 12)

The method of claim 12, further comprising the step of:
determining X in accordance with: ##EQU32##

### Claim 15 (depends on 12)

The method of claim 12, further comprising the step of:
determining X in accordance with:
X=(X.sub.1 DPB.sub.1 B.sub.2 J.sub.w.sup.t +X.sub.2 DPB.sub.1 B.sub.2 J.sub.8-w.sup.t)S.sup.t.

### Claim 16

An apparatus for downsampling image information in the compressed domain, where such information has been compressed in accordance with a discrete cosine transform (DCT) based compression scheme, the apparatus comprising:
a Huffman decoder for decoding a compressed bitstream to extract a plurality of N.times.N DCT-based data blocks;
a processor for processing n of said data blocks in accordance with at least one sampling matrix to produce a single N.times.N DCT-based data block, wherein n is determined by a desired downsampling factor, and wherein said single data block is a determined average of said n data blocks.

### Claim 17 (depends on 16)

The apparatus of claim 16, said processor further comprising:
means for using partially factorized operators.

### Claim 18 (depends on 16)

The apparatus of claim 16, said processing further comprising:
means for determining X, a DCT of an N.times.N vector x, from at least two N-dimensional vectors X.sub.1 and X.sub.2, where X has a plurality of components, each of which is an average of related components in x.sub.1 and x.sub.2, where x.sub.1 =S.sup.-1 X.sub.1 and x.sub.2 =S.sup.-1 X.sub.2, and where S is an N-point DCT matrix.

### Claim 19 (depends on 18)

The apparatus of claim 18, said means for determining further comprising:
means for performing a one-dimensional decimation by a factor of two in accordance with the relationship: ##EQU33## where Q.sub.1 and Q.sub.2 are N.times.N DCT-based matrices; where matrices U.sub.1 =SQ.sub.1 S.sup.-1 and U.sub.2 =SQ.sub.2 S.sup.-1 ;
where a factorization S is represented as follows:
S=DPB.sub.1 B.sub.2 MA.sub.1 A.sub.2 A.sub.3,
where D is a diagonal matrix,
where P is a permutation matrix;
where B.sub.1 B.sub.2 MA.sub.1 A.sub.2 A.sub.3 are N.times.N matrices; and
where products:
F.sub.i =MA.sub.1 A.sub.2 A.sub.3 Q.sub.i A.sub.3.sup.-1 A.sub.2.sup.-1 A.sub.1.sup.-1 M.sup.-1 i=1,2
are sparse matrices, said matrix having a sum F.sub.+ =F.sub.1 +F.sub.2 and a difference F.sub.- =F.sub.1 -F.sub.2.

### Claim 20 (depends on 18)

The apparatus of claim 18, said means for determining further comprising:
means for downsampling by a factor of two in accordance with the relationship: ##EQU34## where Q.sub.1 and Q.sub.2 are N.times.N DCT-based matrices; where matrices U.sub.1 =SQ.sub.1 S.sup.-1 and U.sub.2 =SQ.sub.2 S.sup.-1 ;
where a factorization S is represented as follows:
S=DPB.sub.1 B.sub.2 MA.sub.1 A.sub.2 A.sub.3,
where D is a fixed diagonal matrix,
where P is a fixed permutation matrix;
where B.sub.1 B.sub.2 MA.sub.1 A.sub.2 A.sub.3 are N.times.N fixed matrices; and
where products:
F.sub.i =MA.sub.1 A.sub.2 A.sub.3 Q.sub.i A.sub.3.sup.-1 A.sub.2.sup.-1 A.sub.1.sup.-1 M.sup.-1 i=1,2
are sparse matrices, said matrix having a sum F.sub.+ =F.sub.1 +F.sub.2 and a difference F.sub.- =F.sub.1 -F.sub.2.

### Claim 21 (depends on 18)

The apparatus of claim 18, said means for determining further comprising:
means for downsampling by a factor of four in accordance with the relationship: ##EQU35## where Q.sub.1 and Q.sub.2 are N.times.N DCT-based matrices; where matrices U.sub.1 =SQ.sub.1 S.sup.-1 and U.sub.2 =SQ.sub.2 S.sup.-1 ;
where a factorization S is represented as follows:
S=DPB.sub.1 B.sub.2 MA.sub.1 A.sub.2 A.sub.3,
where D is a fixed diagonal matrix,
where P is a fixed permutation matrix;
where B.sub.1 B.sub.2 MA.sub.1 A.sub.2 A.sub.3 are N.times.N fixed matrices; and
where products:
F.sub.i =MA.sub.1 A.sub.2 A.sub.3 Q.sub.i A.sub.3.sup.-1 A.sub.2.sup.-1 A.sub.1.sup.-1 M.sup.-1 i=1,2
are sparse matrices, said matrix having a sum F.sub.+ =F.sub.1 +F.sub.2 and a difference F.sub.- =F.sub.1 -F.sub.2.

### Claim 22 (depends on 18)

The apparatus of claim 18, said means for determining further comprising:
means for downsampling by a factor of three in accordance with the relationship: ##EQU36## based on the fact that the matrices:
T=T.sub.1 +T.sub.2 +T.sub.3,
T.sub.+ =T.sub.1 +T.sub.3,
T.sub.- =(T.sub.1 -T.sub.3)/2
are relatively sparse, and on the identity: ##EQU37## where D is a fixed diagonal matrix, where P is a fixed permutation matrix; and
where B.sub.1 B.sub.2 MA.sub.1 A.sub.2 A.sub.3 are N.times.N fixed matrices.

### Claim 23 (depends on 22)

The apparatus of claim 22, where: ##EQU38##

### Claim 24

An apparatus for achieving inverse motion compensation in the compressed domain, where such information has been compressed in accordance with a discrete cosine transform (DCT) based compression scheme, the apparatus comprising:
a Huffman decoder for decoding a compressed bitstream to produce a plurality of N.times.N DCT-based data blocks defining a picture T-1 and to produce a motion vector h,w;
a processor processing n of said data blocks in accordance with at least one sampling matrix and with said motion vector to produce a single N.times.N DCT-based data block corresponding to a frame T, wherein n is used to define a region occupied by said frame T-1, and wherein said single data block is a combination of a data block consisting of a determined average of said n data blocks.

### Claim 25 (depends on 24)

The apparatus of claim 24, said processor further comprising:
means for using partially factorized operators.

### Claim 26 (depends on 24)

The apparatus of claim 24, said processor further comprising:
means for determining a DCT X of a current block x=x+e from a given DCT E of a prediction error e, and DCTs X.sub.1, . . . , X.sub.4 of x.sub.1, . . . , x.sub.4 respectively, where X=X+E, and X is a DCT of x, such that X is determined directly from X.sub.1, . . . , X.sub.4.

### Claim 27 (depends on 26)

The apparatus of claim 26, further comprising:
means for precomputing fixed matrices:
J.sub.i .DELTA.U.sub.i (MA.sub.1 A.sub.2 A.sub.3).sup.t, i=1,2, . . . ,8
and
K.sub.i .DELTA.L.sub.i (MA.sub.1 A.sub.2 A.sub.3).sup.t, i=1,2, . . . ,8,
where Q.sub.1 and Q.sub.2 are N.times.N DCT-based matrices;
where matrices U.sub.1 =SQ.sub.1 S.sup.-1 and K.sub.1 =SQ.sub.2 S.sup.-1 ;
where a factorization S is represented as follows:
S=DPB.sub.1 B.sub.2 MA.sub.1 A.sub.2 A.sub.3,
where D is a fixed diagonal matrix,
where P is a fixed permutation matrix; and
where B.sub.1 B.sub.2 MA.sub.1 A.sub.2 A.sub.3 are N.times.N fixed matrices.

### Claim 28 (depends on 27)

The apparatus of claim 27, further comprising: means for determining X in accordance with: ##EQU39##

### Claim 29 (depends on 27)

The apparatus of claim 27, further comprising:
means for determining X in accordance with: ##EQU40##

### Claim 30 (depends on 27)

The apparatus of claim 27, further comprising:
means for determining X in accordance with:
X=(X.sub.1 DPB.sub.1 B.sub.2 J.sub.w.sup.t +X.sub.2 DPB.sub.1 B.sub.2 J.sub.8-w.sup.t)S.sup.t.