# US 7,609,763 B2 · Advanced bi-directional predictive coding of video frames

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 7,609,763 B2
- Document ID: US7609763B2
- Status: Granted
- Kind: B2
- Application number: 10/622,378
- Filed: 2003-07-18
- Published: 2005-01-20
- Granted: 2009-10-27
- Earliest claimed: 2003-07-18
- Assignees: Microsoft Corporation
- Inventors: Kunal Mukerjee; Sridhar Srinivasan; Bruce Chih-Lung Lin
- Web page: https://patentgrep.com/patent/US7609763B2

## CPC

- H04N 19/577: Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
- H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
- H04N 19/56: Motion estimation with initialisation of the vector search, e.g. estimating a good candidate to initiate a search
- H04N 19/587: involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Techniques and tools for coding/decoding of video images, and in particular, B-frames, are described. In one aspect, a video encoder/decoder determines a fraction for a current image in a sequence. The fraction represents an estimated temporal distance position for the current image relative to an interval between a reference images for the current image. The video encoder/decoder processes the fraction along with a motion vector for a first reference image, resulting in a representation of motion (e.g., constant or variable velocity motion) in the current image. Other aspects are also described, including intra B-frames, forward and backward buffers for motion vector prediction, bitplane encoding of direct mode prediction information, multiple motion vector resolutions/interpolation filters for B-frames, proactive dropping of B-frames, and signaling of dropped predicted frames.

## Claims

### Claim 1

In a computing device that implements a video decoder, the computing device including a processor and memory, a method of decoding images in a sequence of video images, the method comprising:
receiving and decoding, with the computing device that implements the video decoder, a code in a bit stream to determine a fraction for a current image in the sequence, wherein the fraction represents an estimated temporal distance position for the current image relative to an interval between a first reference image for the current image and a second reference image for the current image, and wherein the determination of the fraction is independent of actual temporal distance positions of the respective reference images; and
for motion compensation for a direct mode macroblock in the current image, with the computing device that implements the video decoder, processing the fraction along with a motion vector for a co-located macroblock in the first reference image, wherein the motion vector represents motion in the first reference image relative to the second reference image, and wherein the processing the fraction along with the motion vector results in a representation of motion for the direct mode macroblock in the current image relative to the first reference image and relative to the second reference image.

### Claim 2 (depends on 1)

The method of claim 1 wherein the fraction is represented by the code, and wherein the code comprises a variable length code in the bit stream.

### Claim 3 (depends on 1)

The method of claim 1 wherein the fraction is selected from a set of discrete values, wherein each of the values is greater than zero and less than one so as to indicate the estimated temporal distance position within the interval.

### Claim 4 (depends on 1)

The method of claim 1 wherein the fraction is selected from the group consisting of: ½, ⅓, ⅔, ¼, ¾, ⅕, ⅖, ⅗, ⅘, ⅙, ⅚, 1/7, 2/7, and 3/7.

### Claim 5 (depends on 1)

The method of claim 1 wherein the estimated temporal distance position for the current image relative to the interval between the first reference image for the current image and the second reference image for the current image is not the true temporal distance position of the current image.

### Claim 6 (depends on 1)

The method of claim 1 wherein the fraction is based on motion information for the sequence of video images.

### Claim 7 (depends on 1)

The method of claim 1 wherein the fraction is based on a proximity of the current image to an end of the sequence of video images.

### Claim 8 (depends on 1)

The method of claim 1 further comprising, with the computing device that implements the video decoder, repeating the acts of claim 1 for each of plural bi-directionally predicted images in the sequence of video images.

### Claim 9 (depends on 2)

The method of claim 2 wherein the determination of the fraction comprises looking up the variable length code in a variable length code table to obtain a value for the fraction.

### Claim 10 (depends on 9)

The method of claim 9 wherein at least one entry in the variable length code table represents a frame type, and wherein the frame type is B/I-frame.

### Claim 11 (depends on 1)

The method of claim 1 wherein the processing the fraction along with the motion vector comprises scaling the motion vector for the co-located macroblock using the fraction.

### Claim 12 (depends on 11)

The method of claim 11 wherein the scaling the motion vector for the co-located macroblock comprises scaling a vertical component and a horizontal component of the motion vector for the co-located macroblock.

### Claim 13 (depends on 11)

The method of claim 11 wherein the scaling the motion vector for the co-located macroblock comprises:
scaling the motion vector for the co-located macroblock by a factor of the fraction, to obtain an implied forward motion vector for the direct mode macroblock; and
scaling the motion vector for the co-located macroblock by a factor of the fraction minus one, to obtain an implied backward motion vector for the direct mode macroblock.

### Claim 14 (depends on 13)

The method of claim 13 wherein the motion compensation comprises:
addressing a macroblock in the future reference image using the implied forward motion vector;
addressing a macroblock in the previous reference image using the implied backward motion vector; and
predicting the current macroblock using an average of the macroblock in the future reference image and the macroblock in the previous reference image.

### Claim 15

In a computing device that implements a video encoder, the computing device including a processor and memory, a method of encoding images in a sequence of video images, the method comprising:
with the computing device that implements the video encoder, determining a fraction for a current image in the sequence, wherein the current image has a previous reference image and a future reference image, and wherein the fraction represents a temporal position for the current image relative to the respective reference images;
with the computing device that implements the video encoder, selecting direct mode prediction for a current direct mode macroblock in the current image;
with the computing device that implements the video encoder, finding a motion vector for a co-located macroblock in the future reference image;
with the computing device that implements the video encoder, scaling the motion vector for the co-located macroblock using the fraction;
with the computing device that implements the video encoder, using results of the scaling in motion compensation for the current direct mode macroblock in the current image; and
with the computing device that implements the video encoder, outputting a code in a bit stream, wherein the code represents the fraction, and wherein the outputting the code facilitates determination of the fraction independent of actual temporal positions of the respective reference images during decoding.

### Claim 16 (depends on 15)

The method of claim 15 wherein the fraction facilitates representation of variable velocity motion in the direct mode prediction.

### Claim 17 (depends on 15)

The method of claim 15 wherein the scaling the motion vector for the co-located macroblock comprises scaling a vertical component and a horizontal component of the motion vector for the co-located macroblock.

### Claim 18 (depends on 15)

The method of claim 15 wherein the scaling the motion vector for the co-located macroblock comprises:
scaling the motion vector for the co-located macroblock by a factor of the fraction, to obtain an implied forward motion vector for the current direct mode macroblock; and
scaling the motion vector for the co-located macroblock by a factor of the fraction minus one, to obtain an implied backward motion vector for the current direct mode macroblock, wherein for the direct mode macroblock the motion compensation uses the implied forward motion vector and the implied backward motion vector.

### Claim 19

A system comprising:
one or more processors;
memory;
at least one input device, output device or communication connection; and
one or more storage media having stored thereon computer-executable instructions for causing one or more computers to perform a method of decoding images in a sequence of video images, the method comprising:
receiving and decoding a code in a bit stream to determine a fraction for a current image in the sequence, wherein the fraction represents an estimated temporal distance position for the current image relative to an interval between a first reference image for the current image and a second reference image for the current image, and wherein the determination of the fraction is independent of actual temporal distance positions of the respective reference images; and
for motion compensation for a direct mode macroblock in the current image, processing the fraction along with a motion vector for a co-located macroblock in the first reference image, wherein the motion vector represents motion in the first reference image relative to the second reference image, and wherein the processing the fraction along with the motion vector results in a representation of motion for the direct mode macroblock in the current image relative to the first reference image and relative to the second reference image.

### Claim 20 (depends on 19)

The system of claim 19 wherein the fraction is represented by the code, and wherein the code comprises a variable length code in the bit stream.

### Claim 21 (depends on 19)

The system of claim 19 wherein the fraction is selected from a set of discrete values, wherein each of the values is greater than zero and less than one so as to indicate the estimated temporal distance position within the interval.

### Claim 22 (depends on 19)

The system of claim 19 wherein the fraction is selected from the group consisting of: ½, ⅓, ⅔, ¼, ¾, ⅕, ⅖, ⅗, ⅘, ⅙, ⅚, 1/7, 2/7, and 3/7.

### Claim 23 (depends on 19)

The system of claim 19 wherein the estimated temporal distance position for the current image relative to the interval between the first reference image for the current image and the second reference image for the current image is not the true temporal distance position of the current image.

### Claim 24 (depends on 19)

The system of claim 19 wherein the fraction is based on motion information for the sequence of video images.

### Claim 25 (depends on 19)

The system of claim 19 wherein the fraction is based on a proximity of the current image to an end of the sequence of video images.

### Claim 26 (depends on 19)

The system of claim 19 wherein the determination of the fraction comprises looking up the code in a code table to obtain a value for the fraction.

### Claim 27 (depends on 26)

The system of claim 26 wherein at least one entry in the code table represents a frame type, and wherein the frame type is B/I-frame.

### Claim 28

In a computing device that implements a video encoder, the computing device including a processor and memory, a method of encoding images in a sequence of video images, the method comprising:
with the computing device that implements the video encoder, determining a fraction for a current image in the sequence, wherein the fraction represents an estimated temporal distance position for the current image relative to an interval between a first reference image for the current image and a second reference image for the current image, and wherein the determination of the fraction is independent of actual temporal distance positions of the respective reference images;
for motion compensation for a direct mode macroblock in the current image, with the computing device that implements the video encoder, processing the fraction along with a motion vector for a co-located macroblock in the first reference image, wherein the motion vector represents motion in the first reference image relative to the second reference image, and wherein the processing the fraction along with the motion vector results in a representation of motion for the direct mode macroblock in the current image relative to the first reference image and relative to the second reference image; and
with the computing device that implements the video encoder, outputting a code in a bit stream, wherein the code represents the fraction.

### Claim 29 (depends on 28)

The method of claim 28 wherein the code comprises a variable length code in the bit stream.

### Claim 30 (depends on 28)

The method of claim 28 wherein the fraction is selected from a set of discrete values, wherein each of the values is greater than zero and less than one so as to indicate the estimated temporal distance position within the interval.

### Claim 31 (depends on 28)

The method of claim 28 wherein the fraction is selected from the group consisting of: ½, ⅓, ⅔, ¼, ¾, ⅕, ⅖, ⅗, ⅘, ⅙, ⅚, 1/7, 2/7, and 3/7.

### Claim 32 (depends on 28)

The method of claim 28 wherein the estimated temporal distance position for the current image relative to the interval between the first reference image for the current image and the second reference image for the current image is not the true temporal distance position of the current image.

### Claim 33 (depends on 28)

The method of claim 28 wherein the fraction is based on motion information for the sequence of video images.

### Claim 34 (depends on 28)

The method of claim 28 wherein the fraction is based on a proximity of the current image to an end of the sequence of video images.

### Claim 35 (depends on 28)

The method of claim 28 wherein the determination of the fraction comprises looking up the code in a code table to obtain a value for the fraction.

### Claim 36 (depends on 35)

The method of claim 35 wherein at least one entry in the code table represents a frame type, and wherein the frame type is B/I-frame.

### Claim 37 (depends on 28)

The method of claim 28 wherein the determining the fraction comprises:
evaluating each of plural fractions to determine bit costs for encoding the current image using the respective fractions; and
selecting the fraction based on the evaluating.

### Claim 38

A system comprising:
one or more processors;
memory;
at least one input device, output device or communication connection; and
one or more storage media having stored thereon computer-executable instructions for causing one or more computers to perform a method of encoding images in a sequence of video images, the method comprising:
determining a fraction for a current image in the sequence, wherein the fraction represents an estimated temporal distance position for the current image relative to an interval between a first reference image for the current image and a second reference image for the current image, and wherein the determination of the fraction is independent of actual temporal distance positions of the respective reference images;
for motion compensation for a direct mode macroblock in the current image, processing the fraction along with a motion vector for a co-located macroblock in the first reference image, wherein the motion vector represents motion in the first reference image relative to the second reference image, and wherein the processing the fraction along with the motion vector results in a representation of motion for the direct mode macroblock in the current image relative to the first reference image and relative to the second reference image; and
outputting a code in a bit stream, wherein the code represents the fraction.

### Claim 39 (depends on 38)

The system of claim 38 wherein the code comprises a variable length code in the bit stream.

### Claim 40 (depends on 38)

The system of claim 38 wherein the fraction is selected from a set of discrete values, wherein each of the values is greater than zero and less than one so as to indicate the estimated temporal distance position within the interval.

### Claim 41 (depends on 38)

The system of claim 38 wherein the fraction is selected from the group consisting of: ½, ⅓, ⅔, ¼, ¾, ⅕, ⅖, ⅗, ⅘, ⅙, ⅚, 1/7, 2/7, and 3/7.

### Claim 42 (depends on 38)

The system of claim 38 wherein the estimated temporal distance position for the current image relative to the interval between the first reference image for the current image and the second reference image for the current image is not the true temporal distance position of the current image.

### Claim 43 (depends on 38)

The system of claim 38 wherein the fraction is based on motion information for the sequence of video images.

### Claim 44 (depends on 38)

The system of claim 38 wherein the fraction is based on a proximity of the current image to an end of the sequence of video images.

### Claim 45 (depends on 38)

The system of claim 38 wherein the determination of the fraction comprises looking up the code in a code table to obtain a value for the fraction.

### Claim 46 (depends on 45)

The system of claim 45 wherein at least one entry in the code table represents a frame type, and wherein the frame type is B/I-frame.

### Claim 47 (depends on 38)

The system of claim 38 wherein the determining the fraction comprises:
evaluating each of plural fractions to determine bit costs for encoding the current image using the respective fractions; and
selecting the fraction based on the evaluating.

### Claim 48

A system comprising:
one or more processors;
memory;
at least one input device, output device or communication connection; and
one or more storage media having stored thereon computer-executable instructions for causing one or more computers to perform a method of encoding images in a sequence of video images, the method comprising:
determining a fraction for a current image in the sequence, wherein the current image has a previous reference image and a future reference image, and wherein the fraction represents a temporal position for the current image relative to the respective reference images;
selecting direct mode prediction for a current direct mode macroblock in the current image;
finding a motion vector for a co-located macroblock in the future reference image;
scaling the motion vector for the co-located macroblock using the fraction;
using results of the scaling in motion compensation for the current direct mode macroblock in the current image; and
outputting a code in a bit stream, wherein the code represents the fraction, and wherein the outputting the code facilitates determination of the fraction independent of actual temporal positions of the respective reference images during decoding.

### Claim 49 (depends on 48)

The system of claim 48 wherein the fraction facilitates representation of variable velocity motion in the direct mode prediction.

### Claim 50 (depends on 48)

The system of claim 48 wherein the scaling the motion vector for the co-located macroblock comprises scaling a vertical component and a horizontal component of the motion vector for the co-located macroblock.

### Claim 51 (depends on 48)

The system of claim 48 wherein the scaling the motion vector for the co-located macroblock comprises:
scaling the motion vector for the co-located macroblock by a factor of the fraction, to obtain an implied forward motion vector for the current direct mode macroblock; and
scaling the motion vector for the co-located macroblock by a factor of the fraction minus one, to obtain an implied backward motion vector for the current direct mode macroblock, wherein for the current direct mode macroblock the motion compensation uses the implied forward motion vector and the implied backward motion vector.