# US 6,731,818 B1 · System and method for generating video frames

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 6,731,818 B1
- Document ID: US6731818B1
- Status: Granted
- Kind: B1
- Application number: 09/345,686
- Filed: 1999-06-30
- Granted: 2004-05-04
- Earliest claimed: 1999-06-30
- Assignees: RealNetworks, Inc.
- Inventors: Greg Conklin
- Web page: https://patentgrep.com/patent/US6731818B1

## CPC

- H04N 19/503: involving temporal prediction (adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109)
- H04N 19/577: Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
- H04N 19/587: involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

System and method for generating video frames. The system includes a frame generator which generates one or more intermediate frames based upon one base frames. Each of the base frames are comprised of a plurality of macroblocks. Furthermore, one or more of the macroblocks have a motion vector. The macroblocks are comprised of a plurality of pixels. In the frame generation process, the frame generator performs a number of steps such as: (i) determines whether frame generation is appropriate, (ii) examines the first and second base frames to check for the presence of textual characters, (iii) selects a frame generation method based upon information in the first and second frames, (iv) filters the generated frames. In one embodiment, the system includes a server computer having an encoder, a client computer having a decoder, and a network connecting the server computer to the client computer. In this embodiment, the frame generator resides and executes within the client computer and receives the base frames from the decoder.

## Claims

### Claim 1

A method of generating video frames, the method comprising:
receiving a first frame in a memory in an electronic device;
receiving a second frame in the memory in the electronic device;
determining whether to generate an intermediate frame between the first frame and the second frame, wherein determining whether to generate the intermediate frame includes determining whether the second frame is a key frame; and
generating, depending on the step of determining, at least one intermediate frame based on elements of the first frame and/or the second frame.

### Claim 2 (depends on 1)

The method of claim 1, wherein each of the elements are macroblocks having a plurality of rows and columns of pixels, each of the pixels having an associated intensity value.

### Claim 3 (depends on 1)

The method of claim 1, additionally comprising the steps of:
receiving time interval data indicative of the presentation timing of the first frame and the second frame; and
determining whether the difference in time between the first frame and the second frame is greater than a threshold.

### Claim 4 (depends on 3)

The method of claim 3, wherein the magnitude of the threshold is based, at least in part, on the value of a quantizer associated with the first frame and/or the second frame.

### Claim 5

A system for generating video frames, the system comprising:
means for receiving a first frame in a memory in the computer system;
means for receiving a second frame in the memory in the computer system;
means for determining whether to generate an intermediate frame between the first frame and the second frame, wherein determining whether to generate the intermediate frame includes determining whether the second frame is a key frame; and
means for generating, depending on the step of determining, at least one intermediate frame based upon elements of the first frame and the second frame.

### Claim 6 (depends on 5)

The system of claim 5, wherein each of the elements is a macroblock having a plurality of rows and columns of pixels, each of the pixels having an associated intensity value.

### Claim 7 (depends on 5)

The system of claim 5, wherein the magnitude of the threshold is based, at least in part, on the value of a quantizer associated with the first frame and/or the second frame.

### Claim 8

In a computer system, a method of generating video frames, the method comprising:
receiving a first frame having a plurality of macroblocks in a memory in the computer system, the first frame representative of a video image at a first instance in time, each macroblock having a plurality of rows and columns of pixels, each of the pixels having an associated intensity value;
receiving a second frame having a plurality of macroblocks in the memory in the computer system, the second frame representative of a video image at a second instance in time, each macroblock having a plurality of rows and columns of pixels, each of the pixels having an associated intensity value; and
determining whether to generate an intermediate frame between the first frame and the second frame, the determining comprising the steps of:
receiving time interval data indicative of the presentation timing of the first frame and the second frame, and
determining whether the difference in time between the first frame and the second frame is greater than a threshold wherein the magnitude of the threshold is based, at least in part, on the value of a quantizer; and
generating, depending on the step of determining, at least one intermediate frame based upon the macroblocks of the first frame and the second frame.

### Claim 9

A system for generating video frames, the system comprising:
a frame analysis module for receiving frames; and
a frame synthesis module for determining whether the generation of one or more frames between two received frames provides for a smoother visual presentation than without the frame generation, wherein the determining is based at least in part upon whether one of the received frames is a key frame.

### Claim 10 (depends on 9)

The system of claim 9, wherein the frame synthesis module generates one or more frames between two received frames.

### Claim 11

In a computer system, a method of generating video frames, the method comprising:
receiving a first frame in a memory in the computer system, the first frame comprising a first set of elements;
receiving a second frame in the memory in the computer system, the second frame comprising a second set of elements;
selecting one of a plurality of frame generation methods based upon information in the first frame and/or second frame, wherein one of the selectable frame generation methods includes positioning an element for inclusion in a generated frame based upon a motion vector that is associated with the first or second frames; and
applying the selected frame generation method to generate at least one intermediate frame based upon the first set of elements and/or the second set of elements.

### Claim 12 (depends on 11)

The method of claim 11, wherein the frame generation method is selected from the group comprising: motion compensated interpolation and linear interpolation.

### Claim 13 (depends on 12)

The method of claim 12, wherein one or more of the first set of elements and the second set of elements has an associated motion vector.

### Claim 14 (depends on 13)

The method of claim 13, wherein selecting a frame generation method includes:
determining whether, any of the motion vectors exceeds a threshold; and
selecting motion compensated interpolation as the method for frame generation if any of the motion vectors exceeds the threshold.

### Claim 15 (depends on 13)

The method of claim 13, wherein selecting a frame generation method comprises the steps of:
for a group of the one or more elements of the first and second set of elements, identifying with respect to each member element in the group another proximately positioned element in the group;
determining the average distance between the motion vector associated with the respective member element and the motion vector associated with said proximately positioned element;
summing the average distances; and
selecting motion compensated interpolation if the summed total is less than a threshold.

### Claim 16

A system for generating video frames, the system comprising:
means for receiving a first frame in a memory in the computer system, the first frame comprising a first set of elements;
means for receiving a second frame in the memory in the computer system, the second frame comprising a second set of elements;
means for selecting one of a plurality of frame generation methods based upon the first frame and/or second frame, wherein one of the selectable frame generation methods includes positioning an element for inclusion in a generated frame based upon a motion vector that is associated with the first or second frames; and
means for applying the selected frame generation method to generate at least one intermediate frame based upon the first set of elements and/or the second set of elements.

### Claim 17 (depends on 16)

The method of claim 16, wherein the frame generation method is selected from the group comprising: motion compensated interpolation and linear interpolation.

### Claim 18 (depends on 16)

The system of claim 16, wherein one or more of the first set of elements and the second set of elements has an associated motion vector.

### Claim 19 (depends on 18)

The system of claim 18, wherein the means for selecting a frame generation method includes:
means for determining whether any motion vector exceeds a threshold; and
means for selecting motion compensated interpolation as the method for frame generation if any of the motion vectors exceeds the threshold.

### Claim 20

In a computer system, a method of generating video frames, the method comprising:
receiving a first frame in a memory in the computer system;
receiving a second frame in the memory in the computer system;
determining whether to generate an intermediate frame between the first frame and the second frame, wherein determining whether to generate the intermediate frame includes:
receiving time interval data indicative of the presentation timing of the first frame and the second frame, and
determining whether the difference in time between the first frame and the second frame is greater than a threshold; and
generating, depending on the step of determining, at least one intermediate frame based on elements of the first frame and/or the second frame.

### Claim 21 (depends on 20)

The method of claim 20, wherein the magnitude of the threshold is based, at least in part, on the value of a quantizer associated with the first frame and/or the second frame.

### Claim 22 (depends on 20)

The method of claim 20, wherein each of the elements of the second frame is either independently coded or has an associated motion vector, and wherein determining whether to generate the intermediate frame includes determining whether the number of independently coded elements of the second frame exceeds a predefined threshold.

### Claim 23

In a computer system, a method of generating video frames, the method comprising:
receiving a first frame in a memory in the computer system;
receiving a second frame in the memory in the computer system, wherein each of the elements of the second frame is either independently coded or has an associated motion vector;
determining whether to generate an intermediate frame between the first frame and the second frame and whether the number of independently coded elements of the second frame exceeds a predefined threshold; and
generating, depending on the step of determining, at least one intermediate frame based on elements of the first frame and/or the second frame.

### Claim 24 (depends on 23)

The method of claim 23, wherein the magnitude of the threshold is based, at least in part, on the value of a quantizer associated with the first frame and/or the second frame.

### Claim 25 (depends on 23)

The method of claim 23, wherein each of the elements are macroblocks having a plurality of rows and columns of pixels, each of the pixels having an associated intensity value.

### Claim 26

In a computer system, a method of generating video frames, the method comprising:
receiving a first frame in a memory in the computer system;
receiving a second frame in the memory it the computer system, wherein each of the elements of the second frame is either independently coded or has an associated motion vector;
determining whether to generate an intermediate frame between the first frame and the second frame and whether the maximum motion vector in the second frame exceeds a predefined threshold; and
generating, depending on the step of determining, at least one intermediate frame based on elements of the first frame and/or the second frame.

### Claim 27 (depends on 26)

The method of claim 26, wherein the magnitude of the threshold is based, at least in part, on the value of a quantizer associated with the first frame and/or the second frame.

### Claim 28 (depends on 26)

The method of claim 26, wherein each of the elements is a macroblock having a plurality of rows and columns of pixels, each of the pixels having an associated intensity value.

### Claim 29

A system for generating video frames, the system comprising:
means for receiving a first frame in a memory in the computer system;
means for receiving a second frame in the memory in the computer system;
means for determining whether to generate an intermediate frame between the first frame and the second frame, the means for determining comprising:
means for receiving time interval data indicative of the presentation timing of the first frame and the second frame, and
means for determining whether to generate the intermediate frame includes determining whether the difference in time between the first frame and the second frame is greater than a threshold; and
means for generating, depending on the step of determining, at least one intermediate frame based upon elements of the first frame and the second frame.

### Claim 30 (depends on 29)

The system of claim 29, wherein each of the elements is a macroblock having a plurality of rows and columns of pixels, each of the pixels having an associated intensity value.

### Claim 31 (depends on 29)

The system of claim 29, wherein the magnitude of the threshold is based, at least in part, on the value of a quantizer associated with the first frame and/or the second frame.

### Claim 32 (depends on 29)

The system of claim 29, wherein each of the elements of the second frame are either independently coded or have an associated motion vector, and wherein determining whether to generate the intermediate frame includes determining whether the number of independently coded elements of the second frame exceeds a predefined threshold.

### Claim 33 (depends on 29)

The system of claim 29, wherein each of the elements of the second frame are either independently coded or have an associated motion vector, and wherein determining whether to generate the intermediate frame includes determining whether the maximum motion vector in the second frame exceeds a predefined threshold.

### Claim 34

A system for generating video frames, the system comprising:
means for receiving a first frame in a memory in the computer system;
means for receiving a second frame in the memory in the computer system;
means for determining whether to generate an intermediate frame between the first frame and the second frame, wherein each of the elements of the second frame are either independently coded or have an associated notion vector, and wherein determining whether to generate the intermediate frame includes determining whether the number of independently coded elements of the second frame exceeds a predefined threshold; and
means for generating, depending on the step of determining, at least one intermediate frame based upon elements of the first frame and the second frame.

### Claim 35 (depends on 34)

The system of claim 34, wherein each of the elements is a macroblock having a plurality of rows and columns of pixels, each of the pixels having an associated intensity value.

### Claim 36 (depends on 34)

The system of claim 34, wherein the magnitude of the threshold is based, at least in part, on the value of a quantizer associated with the first frame and/or the second frame.

### Claim 37 (depends on 34)

The system of claim 34, wherein each of the elements of the second frame are either independently coded or have an associated motion vector, and wherein determining whether to generate the intermediate frame includes determining whether the number of independently coded elements of the second frame exceeds a predefined threshold.

### Claim 38

A system for generating video frames, the system comprising:
means for receiving a first frame in a memory in the computer system;
means for receiving a second frame in the memory in the computer system, wherein each of the elements of the second frame are either independently coded or have an associated motion vector;
means for determining whether to generate an intermediate frame between the first frame and the second frame, wherein determining whether to generate the intermediate frame includes determining whether the maximum motion vector in the second frame exceeds a predefined threshold; and
means for generating, depending on the step of determining, at least one intermediate frame based upon elements of the first frame and the second frame.

### Claim 39 (depends on 38)

The system of claim 38, wherein each of the elements is a macroblock having a plurality of rows and columns of pixels, each of the pixels having an associated intensity value.

### Claim 40 (depends on 38)

The system of claim 38, wherein the magnitude of the threshold is based, at least in part, on the value of a quantizer associated with the first frame and/or the second frame.

### Claim 41

A program storage device storing instructions that when executed:
receiving a first frame in a memory in the computer system;
receiving a second frame in the memory in the computer system, wherein each of the elements of the second frame is either independently coded or has an associated motion vector;
determining whether to generate an intermediate frame between the first frame and the second frame by determining whether the maximum motion vector in the second frame exceeds a predefined threshold; and
generating, depending on the step of determining whether to generate an intermediate frame, at least one intermediate frame based on elements of the first frame and/or the second frame.

### Claim 42 (depends on 41)

The program storage device of claim 41, wherein the magnitude of the threshold is based, at least in part, on the value of a quantizer associated with the first frame and/or the second frame.

### Claim 43 (depends on 41)

The program storage device of claim 41, wherein each of the elements are macroblocks having a plurality of rows and columns of pixels, each of the pixels having an associated intensity value.

### Claim 44

A method comprising:
receiving a first frame in a memory in a computing device;
receiving a second frame in the memory in the computer system, wherein each of the elements of the second frame is either independently coded or has an associated motion vector;
determining whether to generate an intermediate frame between the first frame and the second frame, independent of a frame rate of the first and second frames, wherein the determining is based upon the magnitude of a threshold that is set as a function of, at least in part, a quantizer value that is related to the first frame and/or the second frame; and
generating, depending on the step of determining, at least one intermediate frame based on elements of the first frame and/or the second frame.

### Claim 45 (depends on 44)

The method of claim 44, wherein each of the elements are macroblocks having a plurality of rows and columns of pixels, each of the pixels having an associated intensity value.

### Claim 46

A method comprising:
receiving a first frame in a memory in a computing device;
receiving a second frame in the memory in the computer device, wherein each of the elements of the second frame is either independently coded or has an associated motion vector;
determining whether to generate an intermediate frame between the first frame and the second frame based upon: a) the content of one of the elements in the first or second frame and b) the magnitude of a threshold that is set as a function of, at least in part, a quantizer value that is related to the first frame and/or the second frame; and
generating, depending on the step of determining, at least one intermediate frame based on elements of the first frame and/or the second frame.

### Claim 47 (depends on 46)

The method of claim 46, wherein each of the elements are macroblocks having a plurality of rows and columns of pixels, each of the pixels having an associated intensity value.

### Claim 48

In a computer system, a method of generating video frames, the method comprising:
receiving a first frame in a memory in the computer system, the first frame comprising a first set of elements;
receiving a second frame in the memory in the computer system, the second frame comprising a second set of elements, wherein one or more of the first set of elements and the second set of elements has an associated motion vector;
selecting a frame generation method based upon information in the first frame and/or second frame; and
applying the selected frame generation method to generate at least one intermediate frame based upon the first set of elements and/or the second set of elements, wherein selecting a frame generation method includes:
for a group of the one or more elements of the first and second set of elements, identifying with respect to each member element in the group another proximately positioned element in the group;
determining the average distance between the motion vector associated with the respective member element and the motion vector associated with said proximately positioned element;
summing the average distances; and
selecting motion compensated interpolation if the summed total is less than a threshold.

### Claim 49 (depends on 48)

The method of claim 48, wherein the element comprises a macroblock.
