# US 8,279,946 B2 · System and method for providing a variable frame rate and adaptive frame skipping on a mobile device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 8,279,946 B2
- Document ID: US8279946B2
- Status: Granted
- Kind: B2
- Application number: 11/944,549
- Filed: 2007-11-23
- Published: 2009-05-28
- Granted: 2012-10-02
- Earliest claimed: 2007-11-23
- Assignees: Research In Motion Limited
- Inventors: David Mak-Fan; Aaron B. Small; Thomas C. Nagy
- Web page: https://patentgrep.com/patent/US8279946B2

## CPC

- H04N 19/115: Selection of the code volume for a coding unit prior to coding
- H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
- H04N 19/156: Availability of hardware or computational resources, e.g. encoding based on power-saving criteria
- H04N 19/159: Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
- H04N 19/172: the region being a picture, frame or field
- H04N 19/44: Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
- H04N 19/587: involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
- H04N 19/61: in combination with predictive coding
- H04N 21/41407: {embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop (constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57)}
- H04N 21/4341: {Demultiplexing of audio and video streams}
- H04N 21/440281: {by altering the temporal resolution, e.g. by frame skipping (television signal recording using magnetic recording on tape for reproducing at a rate different from the recording rate H04N5/783)}
- H04N 21/4424: {Monitoring of the internal components or processes of the client device, e.g. CPU or memory load, processing speed, timer, counter or percentage of the hard disk space used (error monitoring in general G06F11/30; arrangements for monitoring conditions of receiving stations in broadcast systems H04H60/32; diagnosis, testing or measuring for television receivers H04N17/04)}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A processor and method are provided for decoding a multimedia file having video and audio data streams that are configured to be played in synchronization. Frames of the video stream are decoded and played with the audio and, to compensate for saturation of the processor usage, two procedures are performed. The first procedure operates at a first periodic interval and slows down the frame rate to reduce processor usage if needed. The second procedure first attempts to speed up the video to catch up to the audio if they are out of sync and if this cannot be done in the next time interval, the decoding of a select number of frames is skipped such that the video and audio data streams are resynchronized.

## Claims

### Claim 1

A method of decoding a multimedia file using a processor according to usage of said processor, said multimedia file comprising a video data stream including a series of encoded frames and an audio data stream to be played in synchronization with said video data stream as said frames are decoded, said method comprising:
at a first periodic interval:
determining if a target frame rate for said video data stream can be achieved while meeting a predetermined usage threshold for said processor;
if said usage threshold is not met:
determining a first scaling factor required to meet said usage threshold;
modifying said target frame rate according to said first scaling factor to determine a current frame rate; and
decoding subsequent frames at a first modified frame rate while playing said audio data stream therewith;
if said usage threshold is met:
decoding said frames of said video data stream and playing said video data stream at said target frame rate while playing said audio data stream therewith; and
at a second periodic interval which is longer than said first periodic interval:
determining if said video data stream and said audio data stream are out of synchronization;
if said video data stream and said audio data stream are out of synchronization, determining a second scaling factor to enable said video data stream to resynchronize with said audio data stream;
applying said second scaling factor to a current frame rate to determine a second modified frame rate; and
if said second modified frame rate exceeds a capped frame rate, decoding subsequent frames at said capped frame rate and not decoding one or more frames scheduled to be decoded within a next time interval.

### Claim 2 (depends on 1)

The method according to claim 1 wherein if said second modified frame rate does not exceed said capped frame rate, decoding subsequent frames at said second modified frame rate until said video data stream is resynchronized with said audio data stream.

### Claim 3 (depends on 1)

The method according to claim 1 wherein decoding subsequent frames at said capped frame rate and not decoding one or more frames comprises determining a difference between said second modified frame rate and said capped frame rate, determining a number of frames to not be decoded to compensate for said difference, and not decoding said number of frames.

### Claim 4 (depends on 1)

The method according to claim 1, said frames being decoded in groups of I, B and P frames, wherein a counter is incremented after each frame is decoded until a next I frame is encountered and decoding subsequent frames at said capped frame rate and not decoding one or more frames comprises not decoding all frames until said next I frame and continuing decoding from said next I frame.

### Claim 5 (depends on 4)

The method according to claim 4 comprising recording a timestamp for each I frame.

### Claim 6 (depends on 1)

The method according to claim 1 comprising tracking the number of frames not decoded for said determining if said target frame rate can be achieved.

### Claim 7 (depends on 1)

The method according to claim 1 wherein said determining if said target frame rate can be achieved comprises consideration of an average scaling count of a predetermined number of previous frames.

### Claim 8 (depends on 7)

The method according to claim 7 comprising updating said scaling count according to each application of said first scaling factor.

### Claim 9 (depends on 7)

The method according to claim 7 wherein said determining if said target frame rate can be achieved comprises determining how said processor usage dedicated to said decoding should be scaled to maintain a predetermined amount of processor usage for other applications and idle processes according to said average scaling count.

### Claim 10

A non-transitory computer readable medium containing computer executable instructions, which when executed by a processor, enable a device to:
at a first periodic interval:
determine if a target frame rate for said video data stream can be achieved while meeting a predetermined usage threshold for said processor;
if said usage threshold is not met:
determine a first scaling factor required to meet said usage threshold; modify said target frame rate according to said first scaling factor to determine a current frame rate; and
decode subsequent frames at a first modified frame rate while playing said audio data stream therewith;
if said usage threshold is met:
decode said frames of said video data stream and playing said video data stream at said target frame rate while playing said audio data stream therewith; and
at a second periodic interval which is longer than said first periodic interval:
determine if said video data stream and said audio data stream are out of synchronization;
if said video data stream and said audio data stream are out of synchronization, determine a second scaling factor to enable said video data stream to resynchronize with said audio data stream;
apply said second scaling factor to a current frame rate to determine a second modified frame rate; and
if said second modified frame rate exceeds a capped frame rate, decode subsequent frames at said capped frame rate and not decoding one or more frames scheduled to be decoded within a next time interval.

### Claim 11 (depends on 10)

The non-transitory computer readable medium according to claim 10 wherein if said second modified frame rate does not exceed said capped frame rate, decoding subsequent frames at said second modified frame rate until said video data stream is resynchronized with said audio data stream.

### Claim 12 (depends on 10)

The non-transitory computer readable medium according to claim 10 wherein decoding subsequent frames at said capped frame rate and not decoding one or more frames comprises determining a difference between said second modified frame rate and said capped frame rate, determining a number of frames to not be decoded to compensate for said difference, and not decoding said number of frames.

### Claim 13 (depends on 10)

The non-transitory computer readable medium according to claim 10, said frames being decoded in groups of I, B and P frames, wherein a counter is incremented after each frame is decoded until a next I frame is encountered and decoding subsequent frames at said capped frame rate and not decoding one or more frames comprises not decoding all frames until said next I frame and continuing decoding from said next I frame.

### Claim 14 (depends on 13)

The non-transitory computer readable medium according to claim 13 comprising recording a timestamp for each I frame.

### Claim 15 (depends on 10)

The non-transitory computer readable medium according to claim 10 comprising tracking the number of frames not decoded for said determining if said target frame rate can be achieved.

### Claim 16 (depends on 10)

The non-transitory computer readable medium according to claim 10 wherein said determining if said target frame rate can be achieved comprises consideration of an average scaling count of a predetermined number of previous frames.

### Claim 17 (depends on 16)

The non-transitory computer readable medium according to claim 16 comprising updating said scaling count according to each application of said first scaling factor.

### Claim 18 (depends on 16)

The non-transitory computer readable medium according to claim 16 wherein said determining if said target frame rate can be achieved comprises determining how said processor usage dedicated to said decoding should be scaled to maintain a predetermined amount of processor usage for other applications and idle processes according to said average scaling count.

### Claim 19

A processor for decoding a multimedia file according to usage of said processor, said multimedia file comprising a video data stream including a series of encoded frames and an audio data stream to be played in synchronization with said video data as said frames are decoded, said processor being configured for:
at a first periodic interval:
determining if a target frame rate for said video data stream can be achieved while meeting a predetermined usage threshold for said processor;
if said usage threshold is not met:
determining a first scaling factor required to meet said usage threshold;
modifying said target frame rate according to said first scaling factor to determine a current frame rate; and
decoding subsequent frames at a first modified frame rate while playing said audio data stream therewith;
if said usage threshold is met:
decoding said frames of said video data stream and playing said video data stream at said target frame rate while playing said audio data stream therewith; and
at a second periodic interval which is longer than said first periodic interval:
determining if said video data stream and said audio data stream are out of synchronization;
if said video data stream and said audio data stream are out of synchronization, determining a second scaling factor to enable said video data stream to resynchronize with said audio data stream;
applying said second scaling factor to a current frame rate to determine a second modified frame rate; and
if said second modified frame rate exceeds a capped frame rate, decoding subsequent frames at said capped frame rate and not decoding one or more frames scheduled to be decoded within a next time interval.

### Claim 20 (depends on 19)

The processor according to claim 19 wherein if said second modified frame rate does not exceed said capped frame rate, decoding subsequent frames at said second modified frame rate until said video data stream is resynchronized with said audio data stream.

### Claim 21 (depends on 19)

The processor according to claim 19 wherein decoding subsequent frames at said capped frame rate and not decoding one or more frames comprises determining a difference between said second modified frame rate and said capped frame rate, determining a number of frames to not be decoded to compensate for said difference, and not decoding said number of frames.

### Claim 22 (depends on 19)

The processor according to claim 19, said frames being decoded in groups of I, B and P frames, wherein a counter is incremented after each frame is decoded until a next I frame is encountered and decoding subsequent frames at said capped frame rate and not decoding one or more frames comprises not decoding all frames until said next I frame and continuing decoding from said next I frame.

### Claim 23 (depends on 22)

The processor according to claim 22 further configured for recording a timestamp for each I frame.

### Claim 24 (depends on 19)

The processor according to claim 19 further configured for tracking the number of frames not decoded for said determining if said target frame rate can be achieved.

### Claim 25 (depends on 19)

The processor according to claim 19 wherein said determining if said target frame rate can be achieved comprises consideration of an average scaling count of a predetermined number of previous frames.

### Claim 26 (depends on 25)

The processor according to claim 25 further configured for updating said scaling count according to each application of said first scaling factor.

### Claim 27 (depends on 25)

The processor according to claim 25 wherein said determining if said target frame rate can be achieved comprises determining how said processor usage dedicated to said decoding should be scaled to maintain a predetermined amount of processor usage for other applications and idle processes according to said average scaling count.

### Claim 28 (depends on 19)

A mobile device comprising the processor according to claim 19.
