# US 2008/0259799 A1 · Packet Scheduling with Quality-Aware Frame Dropping for Video Streaming

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2008/0259799 A1
- Document ID: US20080259799A1
- Status: Published application
- Kind: A1
- Application number: 11/738,313
- Filed: 2007-04-20
- Published: 2008-10-23
- Earliest claimed: 2007-04-20
- Assignees: Sharp Laboratories of America, Inc.
- Inventors: Petrus J.L. van Beek
- Web page: https://patentgrep.com/patent/US20080259799A1

## CPC

- H04L 47/2416: Real-time traffic
- H04L 47/26: using explicit feedback to the source, e.g. choke packets
- H04L 47/32: by discarding or delaying data units, e.g. packets or frames
- H04N 21/23406: {involving management of server-side video buffer}
- H04N 21/262: Content or additional data distribution scheduling, e.g. sending additional data at off-peak times, updating software modules, calculating the carousel transmission frequency, delaying a video stream transmission, generating play-lists {(scheduling strategies for dispatcher in multiprogramming arrangements G06F9/4881; arrangements for scheduling broadcast services or broadcast-related services H04H60/06; flow control in packet networks H04L47/10; establishing a time schedule or organising the servicing of application requests H04L67/62)}
- H04N 21/2662: Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
- H04N 21/6582: {Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number (arrangements where receivers interact with the broadcast H04H20/38)}
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

The embodiments of the invention relate to video streaming, particularly to quality-aware adaptive and selective transmissions. The embodiments of the present invention provide for a set of scheduling patterns to be evaluated, and from such set determine the target scheduling pattern that is calculated to provide the least distortion increment based on the evaluation set.

## Claims

### Claim 1

A method of transmitting between a sender and a receiver, a data stream comprising a plurality of data units in a transmission buffer at the sender, wherein the plurality of data units comprises a first data unit at the front of the transmission buffer and other data units following the first data unit, the method comprising the steps of:
applying a set of evaluation patterns based on at least one pattern-selection rule, wherein the set of evaluation patterns comprises one or more scheduling patterns with each scheduling pattern comprising a plurality of scheduling decisions, each decision associated with a data unit from the plurality of data units in the transmission buffer, and wherein each scheduling decision indicates whether the associated data unit is evaluated for transmission or dropping;
calculating a distortion increase value for each scheduling pattern from the set of evaluation patterns;
determining a target scheduling pattern based on a least calculated distortion value among the calculated distortion increase values from each scheduling pattern from the set of evaluation patterns; and
processing the first data unit in the transmission buffer for transmission or dropping based on the decision associated with the first data unit from the determined target scheduling pattern.

### Claim 2 (depends on 1)

The method of claim 1, further comprising the steps of:
repeating the steps of claim 1 at a next transmission opportunity.

### Claim 3 (depends on 1)

The method of claim 1, wherein a number of the one or more scheduling patterns in the set of evaluation patterns is based on available resources at the sender.

### Claim 4 (depends on 1)

The method of claim 1, wherein the plurality of data units in the transmission buffer considered is only a portion of data units actually contained in the transmission buffer.

### Claim 5 (depends on 4)

The method of claim 4, wherein the plurality of data units considered is determined based on available resources at the sender.

### Claim 6 (depends on 4)

The method of claim 4, wherein the plurality of data units considered is determined based on the coding type of the data units contained in the transmission buffer.

### Claim 7 (depends on 1)

The method of claim 1, wherein the at least one pattern-selection rule is at least one rule from the following:
transmit all intra-coded data units;
transmit at least one descendant data unit only if all the ancestor data units of the at least one descendant data unit are transmitted;
drop at least one low-temporal-level data unit with a low temporal level only if data units with higher temporal levels higher than the low temporal level of the at least one data unit are dropped;
drop at least one data unit only if data units later in the transmission order are also dropped;
drop at least one co-temporal-level data unit with a temporal level only if data units, earlier in the transmission order, with temporal level equal to the temporal level of the at least one co-temporal-level data unit are also dropped;
drop at least one data unit that is part of a group of consecutive data units-wherein each data unit in the group of consecutive data units have the same temporal level and are immediately next to each other, only if data units from any other groups of consecutive data units with the same temporal level and earlier in the transmission order are also dropped;
drop data units uniformly among groups of consecutive data units, wherein each data unit in each group of consecutive data units have the same temporal level and are immediately next to each other; and
drop data units in an increasing manner among groups of consecutive data units, wherein each data unit in each group of consecutive data units have the same temporal level and are immediately next to each other, and wherein the increasing manner relates to how many data units are being dropped.

### Claim 8 (depends on 1)

The method of claim 1, wherein the at least one pattern-selection rule is based on at least one of the following:
coding dependencies between different data units in the transmission buffer;
coding types of data units in the transmission buffer;
coding order of data units in the transmission buffer;
display order of data units in the transmission buffer;
transmission order of data units in the transmission buffer;
temporal levels of data units in the transmission buffer.

### Claim 9 (depends on 1)

The method of claim 1 wherein the step of calculating the distortion increase value for each scheduling pattern is based on one or more lookups from a peak signal-to-noise ratio (PSNR) parameter table containing a PSNR parameter value for each data unit in a group of picture structure.

### Claim 10 (depends on 1)

The method of claim 1 wherein the step of calculating the distortion increase for each scheduling pattern is based on one or more lookups from a peak signal-to-noise ratio (PSNR) parameter table containing a PSNR parameter value for each unique frame coding type in a group of picture structure.

### Claim 11 (depends on 1)

The method of claim 1 wherein the step of calculating the distortion increase value for each scheduling pattern is based on a linear function of a logarithm of an input bit rate of the data stream.

### Claim 12 (depends on 1)

The method of claim 1 wherein the step of calculating the distortion increase value for each scheduling pattern is based on a probability that one or more data units, each associated with a delivery deadline, are calculated to not arrive at the receiver at the associated delivery deadlines.

### Claim 13 (depends on 12)

The method of claim 12 wherein the probability is determined based on at least one model of the following models: a Gaussian distribution model; a Poisson distribution model.

### Claim 14 (depends on 13)

The method of claim 13 wherein the model describes the distribution of either:
a number of packets delivered to the receiver within a time interval; or
a number of bytes delivered to the receiver within a time interval.

### Claim 15 (depends on 12)

The method of claim 12 wherein the probability is based on a number of bytes comprising the data units transmitted given a scheduling pattern.

### Claim 16 (depends on 12)

The method of claim 12 wherein the probability is based on estimating an average of a number of bytes delivered to the receiver within a time interval.

### Claim 17 (depends on 16)

The method of claim 16 wherein the probability is further based on estimating a standard deviation of the number of bytes delivered to the receiver within a time interval.

### Claim 18 (depends on 17)

The method of claim 17 wherein said estimating is based on a feedback message received by the sender from the receiver.

### Claim 19 (depends on 1)

The method of claim 1 wherein the step of determining the target scheduling pattern from the set of evaluation patterns further comprises maintaining an interim minimum distortion variable.

### Claim 20 (depends on 19)

The method of claim 19 wherein the step of calculating the distortion increase value for each scheduling pattern further comprises
calculating a distortion increase at a data unit level for each data unit associated with a scheduling decision in the plurality of scheduling decisions of the scheduling pattern;
accumulating the calculated distortion increase at the data unit level in a temporary variable;
comparing the temporary variable value with the interim minimum distortion variable value; and
if the value of the temporary variable is greater than the interim minimum distortion variable value, then skipping the step of calculating the distortion increase at a data unit level for data units in the scheduling pattern not yet evaluated.

### Claim 21 (depends on 1)

The method of claim 1 wherein the step of calculating the distortion increase for each scheduling pattern further comprises calculating and accumulating a distortion increase at a data unit level for each data unit associated with a scheduling decision in the plurality of scheduling decisions of the scheduling pattern.

### Claim 22 (depends on 1)

The method of claim 1 wherein the set of evaluation patterns comprising the one or more scheduling patterns are dynamically generated via a set of program instructions.

### Claim 23 (depends on 1)

The method of claim 1 wherein the step of calculating the distortion increase value for each scheduling pattern is based on a defined earlier target delivery deadline, earlier in time than a delivery deadline originally associated with each data unit.

### Claim 24 (depends on 1)

The method of claim 1 wherein the step of calculating the distortion increase value for each scheduling pattern is based on a receiver buffer fullness parameter.

### Claim 25 (depends on 1)

The method of claim 1 wherein the data stream corresponds to a video stream.

### Claim 26 (depends on 25)

The method of claim 25 wherein each data unit corresponds to a coded video frame.

### Claim 27

A device adapted to be operably connected to a network, the device comprising:
a transmission buffer comprising a data stream, wherein the data stream comprises a plurality of data units, and wherein the plurality of data units comprises a first data unit at the front of the buffer and other data units following the first data unit;
a pattern module adapted to:
apply a set of evaluation patterns based on at least one pattern-selection rule, wherein the set of evaluation patterns comprises one or more scheduling patterns with each scheduling pattern comprising a plurality of scheduling decisions, each decision associated with a data unit from the plurality of data units in the transmission buffer, and wherein each scheduling decision indicates whether the associated data unit is evaluated for transmission or dropping;
a distortion module adapted to:
calculate a distortion increase value for each scheduling pattern from the set of evaluation patterns; and
a target pattern determinator module adapted to:
determine a target scheduling pattern based on a least calculated distortion value among the calculated distortion increase values from each scheduling pattern from the set of evaluation patterns; and
process the first data unit in the transmission buffer for transmission or dropping based on the decision associated with the first data unit from the determined target scheduling pattern.

### Claim 28

A system comprising:
a first device comprising operably coupled to a second device via one or more network segments, the first device comprising:
a transmission buffer comprising a data stream, wherein the data stream comprises a plurality of data units, and wherein the plurality of data units comprises a first data unit at the front of the buffer and other data units following the first data unit;
a pattern module adapted to:
apply a set of evaluation patterns based on at least one pattern-selection rule, wherein the set of evaluation patterns comprises one or more scheduling patterns with each scheduling pattern comprising a plurality of scheduling decisions, each decision associated with a data unit from the plurality of data units in the transmission buffer, and wherein each scheduling decision indicates whether the associated data unit is evaluated for transmission or dropping;
a distortion module adapted to:
calculate a distortion increase value for each scheduling pattern from the set of evaluation patterns; and
a target pattern determinator module adapted to:
determine a target scheduling pattern based on a least calculated distortion value among the calculated distortion increase values from each scheduling pattern from the set of evaluation patterns; and
process the first data unit in the transmission buffer for transmission or dropping based on the decision associated with the first data unit from the determined target scheduling pattern.
the second device adapted to:
receive data units transmitted by the first device, wherein the data units received by the second device include data units from the transmission buffer processed for transmission by the first device; and
the one or more network segments.
