# US 9,883,214 B2 · Efficient approach to dynamic frame size and frame rate adaptation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,883,214 B2
- Document ID: US9883214B2
- Status: Granted
- Kind: B2
- Application number: 14/674,348
- Filed: 2015-03-31
- Published: 2016-10-06
- Granted: 2018-01-30
- Earliest claimed: 2015-03-31
- Assignees: Dialogic Corporation
- Applicants: Dialogic Corporation
- Inventors: Kyeong Ho Yang; Beril Erkin; Antony R. Martin
- Web page: https://patentgrep.com/patent/US9883214B2

## CPC

- H04N 21/234363: {by altering the spatial resolution, e.g. for clients with a lower screen resolution}
- H04N 21/234381: {by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping}
- H04N 21/23805: {Controlling the feeding rate to the network, e.g. by controlling the video pump}
- H04N 21/2402: {Monitoring of the downstream path of the transmission network, e.g. bandwidth available (traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876)}
- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Improved systems and methods of performing multimedia communications over multimedia communications networks, in which video data senders can maintain high video quality of experience (QoE) levels with increased reliability despite changes in available bandwidths of video data receivers. In the disclosed systems and methods, video encoding parameters employed by the video data senders, including at least the video frame size and/or the video frame rate, can be dynamically adapted to the available bandwidths of the video data receivers, taking into account possible effects of spatial scaling and/or temporal scaling of video frames on the resulting video QoE.

## Claims

### Claim 1

In a multimedia communications system, a method of dynamically adapting video encoding parameters of a video data sender to an available bandwidth of a video data receiver, comprising:
obtaining a target bitrate of encoded video for the video data receiver;
obtaining a set of scaled frame size/frame rate pairs, each scaled frame size/frame rate pair including a frame size scaled by a predetermined spatial scaling factor, and a frame rate scaled by a predetermined temporal scaling factor;
for each scaled frame size/frame rate pair:
obtaining a fair quality bitrate of the encoded video as a function of at least the scaled frame size, the scaled frame rate, and a content complexity of the encoded video;
obtaining a scaling penalty as a function of at least a frame size of a previous encoded frame of the encoded video, a frame rate of the previous encoded frame of the encoded video, and the content complexity of the encoded video; and
obtaining a score value as a function of at least the scaling penalty and an absolute difference between the fair quality bitrate and the target bitrate of the encoded video; and
configuring a video encoder of the video data sender with the scaled frame size and the scaled frame rate included in the respective scaled frame size/frame rate pair having a score value satisfying a predetermined condition.

### Claim 2 (depends on 1)

The method of claim 1 wherein the obtaining of the fair quality bitrate of the encoded video includes obtaining the fair quality bitrate as a function of at least the scaled frame size, the scaled frame rate, and a number of bits per macroblock of the encoded video, the number of bits per macroblock of the encoded video being a function of at least the content complexity of the encoded video.

### Claim 3 (depends on 2)

The method of claim 2 further comprising:
obtaining the content complexity of the encoded video as a function of at least a resolution of frames encoded by the video encoder, a bitrate of the encoded video, and a quantization step size of the encoded video.

### Claim 4 (depends on 3)

The method of claim 3 further comprising:
obtaining the quantization step size of the encoded video as a function of at least a codec type of the video encoder, and a quantization parameter employed by the video encoder.

### Claim 5 (depends on 3)

The method of claim 3 further comprising:
modeling the content complexity of the encoded video in terms of a rate-distortion model.

### Claim 6 (depends on 5)

The method of claim 5 wherein the modeling of the content complexity of the encoded video includes modeling the content complexity in terms of the rate-distortion model defined as,
σ2=αR*Qstep2*h(frame_size_enc)*i(frame_rate_enc),
wherein “σ2” is the content complexity, “α” is a predetermined constant, “R” is the number of bits per macroblock of the encoded video, “Qstep” is the quantization step size of the encoded video, “h(frame_size_enc)” is a function of a frame size encoding parameter of the video encoder, and “i(frame_rate_enc)” is a function of a frame rate encoding parameter of the video encoder.

### Claim 7 (depends on 6)

The method of claim 6 further comprising:
obtaining a current number of bits per macroblock of a current encoded frame as a function of the bitrate of the encoded video, the frame size encoding parameter, and the frame rate encoding parameter.

### Claim 8 (depends on 7)

The method of claim 7 further comprising:
obtaining the quantization step size of the encoded video as a weighted average of at least a current quantization step size of the current encoded frame and a previous quantization step size of the previous encoded frame.

### Claim 9 (depends on 8)

The method of claim 8 further comprising:
obtaining the number of bits per macroblock of the encoded video as a weighted average of at least the current number of bits per macroblock of the current encoded frame and a previous number of bits per macroblock of the previous encoded frame.

### Claim 10 (depends on 9)

The method of claim 9 wherein the current quantization step size of the current encoded frame and the current number of bits per macroblock of the current encoded frame each have an associated weighting value, and wherein the method further comprises:
detecting whether a scene change has occurred in a sequence of video frames; and
having detected the scene change in the sequence of video frames, modifying the associated weighting value of one or more of the current quantization step size of the current encoded frame, and the current number of bits per macroblock of the current encoded frame.

### Claim 11 (depends on 7)

The method of claim 7 wherein a current quantization step size of the current encoded frame and the current number of bits per macroblock of the current encoded frame each have an associated weighting value, and wherein the method further comprises:
determining whether the current encoded frame is an inter-frame, the inter-frame having at least one associated reference frame; and
having determined that the current encoded frame is an inter-frame, determining a video quality of the associated reference frame.

### Claim 12 (depends on 11)

The method of claim 11 wherein the determining of the video quality of the associated reference frame includes determining the video quality of the associated reference frame based at least on a quantization parameter used to encode the associated reference frame.

### Claim 13 (depends on 12)

The method of claim 12 wherein the determining of the video quality of the associated reference frame includes determining a first average quantization parameter used to encode a first plurality of macroblocks of the associated reference frame, and determining a second average quantization parameter used to encode a second plurality of macroblocks of at least one previous reference frame.

### Claim 14 (depends on 13)

The method of claim 13 further comprising:
determining whether a value of the first average quantization parameter is different from a value of the second average quantization parameter by a predetermined amount; and
having determined that the value of the first average quantization parameter is different from the value of the second average quantization parameter by the predetermined amount, modifying the associated weighting value of one or more of the current quantization step size of the current encoded frame, and the current number of bits per macroblock of the current encoded frame.

### Claim 15 (depends on 6)

The method of claim 6 wherein the modeling of the content complexity in terms of the rate-distortion model includes defining the function of the frame size encoding parameter as,
h(frame_size_enc)=1+√(frame_size_enc),
wherein “frame_size_enc” corresponds to the frame size encoding parameter.

### Claim 16 (depends on 6)

The method of claim 6 wherein the modeling of the content complexity in terms of the rate-distortion model includes defining the function of the frame rate encoding parameter as,
i(frame_rate_enc)=1+√(frame_rate_enc),
wherein “frame_rate_enc” corresponds to the frame rate encoding parameter.

### Claim 17 (depends on 5)

The method of claim 5 further comprising:
obtaining the content complexity of the encoded video, modeled in terms of the rate-distortion model, for at least one predetermined region of interest within first and second domains defined by the number of bits per macroblock of the encoded video and the quantization step size of the encoded video, respectively.

### Claim 18 (depends on 17)

The method of claim 17 wherein the predetermined region of interest has a first set of boundaries relative to the first domain defined by number of bits per macroblock of the encoded video, and wherein the method further comprises:
determining the first set of boundaries of the predetermined region of interest as a function of the content complexity of the encoded video.

### Claim 19 (depends on 17)

The method of claim 17 wherein the predetermined region of interest has a second set of boundaries relative to the second domain defined by the quantization step size of the encoded video, and wherein the method further comprises:
determining the second set of boundaries of the predetermined region of interest as a function of the content complexity of the encoded video.

### Claim 20 (depends on 17)

The method of claim 17 wherein the modeling of the content complexity of the encoded video includes modeling the content complexity of the encoded video in terms of a first modified rate-distortion model.

### Claim 21 (depends on 20)

The method of claim 20 further comprising:
obtaining the content complexity of the encoded video, modeled in terms of the first modified rate-distortion model, for at least one predetermined upper subdomain region outside the region of interest within the first and second domains defined by the number of bits per macroblock of the encoded video and the quantization step size of the encoded video, respectively.

### Claim 22 (depends on 21)

The method of claim 21 wherein the obtaining of the content complexity of the encoded video for the predetermined upper subdomain region outside the region of interest includes obtaining the content complexity of the encoded video, modeled in terms of the first modified rate-distortion model, for the predetermined upper subdomain region, the predetermined upper subdomain region being unbounded for one or more values of the number of bits per macroblock of the encoded video greater than a first predetermined value.

### Claim 23 (depends on 20)

The method of claim 20 wherein the modeling of the content complexity of the encoded video includes modeling the content complexity of the encoded video in terms of a second modified rate-distortion model.

### Claim 24 (depends on 23)

The method of claim 23 further comprising:
obtaining the content complexity of the encoded video, modeled in terms of the second modified rate-distortion model, for at least one predetermined lower subdomain region outside the region of interest within the first and second domains defined by the number of bits per macroblock of the encoded video and the quantization step size of the encoded video, respectively.

### Claim 25 (depends on 24)

The method of claim 24 wherein the obtaining of the content complexity of the encoded video for the predetermined lower subdomain region outside the region of interest includes obtaining the content complexity of the encoded video, modeled in terms of the second modified rate-distortion model, for the predetermined lower subdomain region, the predetermined lower subdomain region being unbounded for values of the number of bits per macroblock of the encoded video less than a second predetermined value.

### Claim 26 (depends on 6)

The method of claim 6 wherein the modeling of the content complexity in terms of the rate-distortion model includes defining a target quantization step size of the encoded video to be a function of the content complexity and the frame rate encoding parameter.

### Claim 27 (depends on 26)

The method of claim 26 wherein the modeling of the content complexity in terms of the rate-distortion model includes obtaining the number of bits per macroblock, R, of the encoded video as
R=logα(σ2)−logα(Qstep_target2)−logα(h(frame_size_enc))−logα(i(frame_rate_enc)),
wherein “Qstep_target” is the target quantization step size of the encoded video.

### Claim 28 (depends on 26)

The method of claim 26 further comprising:
implementing the function of the content complexity and the frame rate encoding parameter as a lookup table, in which higher video content complexity values correspond to increased values of the target quantization step size of the encoded video, and lower video content complexity values correspond to reduced values of the target quantization step size of the encoded video.

### Claim 29 (depends on 1)

The method of claim 1 wherein the obtaining of the fair quality bitrate of the encoded video includes obtaining the fair quality bitrate as a product of the scaled frame size, the scaled frame rate, and the content complexity of the encoded video.

### Claim 30 (depends on 29)

The method of claim 29 wherein the obtaining of the fair quality bitrate of the encoded video includes, in the event the encoded video contains dynamic video content, obtaining the fair quality bitrate as
fair_quality_bitratenm=R*scaled_frame_sizen*(ε*original_frame_rate+(1−ε)*scaled_frame_ratem),
wherein “R” is the number of bits per macroblock of the encoded video, “original frame rate” is an original frame rate of the video encoder, and “ε” is a predetermined constant.

### Claim 31 (depends on 1)

The method of claim 1 wherein the obtaining of the scaling penalty includes one or more of obtaining a spatial scaling penalty of the encoded video, obtaining a temporal scaling penalty of the encoded video, and obtaining a motion scaling penalty of the encoded video.

### Claim 32 (depends on 31)

The method of claim 31 wherein the obtaining of the spatial scaling penalty of the encoded video includes obtaining the spatial scaling penalty as
scaling_penalty_spatialn={1.0+30*spatio_temporal_factor*n|n=0, 1, . . . , N−1},
wherein “N” is a predetermined number of scaling factors for obtaining the respective scaled frame sizes, “n” is an nth one of the predetermined number, N, of scaling factors, and “spatio_temporal_factor” is a predetermined factor for use in prioritizing either spatial scaling or temporal scaling of the encoded video based on a resolution of frames encoded by the video encoder.

### Claim 33 (depends on 32)

The method of claim 32 wherein the obtaining of the temporal scaling penalty of the encoded video includes obtaining the temporal scaling penalty as
scaling_penalty_temporalm={1.0+(m/spatio_temporal_factor)|m=0, 1, . . . , M−1},
wherein “M” is a predetermined number of scaling factors for obtaining the respective scaled frame rates, and “m” is an mth one of the predetermined number, M, of scaling factors.

### Claim 34 (depends on 33)

The method of claim 33 further comprising:
obtaining the predetermined factor for use in prioritizing either the spatial scaling or the temporal scaling of the encoded video as
spatio
—
⁢
temporal
—
⁢
factor
=
6
*
(
12
-
frame
—
⁢
size
—
⁢
target
ϕ
)
frame
—
⁢
rate
—
⁢
target
θ
,
wherein “frame_size_target” is a target frame size of the video encoder, “frame_rate_target” is a target frame rate of the video encoder, and “φ” and “θ” are each predetermined constants.

### Claim 35 (depends on 33)

The method of claim 33 wherein the obtaining of the motion scaling penalty of the encoded video includes obtaining the motion scaling penalty as
scaling
—
⁢
penalty
—
⁢
motion
m
=
1
+
motion
—
⁢
complexity
MAX
—
⁢
MOTION
—
⁢
COMPLEXITY
*
|
frame
—
⁢
rate
—
⁢
target
-
scaled
—
⁢
frame
—
⁢
rate
m
⁢
|
v
frame
—
⁢
rate
—
⁢
target
,
wherein “motion_complexity” is the motion complexity of the encoded video, “frame_rate_target” is a target frame rate of the video encoder, “scaled_frame_ratem” is the scaled frame rate for the mth one of the predetermined number, M, of scaling factors, and “MAX_MOTION_COMPLEXITY” and “ν” are each predetermined constants.

### Claim 36 (depends on 35)

The method of claim 35 wherein the obtaining of the scaling penalty further includes one or more of obtaining a change scaling penalty of the encoded video and obtaining a details scaling penalty of the encoded video, wherein the change scaling penalty pertains to at least one change in one or more of a video frame size and a video frame rate of the encoded video, and wherein the details scaling penalty pertains to at least one change in a level of spatial details in the encoded video.

### Claim 37 (depends on 36)

The method of claim 36 wherein the obtaining of the change scaling penalty of the encoded video includes obtaining the change scaling penalty as
scaling
—
⁢
penalty
—
⁢
change
nm
=
1
+
(
|
scaled
—
⁢
frame
—
⁢
size
n
-
frame
—
⁢
size
—
⁢
enc
⁢
|
v
frame
—
⁢
size
—
⁢
enc
)
+
(
|
scaled
—
⁢
frame
—
⁢
rate
m
-
frame
—
⁢
rate
—
⁢
enc
⁢
|
v
frame
—
⁢
rate
—
⁢
enc
)
,
wherein “frame_size_enc” is a frame size encoding parameter of the video encoder, “frame_rate_enc” is a frame rate encoding parameter of the video encoder, and “υ” and “ν” are each predetermined constants.

### Claim 38 (depends on 37)

The method of claim 37 wherein the content complexity of the encoded video includes a spatial complexity of the encoded video, and wherein the obtaining of the details scaling penalty of the encoded video includes obtaining the details scaling penalty as
scaling
—
⁢
penalty
—
⁢
details
n
=
1
+
spatial
—
⁢
complexity
MAX
—
⁢
SPATIAL
—
⁢
COMPLEXITY
*
|
frame
—
⁢
size
—
⁢
target
-
scaled
—
⁢
frame
—
⁢
size
n
⁢
|
v
frame
—
⁢
size
—
⁢
target
,
wherein “spatial_complexity” corresponds to the spatial complexity of the encoded video, “frame_size_target” is a target frame size of the video encoder, and “MAX_SPATIAL_COMPLEXITY” is a predetermined constant.

### Claim 39 (depends on 38)

The method of claim 38 wherein the obtaining of the scaling penalty includes obtaining a product of two or more of the spatial scaling penalty of the encoded video, the temporal scaling penalty of the encoded video, the motion scaling penalty of the encoded video, the change scaling penalty of the encoded video, and the details scaling penalty of the encoded video.

### Claim 40 (depends on 1)

The method of claim 1 wherein the configuring of the video encoder includes configuring the video encoder with the scaled frame size and the scaled frame rate included in the respective scaled frame size/frame rate pair having the score value that satisfies the predetermined condition of being a maximum score value.

### Claim 41

A system for dynamically adapting video encoding parameters of a video data sender to an available bandwidth of a video data receiver, comprising:
at least one processor operative to execute at least one program out of at least one memory:
to obtain a target bitrate of encoded video for the video data receiver;
to obtain a set of scaled frame size/frame rate pairs, each scaled frame size/frame rate pair including a frame size scaled by a predetermined spatial scaling factor, and a frame rate scaled by a predetermined temporal scaling factor;
for each scaled frame size/frame rate pair:
to obtain a fair quality bitrate of the encoded video as a function of at least the scaled frame size, the scaled frame rate, and a content complexity of the encoded video;
to obtain a scaling penalty as a function of at least a frame size of a previous encoded frame of the encoded video, a frame rate of the previous encoded frame of the encoded video, and the content complexity of the encoded video; and
to obtain a score value as a function of at least the scaling penalty and an absolute difference between the fair quality bitrate and the target bitrate of the encoded video; and
to configure a video encoder of the video data sender with the scaled frame size and the scaled frame rate included in the respective scaled frame size/frame rate pair having a score value satisfying a predetermined condition.

### Claim 42 (depends on 41)

The system of claim 41 wherein the at least one processor is further operative to execute the at least one program out of the at least one memory to configure the video encoder with the scaled frame size and the scaled frame rate included in the respective scaled frame size/frame rate pair having the score value that satisfies the predetermined condition of being a maximum score value.

### Claim 43

In a multimedia communications system, a method of dynamically adapting video encoding parameters of a video data sender to an available bandwidth of a video data receiver, comprising:
obtaining a set of scaled frame size/frame rate pairs, each scaled frame size/frame rate pair including a frame size scaled by a predetermined spatial scaling factor, and a frame rate scaled by a predetermined temporal scaling factor;
for each scaled frame size/frame rate pair:
obtaining a fair quality bitrate of encoded video as a function of at least the scaled frame size, the scaled frame rate, and a content complexity of the encoded video;
obtaining a scaling penalty as a function of at least a frame size of a previous encoded frame of the encoded video, a frame rate of the previous encoded frame of the encoded video, and the content complexity of the encoded video; and
obtaining a score value as a function of at least the scaling penalty, the fair quality bitrate, and a target bitrate of the encoded video for the video data receiver; and
configuring a video encoder of the video data sender with the scaled frame size and the scaled frame rate included in the respective scaled frame size/frame rate pair having a score value satisfying a predetermined condition.

### Claim 44 (depends on 43)

The method of claim 43 wherein the configuring of the video encoder includes configuring the video encoder with the scaled frame size and the scaled frame rate included in the respective scaled frame size/frame rate pair having the score value that satisfies the predetermined condition of being a maximum score value.