US 10,313,692 B2Granted

Visual perception characteristics-combining hierarchical video coding method

AssigneeBeijing University of Technology

InventorsPengyu Liu, Kebin Jia

Life of this patent

Estimating the term…

Abstract

A visual perception characteristics-combining hierarchical video coding method includes: setting priority of visual interest area and setting allocation scheme of video coding resource. Setting priority of visual interest area includes analyzing both time and space visual characteristic saliency of video content. Setting allocation scheme of video coding resource includes, according to the priority of visual interest area, macroblock coding resource of interest area is satisfied preferentially to realize hierarchical coding.

Claims17

Dependent claims

A visual perception characteristics-combining hierarchical video coding method comprising:

setting priority of visual interest area and setting allocation scheme of video coding resource;
wherein:
the setting priority of visual interest area mainly is:
calculation formula for labeling visual characteristic saliency area is expressed as:
ROI
⁡
(
x
,
y
)
=
{
3
,
(
(
T
⁡
(
x
,
y
,
MV
)
=
2
⁢
⁢
or
⁢
⁢
T
⁡
(
x
,
y
,
MV
)
=
1
)
||
S
⁢
(
x
,
y
,
Mode
)
=
1
)
)
⁢
orS
⁡
(
x
,
y
,
Mode
)
=
2
2
,
(
T
⁡
(
x
,
y
,
MV
)
=
2
⁢
⁢
or
⁢
⁢
T
⁡
(
x
,
y
,
MV
)
=
1
)
||
S
⁡
(
x
,
y
,
Mode
)
=
0
1
,
T
⁡
(
x
,
y
,
MV
)
=
0
||
S
⁡
(
x
,
y
,
Mode
)
=
1
0
,
T
⁡
(
x
,
y
,
MV
)
=
0
)
||
(
S
⁡
(
x
,
y
,
Mode
)
=
0
where ROI(x, y) represents visual interest priority of current coding macroblock, T(x, y,MV) represents time visual characteristic saliency of current coding macroblock, S(x, y,Mode) represents space visual characteristic saliency of current coding macroblock, (x, y) represents coordinate of current coding macroblock,
the setting allocation scheme of video coding resource comprises optimization of macroblock coding of the visual interest area so as to improve real-time performance of video coding;
using fast intra-frame prediction algorithm and gray scaly histogram of macroblock to describe a flat level of macroblock and according to the flat level of macroblock to select possible set of intra-frame prediction modes;
using fast inter-frame prediction algorithm, through analyzing statistical characteristics of probability of occurrence of inter-frame prediction modes, to prejudge specific inter-frame prediction modes so that unnecessary inter-frame prediction mode search and calculation of RD cost can be stopped in advance;
using motion estimation search algorithm, based on relevance of macroblock motion vector, according to coding block movement, to identify search level, realizing high efficient search;
wherein the setting priority of visual interest area comprises:
labeling time visual characteristic saliency area, which includes step 1 and step 2, step 1 being motion vector noise detection and step 2 being translation motion vector detection, the motion vector noise detection and the translation motion vector detection are used for reduction of the influence to accuracy of time visual saliency area measurement due to motion vector noise and translation motion vector generated by camera motion respectively, and to perform separation between foreground and background,
labeling space visual characteristic saliency area, and
labeling, according to labeling results of the time visual characteristic saliency area and the space visual characteristic saliency area, the visual characteristic saliency area;
wherein:
when labeling the time visual characteristic saliency area, calculation formula for the motion vector noise detection in step 1 is expressed as:
T
1
⁡
(
x
,
y
,
MV
)
=
{
3
,
if
⁢
⁢

V
rr
_

=
0
2
,
elseif
⁢
⁢

V
→
s

≥

V
rr
_

T
2
⁡
(
x
,
y
,
MV
)
esle
(
1
)
where (x, y) in (1) represents position coordinates, {right arrow over (V)}s represents motion vector of current macroblock, Vrr is average vector of motion reference area Crr, defined as:
V
_
rr
=
∑
∈
C
rr
⁢
v
rr
→
num
C
rr
{right arrow over (v)}rr represents macroblock motion vector in Crr, numCrr represents accumulation times
Crr is defined so that shape, position and area of reference area Crr can be adaptively adjusted with the change of motion vector {right arrow over (V)}s
four macroblocks located in upper right, upper left, lower right and lower left of Crr are MB1, MB2, MB3, MB4, position coordinates of MB1, MB2, MB3, MB4 are defined as:
MB
k
⁡
(
x
k
,
y
k
)
k
=
1
,
2
,
3
,
4
=
{
MB
1
⁡
(
x
-
j
c
,
y
-
i
c
)
MB
2
⁡
(
x
-
j
c
,
y
+
i
c
)
MB
3
⁡
(
x
+
j
c
,
y
-
i
c
)
MB
4
⁡
(
x
+
j
c
,
y
+
i
c
)
⁢
⁢
where
⁢
⁢
i
c
=
[

V
sx
→

w
s
+
1
]
,
j
c
=
[

V
sy
→

h
s
+
1
]
,

V
sx
→

,
and
⁢
⁢

V
sy
→

are motion amplitude of current motion vector {right arrow over (V)}s in horizontal and vertical, ws and hs represent width and height of current code block, respectively, [⋅] represents integer calculation,
if |Vrr|=0, it indicates that there is no motion vector in Crr, so that {right arrow over (V)}s is caused by motion noise and thus is removed, and {right arrow over (V)}s is set as 0, and labeled as T1(x, y,MV)=3,
if |{right arrow over (V)}s|≥|Vrr|, it indicates that current code block has higher motion characteristic saliency than adjacent macroblocks, which belongs to foreground dynamic area, and labeled as T1(x, y,MV)=2,
otherwise, current code block has substantially the same motion characteristic saliency as adjacent macroblocks, then translation motion vector detection is performed to confirm whether the current code block belongs to background area or foreground translation area, which is labeled as T2 (x, y,MV),
calculation formula of translation motion vector detection of step 2 is:
T
2
⁡
(
x
,
y
,
MV
)
=
{
1
,
if
⁢
⁢
SAD
(
x
,
y
)
≥
S
_
c
0
,
else
(
2
)
where (x,y) represents position coordinates of current code block, Sc represents dynamic threshold, SAD(x,y) represents sum of absolute differences between current code block and corresponding station block in previous frame, used to characterize the change between two adjacent corresponding code blocks, defined as:
SAD
(
x
,
y
)
=
∑
i
=
1
M
⁢
∑
j
=
1
N
⁢

s
⁡
(
i
,
j
)
-
c
⁡
(
i
,
j
)

where s(i, j) represents pixel value of current code block, c(i, j) represents corresponding pixel value in pervious frame, MN represents length and wide of current code block respectively,
Sc represents average value of SAD of all code blocks in background area of pervious frame, defined as:
S
_
c
=
∑
x
,
y
∈
S
c
⁢
SAD
(
x
,
y
)
Num
Sc represents background area of pervious frame,
∑
x
,
y
∈
S
c
⁢
SAD
(
x
,
y
)
represents the sum of SAD value of current code blocks in ScNum represents accumulation times,
formula for labeling the time visual characteristic saliency area is expressed as:
T
⁡
(
x
,
y
,
MV
)
=
{
3
,
if
⁢
⁢

V
rr
_

=
0
2
,
elseif
⁢
⁢

V
s
→

≥

V
rr
_

1
,
elseif
⁢
⁢
SAD
(
x
,
y
)
≥
S
_
c
0
,
else
(
3
)
where parameters of (3) are same with (1) and (2)
then, performing labeling of the space visual characteristic saliency area, calculation formula is expressed as:
S
⁡
(
x
,
y
,
Mode
)
=
{
2
,
Mode
P
∈
{
Intra
}
1
,
Mode
P
∈
{
Inter
⁢
⁢
8
}
⁢
⁢
or
⁢
⁢
Mode
I
∈
Intra
⁢
⁢
4
×
4
0
,
Mode
P
∈
{
Inter
⁢
⁢
16
}
⁢
⁢
or
⁢
⁢
Mode
I
∈
Intra
⁢
⁢
16
×
16
(
4
)
where (x, y) represents position coordinates of current code block, Mode represents prediction pattern of current code block, modeP represents prediction pattern of current code block in P frame of code, modeI represents prediction pattern of current code block in I frame of code,
if modeP chooses intra-frame prediction mode, it indicates that space visual characteristic saliency is the highest, which belongs to sensitive area, and is labeled as S(x, y, Mode)=2;
if modeP chooses a set of sub-block prediction modes, Inter8(8×8, 8×4, 4×8, 4×4) or Intra4×4 prediction mode, it indicates that spatial detail is rich, and the space visual characteristic saliency is high, which belongs to interest area, and is labeled as:
S(x, y, Mode)=1;
if modeP chooses a set of inter-frame prediction modes, Inter16(Skip, 16×16, 16×8, 8×16) or modeI chooses Intra16×16 prediction mode, it indicates that spatial change is flat, and the space visual characteristic saliency is low, which belongs to non-obvious area, and labeled as S(x, y,Mode)=0;
labeling, according to labeling results of the space visual characteristic saliency area and the time visual characteristic saliency area, the visual characteristic saliency area.

626 more elements

A visual perception characteristics-combining hierarchical video coding method comprising:

setting priority of visual interest area and setting allocation scheme of video coding resource;
wherein:
the setting priority of visual interest area mainly is:
calculation formula for labeling visual characteristic saliency area is expressed as:
ROI
⁡
(
x
,
y
)
=
{
3
,
(
(
T
⁡
(
x
,
y
,
MV
)
=
2
⁢
⁢
or
⁢
⁢
T
⁢
⁢
(
x
,
y
,
MV
)
=
1
)
||
S
⁡
(
x
,
y
,
Mode
)
=
1
)
)
⁢
orS
⁡
(
x
,
y
,
Mode
)
=
2
2
,
(
T
⁡
(
x
,
y
,
MV
)
=
2
⁢
⁢
or
⁢
⁢
T
⁢
⁢
(
x
,
y
,
MV
)
=
1
)
||
S
⁡
(
x
,
y
,
Mode
)
=
0
1
,
T
⁡
(
x
,
y
,
MV
)
=
0
||
S
⁡
(
x
,
y
,
Mode
)
=
1
0
,
T
⁡
(
x
,
y
,
MV
)
=
0
)
||
(
S
⁡
(
x
,
y
,
Mode
)
=
0
where ROI(x, y) represents visual interest priority of current coding macroblock, T(x, y,MV) represents time visual characteristic saliency of current coding macroblock, S(x, y, Mode) represents space visual characteristic saliency of current coding macroblock, (x, y) represents coordinate of current coding macroblock,
the setting allocation scheme of video coding resource comprises optimization of macroblock coding of the visual interest area so as to improve real-time performance of video coding;
using fast intra-frame prediction algorithm and gray scaly histogram of macroblock to describe a flat level of macroblock and according to the flat level of macroblock to select possible set of intra-frame prediction modes;
using fast inter-frame prediction algorithm, through analyzing statistical characteristics of probability of occurrence of inter-frame prediction modes, to prejudge specific inter-frame prediction modes so that unnecessary inter-frame prediction mode search and calculation of RD cost can be stopped in advance;
using motion estimation search algorithm, based on relevance of macroblock motion vector, according to coding block movement, to identify search level, realizing high efficient search;
wherein the fast inter-frame prediction algorithm comprises:
step 1, pre judging of skip mode;
step 1.1, calculate rate-distortion cost value Jskip of skip mode, if it is lower than a threshold value T, stop searching for other modes, choose skip as best prediction mode, then skip to step 4, otherwise, pass to step 1.2,
where:
T=(0.7−Min_cos t/18000)×Min_cos t, Min_cos t represents best rate-distortion cost value of pervious code macroblock;
step 1.2, calculate rate-distortion cost value J16×16 of Inter16×16 mode, if J16×16>Jskip, choose skip as best coding mode, then skip to step 4, otherwise pass to step 2;
step 2, the judgment of inter-frame prediction mode of macroblock/sub-block;
step 2.1, calculate rate-distortion cost value J16×16 and J8×8, of Inter16×16 mode and Inter8×8 mode, if J8×8−J16×16>T0, choose Inter16×16 mode as best inter-frame coding mode, skip to step 4, otherwise pass to step 2.2,
where T0=0.2×Min_cos t, which is an empirical threshold value obtained from experiment data, Min_cos t represents best rate-distortion cost value of pervious code macroblock;
step 2.2, calculate rate-distortion cost value J4×4 of Inter4×4 mode, if J4×4<min(J16×16, J8×8), take inter-frame subsegmentation prediction mode Inter8×8, Inter8×4, Inter4×8 and Inter4×4 as alternative inter-frame prediction mode set,
otherwise, take macroblock level inter-frame prediction mode Inter16×16, Inter16×8, Inter8×16 as alternative inter-frame prediction mode set, and abandon the inter-frame subsegmentation prediction mode,
step 3, the judgment of intra-frame prediction mode;
step 3.1, calculate ABE (Average Boundary Error) and SBE (Summation Boundary Error) of current code macroblock, ABE reflects macroblock time coherence, ABE=SBE/64
SBE
=
∑
i
=
0
15
⁢

Y
orig
⁡
(
x
+
i
,
y
)
-
Y
rec
⁡
(
x
+
i
,
y
-
1
)

+
∑
i
=
0
15
⁢

Y
orig
⁡
(
x
,
y
+
i
)
-
Y
rec
⁡
(
x
-
1
,
y
+
i
)

where Yorig represents pixel value of current macroblock, Yrec represents pixel value of rebuild macroblock, (x, y) represents position coordinates of current macroblock,
step 3.2, calculate AR (Average Rate) of current macroblock, AR reflects macroblock space coherence,
AR=λ Rate/384
λ represents lagrange gene, Rate represents bit number needed by macroblock,
step 3.3, compare ABE and AR of macroblock, if ABE<C·AR (C=0.95), it indicates that space redundancy is lower than time redundancy, stop traverse calculation of intra-frame prediction modes, skip to step 4, otherwise, keep traverse calculation of intra-frame prediction modes, and enter step 4;
step 4, based on rate-distortion rule, calculate and choose best inter-frame prediction mode, complete inter-frame prediction coding.

254 more elements

A visual perception characteristics-combining hierarchical video coding method comprising:

setting priority of visual interest area and setting allocation scheme of video coding resource;
wherein:
the setting priority of visual interest area mainly is:
calculation formula for labeling visual characteristic saliency area is expressed as:
ROI
⁡
(
x
,
y
)
=
{
3
,
(
(
T
⁡
(
x
,
y
,
MV
)
=
2
⁢
orT
⁡
(
x
,
y
,
MV
)
=
1
)
||
S
⁡
(
x
,
y
,
Mode
)
=
1
)
)
⁢
orS
⁡
(
x
,
y
,
Mode
)
=
2
2
,
(
T
⁡
(
x
,
y
,
MV
)
=
2
⁢
orT
⁡
(
x
,
y
,
MV
)
=
1
||
S
⁡
(
x
,
y
,
Mode
)
=
0
1
,
T
⁡
(
x
,
y
,
MV
)
=
0
||
S
⁡
(
x
,
y
,
Mode
)
=
1
0
,
T
⁡
(
x
,
y
,
MV
)
=
0
)
||
(
S
⁡
(
x
,
y
,
Mode
)
=
0
where ROI(x, y) represents visual interest priority of current coding macroblock, T(x, y,MV) represents time visual characteristic saliency of current coding macroblock, S(x, y, Mode) represents space visual characteristic saliency of current coding macroblock, (x, y) represents coordinate of current coding macroblock,
the setting allocation scheme of video coding resource comprises optimization of macroblock coding of the visual interest area so as to improve real-time performance of video coding;
using fast intra-frame prediction algorithm and gray scaly histogram of macroblock to describe a flat level of macroblock and according to the flat level of macroblock to select possible set of intra-frame prediction modes;
using fast inter-frame prediction algorithm, through analyzing statistical characteristics of probability of occurrence of inter-frame prediction modes, to prejudge specific inter-frame prediction modes so that unnecessary inter-frame prediction mode search and calculation of RD cost can be stopped in advance;
using motion estimation search algorithm, based on relevance of macroblock motion vector, according to coding block movement, to identify search level, realizing high efficient search;
wherein the motion estimation search algorithm comprises,
step 1, description of macroblock motion characteristics,
step 1.1, based on rate-distortion rule, calculate motion estimation rate-distortion cost value RD costmotion of current macroblock,
Jmotion(mv,ref|λmotion)=SAD[s,r(ref,mv)]+λmotion[R(mv−pred)+R(ref)]
where s represents current pixel value of current macroblock, mv represents macroblock motion vector, pred represents prediction vector, ref represents selected reference frame, r(ref,mv) represents pixel value of current reference macroblock, R represents consuming bit number when motion vector is doing differential encoding, including coding bit number which is the difference between motion vector and its' prediction value and reference coding bit number, λmotion represents lagrange multipliers SAD represents sum of absolute differences between current block and reference block which is defined as:
SAD
⁡
(
s
,
c
⁡
(
m
)
)
=
∑
x
=
1
,
y
=
1
M
,
N
⁢

s
⁡
[
x
,
y
]
-
c
⁡
[
x
-
m
x
,
y
-
m
y
]

M,N represents width and height of current macroblock, x,y represents macroblock position, s represents real value, c represents predictive value, m=(mx,my)T represents macroblock motion vector, mx and my represents macroblock motion vector in horizontal and vertical directions,
step 1.2, based on rate-distortion rule, calculate rate-distortion cost value RD costmode in mode scheme,
Jmode(s,c, mod e|λmode)=SSD(s,c, mod e|QP)+λmode×R(s,c, mod e|QP)
where mode represents inter-frame coding mode of current macroblock, s represents origin video signals, c represents rebuild video signals after mode coding, λmode represents lagrange multipliers, R(s,c, mode|QP) represents sum of binary number including macroblock head information, motion vector and all DCT block information related to scheme and quantization parameter, QP represents coding quantization step, SSD(s,c, mode) represents sum of difference of two squares between origin signals and rebuild signals,
SSD
⁡
(
s
,
c
,
mod
⁢
⁢
e
|
QP
)
=
∑
i
=
1
,
j
=
1
B
,
B
2
⁢
(
s
Y
⁡
[
x
,
y
]
-
c
Y
⁡
[
x
,
y
,
mod
⁢
⁢
e
|
QP
]
)
2
+
∑
i
=
1
,
j
=
1
B
,
B
2
⁢
(
s
U
⁡
[
x
,
y
]
-
c
U
⁡
[
x
,
y
,
mod
⁢
⁢
e
|
QP
]
)
2
+
∑
i
=
1
,
j
=
1
B
,
B
2
⁢
(
s
V
⁡
[
x
,
y
]
-
c
V
⁡
[
x
,
y
,
mod
⁢
⁢
e
|
QP
]
)
2
where B1 and B2 represent pixel number in horizontal and vertical direction of coding block, respectively, it is one of 1684;
sY[x, y]cY[x, y, mode|QP] represent value of brightness signals between origin video and rebuild video, cU, cV and sU, sV represent corresponding value of color difference signals,
step 1.3, select minimum rate-distortion cost value RD_min cos t from RD costmotion and RD costmode;
step 2, judgment of severe degree of macroblock motion,
calculation formula for the judgment of severe degree of macroblock motion is expressed as:
{
RD_min
⁢
⁢
cos
⁢
⁢
t
<
(
1
+
γ
)
×
pred_min
⁢
⁢
cos
⁢
⁢
t
low
⁢
⁢
motion
⁢
⁢
degree
(
1
+
γ
)
×
pred_min
⁢
⁢
cos
⁢
⁢
t
<
RD_min
⁢
⁢
cos
⁢
⁢
t
<
(
1
+
δ
)
×
pred_min
⁢
⁢
cos
⁢
⁢
t
medium
⁢
⁢
motion
degree
RD_min
⁢
⁢
cos
⁢
⁢
t
>
(
1
+
δ
)
×
pred_min
⁢
⁢
cos
⁢
⁢
t
high
⁢
⁢
motion
⁢
⁢
degree
where γ,δ represent tuning factor of macroblock motion degree, defined as:
γ
=
Bsize
⁡
[
blocktype
]
pred_min
⁢
⁢
cos
⁢
⁢
t
2
-
α
1
⁡
[
blocktype
]
,
δ
=
Bsize
⁡
[
blocktype
]
pred_min
⁢
⁢
cos
⁢
⁢
t
2
-
α
2
⁡
[
blocktype
]
where Bsize[blocktype] represents size of current macroblock, which is one of 16×1616×88×168×88×44×84×4;
pred_min cos t is determined by motion vector prediction mode which is chosen by origin search point of UMHexagonS algorithm,
(1) if origin search point uses time prediction motion vector, pred_min cos t chooses reference frame motion vector;
(2) if origin search point does not use time prediction motion vector, then:
(2.1) if current motion estimation macroblock inter-frame prediction mode is one of large size inter-frame prediction mode 16×1616×8 or 8×16, pred_min cos t chooses median prediction motion vector;
(2.2) if current motion estimation macroblock inter-frame prediction mode is one of small size inter-frame prediction mode 8×88×44×8 or 4×4, pred_min cos t chooses upper layer prediction motion vector,
based on experimental data, define array α1[blocktype] and array α2[blocktype] respectively:
α1[blocktype]=[−0.23, −0.23, −0.23, −0.25, −0.27, −0.27, −0.28]
α2[blocktype]=[−2.39, −2.40, −2.40, −2.41, −2.45, −2.45, −2.48]
step 3, determine macroblock motion and estimate search hierarchical,
step 3.1 when macroblock motion degree is low, performing hexagonal lattice grid search from layer1 to layer3 in step of non-uniform 4 layers of hexagonal lattice grid search of the UMHexagonS algorithm,
step 3.2 when macroblock motion degree is medium, performing hexagonal lattice grid search from layer1 to layer3 in step of non-uniform 4 layers of hexagonal lattice grid search of the UMHexagonS algorithm,
step 3.3:
when macroblock motion degree is high, performing 4 layers hexagonal lattice grid search in step of non-uniform 4 layers of hexagonal lattice grid search of the UMHexagonS algorithm.

514 more elements

Dependencies are inferred from the claim text.

Facts

Application: 15/124,672Filed 31 Dec 2015
Priority: CN 2015 1 0025201120 Jan 2015
Published: US 2017/0085892 A123 Mar 2017
Granted: 4 Jun 2019
Term ends: Not estimatedThe source records don’t support an estimate.

Classification16 codes

H04N 19/36
Using hierarchical techniques, e.g. scalability: scalability techniques involving formatting the layers as a function of picture distortion after…
H04N 19/10
Methods or arrangements for coding, decoding, compressing or decompressing digital video signals: using adaptive coding
H04N 19/103
Characterised by the element, parameter or selection affected or controlled by the adaptive coding: selection of coding mode or of prediction mode
H04N 19/11
Selection of coding mode or of prediction mode: among a plurality of spatial predictive coding modes

Family

US 2017/0085892 A1Pre-grant publication · published 23 Mar 2017
US 10,313,692 B2This patent

From the US filing history. Foreign family members aren’t listed.

Timeline

Citations

Open elsewhere

Building an agent or pulling many patents? Use the API or MCP server instead of scraping pages. Get an API key

patentgrep

Loading patent…

Abstract

A visual perception characteristics-combining hierarchical video coding method comprising:

setting priority of visual interest area and setting allocation scheme of video coding resource;
wherein:
the setting priority of visual interest area mainly is:
calculation formula for labeling visual characteristic saliency area is expressed as:
ROI
⁡
(
x
,
y
)
=
{
3
,
(
(
T
⁡
(
x
,
y
,
MV
)
=
2
⁢
⁢
or
⁢
⁢
T
⁡
(
x
,
y
,
MV
)
=
1
)
||
S
⁢
(
x
,
y
,
Mode
)
=
1
)
)
⁢
orS
⁡
(
x
,
y
,
Mode
)
=
2
2
,
(
T
⁡
(
x
,
y
,
MV
)
=
2
⁢
⁢
or
⁢
⁢
T
⁡
(
x
,
y
,
MV
)
=
1
)
||
S
⁡
(
x
,
y
,
Mode
)
=
0
1
,
T
⁡
(
x
,
y
,
MV
)
=
0
||
S
⁡
(
x
,
y
,
Mode
)
=
1
0
,
T
⁡
(
x
,
y
,
MV
)
=
0
)
||
(
S
⁡
(
x
,
y
,
Mode
)
=
0
where ROI(x, y) represents visual interest priority of current coding macroblock, T(x, y,MV) represents time visual characteristic saliency of current coding macroblock, S(x, y,Mode) represents space visual characteristic saliency of current coding macroblock, (x, y) represents coordinate of current coding macroblock,
the setting allocation scheme of video coding resource comprises optimization of macroblock coding of the visual interest area so as to improve real-time performance of video coding;
using fast intra-frame prediction algorithm and gray scaly histogram of macroblock to describe a flat level of macroblock and according to the flat level of macroblock to select possible set of intra-frame prediction modes;
using fast inter-frame prediction algorithm, through analyzing statistical characteristics of probability of occurrence of inter-frame prediction modes, to prejudge specific inter-frame prediction modes so that unnecessary inter-frame prediction mode search and calculation of RD cost can be stopped in advance;
using motion estimation search algorithm, based on relevance of macroblock motion vector, according to coding block movement, to identify search level, realizing high efficient search;
wherein the setting priority of visual interest area comprises:
labeling time visual characteristic saliency area, which includes step 1 and step 2, step 1 being motion vector noise detection and step 2 being translation motion vector detection, the motion vector noise detection and the translation motion vector detection are used for reduction of the influence to accuracy of time visual saliency area measurement due to motion vector noise and translation motion vector generated by camera motion respectively, and to perform separation between foreground and background,
labeling space visual characteristic saliency area, and
labeling, according to labeling results of the time visual characteristic saliency area and the space visual characteristic saliency area, the visual characteristic saliency area;
wherein:
when labeling the time visual characteristic saliency area, calculation formula for the motion vector noise detection in step 1 is expressed as:
T
1
⁡
(
x
,
y
,
MV
)
=
{
3
,
if
⁢
⁢

V
rr
_

=
0
2
,
elseif
⁢
⁢

V
→
s

≥

V
rr
_

T
2
⁡
(
x
,
y
,
MV
)
esle
(
1
)
where (x, y) in (1) represents position coordinates, {right arrow over (V)}s represents motion vector of current macroblock, Vrr is average vector of motion reference area Crr, defined as:
V
_
rr
=
∑
∈
C
rr
⁢
v
rr
→
num
C
rr
{right arrow over (v)}rr represents macroblock motion vector in Crr, numCrr represents accumulation times
Crr is defined so that shape, position and area of reference area Crr can be adaptively adjusted with the change of motion vector {right arrow over (V)}s
four macroblocks located in upper right, upper left, lower right and lower left of Crr are MB1, MB2, MB3, MB4, position coordinates of MB1, MB2, MB3, MB4 are defined as:
MB
k
⁡
(
x
k
,
y
k
)
k
=
1
,
2
,
3
,
4
=
{
MB
1
⁡
(
x
-
j
c
,
y
-
i
c
)
MB
2
⁡
(
x
-
j
c
,
y
+
i
c
)
MB
3
⁡
(
x
+
j
c
,
y
-
i
c
)
MB
4
⁡
(
x
+
j
c
,
y
+
i
c
)
⁢
⁢
where
⁢
⁢
i
c
=
[

V
sx
→

w
s
+
1
]
,
j
c
=
[

V
sy
→

h
s
+
1
]
,

V
sx
→

,
and
⁢
⁢

V
sy
→

are motion amplitude of current motion vector {right arrow over (V)}s in horizontal and vertical, ws and hs represent width and height of current code block, respectively, [⋅] represents integer calculation,
if |Vrr|=0, it indicates that there is no motion vector in Crr, so that {right arrow over (V)}s is caused by motion noise and thus is removed, and {right arrow over (V)}s is set as 0, and labeled as T1(x, y,MV)=3,
if |{right arrow over (V)}s|≥|Vrr|, it indicates that current code block has higher motion characteristic saliency than adjacent macroblocks, which belongs to foreground dynamic area, and labeled as T1(x, y,MV)=2,
otherwise, current code block has substantially the same motion characteristic saliency as adjacent macroblocks, then translation motion vector detection is performed to confirm whether the current code block belongs to background area or foreground translation area, which is labeled as T2 (x, y,MV),
calculation formula of translation motion vector detection of step 2 is:
T
2
⁡
(
x
,
y
,
MV
)
=
{
1
,
if
⁢
⁢
SAD
(
x
,
y
)
≥
S
_
c
0
,
else
(
2
)
where (x,y) represents position coordinates of current code block, Sc represents dynamic threshold, SAD(x,y) represents sum of absolute differences between current code block and corresponding station block in previous frame, used to characterize the change between two adjacent corresponding code blocks, defined as:
SAD
(
x
,
y
)
=
∑
i
=
1
M
⁢
∑
j
=
1
N
⁢

s
⁡
(
i
,
j
)
-
c
⁡
(
i
,
j
)

where s(i, j) represents pixel value of current code block, c(i, j) represents corresponding pixel value in pervious frame, MN represents length and wide of current code block respectively,
Sc represents average value of SAD of all code blocks in background area of pervious frame, defined as:
S
_
c
=
∑
x
,
y
∈
S
c
⁢
SAD
(
x
,
y
)
Num
Sc represents background area of pervious frame,
∑
x
,
y
∈
S
c
⁢
SAD
(
x
,
y
)
represents the sum of SAD value of current code blocks in ScNum represents accumulation times,
formula for labeling the time visual characteristic saliency area is expressed as:
T
⁡
(
x
,
y
,
MV
)
=
{
3
,
if
⁢
⁢

V
rr
_

=
0
2
,
elseif
⁢
⁢

V
s
→

≥

V
rr
_

1
,
elseif
⁢
⁢
SAD
(
x
,
y
)
≥
S
_
c
0
,
else
(
3
)
where parameters of (3) are same with (1) and (2)
then, performing labeling of the space visual characteristic saliency area, calculation formula is expressed as:
S
⁡
(
x
,
y
,
Mode
)
=
{
2
,
Mode
P
∈
{
Intra
}
1
,
Mode
P
∈
{
Inter
⁢
⁢
8
}
⁢
⁢
or
⁢
⁢
Mode
I
∈
Intra
⁢
⁢
4
×
4
0
,
Mode
P
∈
{
Inter
⁢
⁢
16
}
⁢
⁢
or
⁢
⁢
Mode
I
∈
Intra
⁢
⁢
16
×
16
(
4
)
where (x, y) represents position coordinates of current code block, Mode represents prediction pattern of current code block, modeP represents prediction pattern of current code block in P frame of code, modeI represents prediction pattern of current code block in I frame of code,
if modeP chooses intra-frame prediction mode, it indicates that space visual characteristic saliency is the highest, which belongs to sensitive area, and is labeled as S(x, y, Mode)=2;
if modeP chooses a set of sub-block prediction modes, Inter8(8×8, 8×4, 4×8, 4×4) or Intra4×4 prediction mode, it indicates that spatial detail is rich, and the space visual characteristic saliency is high, which belongs to interest area, and is labeled as:
S(x, y, Mode)=1;
if modeP chooses a set of inter-frame prediction modes, Inter16(Skip, 16×16, 16×8, 8×16) or modeI chooses Intra16×16 prediction mode, it indicates that spatial change is flat, and the space visual characteristic saliency is low, which belongs to non-obvious area, and labeled as S(x, y,Mode)=0;
labeling, according to labeling results of the space visual characteristic saliency area and the time visual characteristic saliency area, the visual characteristic saliency area.

626 more elements

Facts

Application

15/124,672Filed 31 Dec 2015

Priority

CN 2015 1 0025201120 Jan 2015

Published

US 2017/0085892 A123 Mar 2017

Granted

4 Jun 2019

Term ends

Not estimatedThe source records don’t support an estimate.

Classification16 codes

H04N 19/36
Using hierarchical techniques, e.g. scalability: scalability techniques involving formatting the layers as a function of picture distortion after…
H04N 19/10
Methods or arrangements for coding, decoding, compressing or decompressing digital video signals: using adaptive coding
H04N 19/103
Characterised by the element, parameter or selection affected or controlled by the adaptive coding: selection of coding mode or of prediction mode
H04N 19/11
Selection of coding mode or of prediction mode: among a plurality of spatial predictive coding modes