# US 2022/0092751 A1 · Efficient hardware guided filtering method for use in multi-label problem

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0092751 A1
- Document ID: US20220092751A1
- Status: Published application
- Kind: A1
- Application number: 17/422,455
- Filed: 2020-01-02
- Published: 2022-03-24
- Earliest claimed: 2019-01-18
- Assignees: Nanjing University of Science and Technology
- Applicants: Nanjing University of Science and Technology
- Inventors: Longquan DAI; Jingru WANG; Jinhui TANG
- Web page: https://patentgrep.com/patent/US20220092751A1

## CPC

- G06T 5/20: using local operators
- G06F 17/12: Simultaneous equations {, e.g. systems of linear equations}
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06T 5/70: Denoising; Smoothing
- G06T 5/73: Deblurring; Sharpening
- G06T 2207/10024: Color image
- G06T 2207/20192: Edge enhancement; Edge preservation
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The present invention provides an efficient hardware guided filtering method for use in solving a multi-label problem. The method includes the following steps: inputting an input guidance of a multi-label image; defining an efficient hardware guided filtering (HGF) model; calculating a vector by a customized matrix inversion operation; inputting guidance through a mapping program for adding up result of each channel to form a polynomial guidance, and introducing nonlinearity into the linear model; and obtaining a filtering result in an efficient hardware mode by element-wise calculation and box filtering.

## Claims

### Claim 1

An efficient hardware guided filtering method for use in a multi-label problem comprising:
step 1: inputting the input guidance of a multi-label image;
step 2: defining an efficient hardware guided filtering (HGF) model;
step 3: calculating a vector {right arrow over (wp)} by means of a customized matrix inversion operation;
step 4: inputting guidance through a mapping program for adding up the results of each channel to form a polynomial guidance, and introducing nonlinearity into the linear model; and
step 5: obtaining a filtering result in an efficient hardware mode by means of element-wise calculation and box filtering.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the step 2 comprises the following steps:
step 201: defining the HGF by:
Z(q)=Σi=1n{right arrow over (wp)}(i)Gi(q)+{right arrow over (wp)}(0),∀q∈Ωp (1)
wherein {right arrow over (wp)}(i) is ith coefficient; {right arrow over (wp)}(0) is an initial coefficient; Gi(q) is a recorded value of a pixel q;
step 202: minimizing linear ridge regression (2) to estimate equation (1), wherein coefficient {right arrow over (wp)} of HGF is obtained by:
min
W
p
→
⁢
λ
⁢

w
p
→

2
2
+
∑
q
∈
Ω
p
⁢
(
Y
⁡
(
q
)
-
∑
i
=
1
n
⁢
w
p
→
⁢
(
i
)
⁢
G
i
⁡
(
q
)
-
w
p
→
⁡
(
0
)
)
2
(
2
)
wherein Y(q) is an input image; λ is a regularization parameter; and Ωp is a neighboring region centered on a pixel P;
step 203: obtaining a set of value Z′p(q) of Ωp, adding filtering results of the two pixels together, wherein an average value
1

Ω
q

⁢
∑
p
∈
Ω
q
⁢
Z
p
′
⁡
(
q
)
is regarded as a final filtering result Z(q), wherein
Z
p
′
⁡
(
q
)
=
∑
i
=
1
n
⁢
w
→
i
,
p
⁢
G
i
⁡
(
q
)
+
w
→
0
,
p
,
q
∈
Ω
p
Z
⁡
(
q
)
=
⁢
1

Ω
q

⁢
∑
p
∈
Ω
q
⁢
∑
i
=
1
n
⁢
w
p
→
⁡
(
i
)
⁢
G
i
⁡
(
q
)
+
w
p
→
⁡
(
0
)
=
⁢
∑
i
=
1
n
⁢
w
→
q
a
⁡
(
i
)
⁢
G
i
⁡
(
q
)
+
w
→
q
a
⁡
(
0
)
(
3
)
wherein {right arrow over (w)}i,p is ith coefficient of pixel p, {right arrow over (w)}0,p is an initial coefficient of pixel p, and {right arrow over (w)}qa is a mean value of {right arrow over (w)}p in Ωq.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the step 3 comprises the following steps:
step 301: calculating the vector {right arrow over (wp)} according to the following equation (4) including the customized matrix inversion operation (λE+XPTXp)−1:
{right arrow over (wp)}=(λE+XPTXp)−1XpT{right arrow over (c)}n,p (4)
wherein E is an identity matrix; Xp=[{right arrow over (c)}0,p, . . . , {right arrow over (c)}n,p] is a matrix of input image pixel p, and {right arrow over (c)}n,p is the nth vector of pixel p, and
wherein with respect to ith vector {right arrow over (c)}i,p of {right arrow over (c)}i,p=[Gi(q1), . . . , Gi(q|Ωp|)]T (0≤i≤n+1) is added to a recorded value Gi(qk) of the kth output pixel q, wherein qk∈Ωp, Ωp represents a neighboring region centered on the pixel P; and |Ωp| represents the total number of pixels in Ωp;
step 302: replacing λE+XPTXp by λE+Σi=0n {right arrow over (c)}i,p {right arrow over (c)}i,pT to obtain equation (5), wherein if the equation is invertible then the equation (6) is valid:
w
p
→
=
X
P
T
⁡
(
λ
⁢
E
+
X
P
T
⁢
X
p
)
-
1
⁢
c
→
n
+
1
,
p
=
[
c
→
0
,
p
T
,
…
⁢
,
c
→
n
,
p
T
]
T
⁢
(
λ
⁢
⁢
E
+
∑
i
=
0
n
⁢
c
→
i
,
p
⁢
c
→
i
,
p
T
)
-
1
⁢
c
→
n
+
1
,
p
(
5
)
(
λ
⁢
E
+
∑
i
=
0
n
⁢
c
→
i
,
p
⁢
c
→
i
,
p
T
)
-
1
=
λ
-
1
⁢
E
+
∑
i
,
j
=
0
n
⁢
α
i
⁢
j
,
p
⁢
c
→
i
,
p
⁢
c
→
j
,
p
T
(
6
)
α
i
⁢
j
,
p
k
=
{
γ
p
k
⁢
F
i
⁢
j
,
p
k
+
α
ij
,
p
k
-
1
⁢
i
<
k
,
j
<
k
λ
-
1
⁢
γ
p
k
⁡
(
∑
n
=
0
k
-
1
⁢
α
i
⁢
n
,
p
k
-
1
⁢
G
n
⁢
j
,
p
)
⁢
i
<
k
,
j
=
k
λ
-
1
⁢
γ
p
k
⁡
(
∑
m
=
0
k
-
1
⁢
α
m
⁢
j
,
p
k
-
1
⁢
G
i
⁢
m
,
p
)
⁢
i
=
k
,
j
<
k
λ
-
2
⁢
γ
p
k
i
<
k
,
j
=
k
(
7
)
F
ij
,
p
k
=
∑
m
,
n
=
0
k
-
1
⁢
α
i
⁢
m
,
p
k
-
1
⁢
α
n
⁢
j
,
p
k
-
1
⁢
G
m
⁢
k
,
p
⁢
G
k
⁢
n
,
p
γ
p
k
=
-
(
1
+
λ
-
1
⁢
G
k
⁢
k
,
p
+
∑
m
,
n
=
0
k
-
1
⁢
α
m
⁢
n
,
p
k
-
1
⁢
G
k
⁢
m
,
p
⁢
G
n
⁢
k
,
p
)
-
1
wherein, k, i, j, m, and n represent superscripts and subscripts of a corresponding matrix,
step 303: putting the equation (6) into the equation (5), and converting the kth element {right arrow over (wp)}(k) of {right arrow over (wp)} into a linear combination:
w
p
→
⁡
(
k
)
=
⁢
c
→
k
,
p
T
⁡
(
λ
⁢
E
+
∑
i
=
0
n
⁢
c
→
i
,
p
⁢
c
→
i
,
p
T
)
-
1
⁢
c
→
n
+
1
,
p
=
⁢
c
→
k
,
p
T
⁡
(
λ
-
1
⁢
E
+
∑
i
,
j
=
0
n
⁢
α
i
⁢
j
,
p
⁢
c
→
i
,
p
⁢
c
→
j
,
p
T
)
⁢
c
→
n
+
1
,
p
=
⁢
λ
-
1
⁢
I
k
⁢
n
+
1
,
p
+
Σ
i
,
j
=
0
n
⁢
α
i
⁢
j
,
p
⁢
I
k
⁢
i
,
p
⁢
I
j
⁢
n
+
1
,
p
(
8
)
step 304: obtaining vector inner product result Gij,p and box filtering result Gij(p) of point p:
Gij,p={right arrow over (c)}i,pT{right arrow over (c)}j,p=Σk=1|Ωp|Gi(qk)Gj(qk)=Σq∈ΩpGi(q)Gj(q)
Gij(p)=Σq∈ΩpGi(q)Gj(q)
step 305: if the neighboring region Ωp of p is a box window, then Gij,p=Gij(p), applying the box filter to an element to generate image to obtain Gij, and calculating {right arrow over (wp)}(k) according to a linear combination of Gij to form {right arrow over (wp)}.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the step 4 comprises:
when the input guidance is multi-channel, the mapping program G(i-1)d+j=Iij is applied directly to each channel independently, and results of each channel of the input multi-channel guide I are superimposed to form a polynomial guidance, nonlinearity is assigned to a generalized linear model (1) of the HGF to obtain a nonlinear polynomial model (10):
Z(q)=Σi=1nΣj=1d{right arrow over (wp)}((i−1)d+j)Iij(q)+{right arrow over (wp)}(0) (10)
wherein d is a degree of the polynomial function; Ii represents the ith channel of the multi-channel guide I, and n is the number of channels

### Claim 5 (depends on 1)

The method according to claim 1, wherein the step 5 comprises the following steps:
step 501: representing a box filtering result of image X as (X), Wi and αij recording values of {right arrow over (wp)}(k) and αij,p at any p point in the image region;
step 502: extending equation for calculating {right arrow over (wp)}(k) to those which only use the box filtering result Gij (11) and element-wise arithmetic calculation (12)
G
i
⁢
j
=
ℬ
⁡
(
G
i
⁢
G
j
)
(
11
)
W
i
=
λ
-
1
⁢
G
k
⁡
(
n
+
1
)
+
∑
i
,
j
=
0
n
⁢
α
i
⁢
j
⁢
G
k
⁢
i
⁢
G
j
⁡
(
n
+
1
)
(
12
)
α
ij
k
=
{
F
p
k
+
α
ij
k
-
1
i
<
k
,
j
<
k
λ
-
1
⁢
γ
k
⁡
(
∑
n
=
0
k
-
1
⁢
α
i
⁢
⁢
n
k
-
1
⁢
G
nk
)
i
<
k
,
j
=
k
λ
-
1
⁢
γ
k
⁡
(
∑
n
=
0
k
-
1
⁢
α
mj
k
-
1
⁢
G
k
⁢
⁢
m
)
i
=
k
,
j
<
k
λ
-
2
⁢
γ
k
i
=
j
=
k
(
13
)
wherein, G0 represents all-ones matrix; Gi(1≤i≤n) represents the ith channel of G guided by the synthetic polynomial n channel; Gn+1 represents an input image Y; λ is a regularization parameter; i, j, k, (n+1) represent superscripts and subscripts of a corresponding matrix
a000=−(λ+G00,p)−1
Fk=Σm,n=0k-1αimk-1αnjk-1GmkGkn
γk=−(1+λ−1Gkk+Σm,n=0k-1αmnk-1GkmGnk)−1;
step 503: HGF calculating the filtering result Z according to an average value of coefficient {right arrow over (wp)}, defining an average operator (X)=(X)/(G0), the element-wise arithmetic calculation form of the filtering result calculation equation of equation (3) is expressed as equation (14):
Z=Σi=1n(Wi)Gi+(W0) (14);
step 504, obtaining the filtering result after HGF fast filtering.
