# US 11,113,793 B2 · Method and apparatus for smoothing a motion trajectory in a video

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,113,793 B2
- Document ID: US11113793B2
- Status: Granted
- Kind: B2
- Application number: 16/689,108
- Filed: 2019-11-20
- Published: 2021-05-20
- Granted: 2021-09-07
- Earliest claimed: 2019-11-20
- Applicants: Pacific future technology (shenzhen) co., LTD
- Inventors: Jiali Wang; Kien Yi Lee
- Web page: https://patentgrep.com/patent/US11113793B2

## CPC

- G06T 5/70: Denoising; Smoothing
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20201: Motion blur correction
- G06T 2207/30241: Trajectory
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Disclosed is a method and apparatus for smoothing a motion trajectory in a video. According to this method, an original rotation matrix sequence of a video frame sequence of an input video is acquired. Then an objective function is determined according to the Riemann manifold architecture, and the smooth rotation matrix that minimizes the objective function is iteratively solved for according to the Riemannian manifold constrained smoothing algorithm, thus obtaining a smooth rotation matrix sequence after the smoothing. Then the video frame sequence is subjected to image stabilization based on the smooth rotation matrix sequence, thereby eliminating motion jitter, making the motion in the video smoother, and improving the video quality.

## Claims

### Claim 1

A method for smoothing a motion trajectory in a video, comprising:
obtaining a video frame sequence of an input video, and obtaining an original rotation matrix Rold of each video frame;
obtaining an original rotation matrix sequence xold=[R1old, R2old, . . . , Rnold]T of the video frame sequence that varies with time, according to the original rotation matrix of each video frame Rold;
assuming a smooth rotation matrix sequence obtained after smoothing the original rotation matrix xold is xnew=[R1new, R2new, . . . , Rnnew]T, then a geodesic distance between a smooth rotation matrix Rnnew and a corresponding original rotation matrix Rnold is dg2(Rnold, Rnnew), and a geodesic distance between two adjacent smooth rotation matrices is dg2(Rnnew, Rn+1new);
determining an objective function based on Riemann manifold architecture:
min
{
R
n
new
}
⁢
∑
n
=
1
N
⁢
1
2
⁢
d
g
2
⁡
(
R
n
old
,
R
n
new
)
+
α
⁢
∑
n
=
1
N
-
1
⁢
1
2
⁢
d
g
2
⁡
(
R
n
new
,
R
n
+
1
new
)
iteratively solving for the smooth rotation matrix Rnnew that minimizes the objective function according to Riemannian manifold constrained smoothing algorithm, and further obtaining the smooth rotation matrix sequence xnew; and
performing image stabilization on the video frame sequence, based on the smooth rotation matrix sequence xnew.

### Claim 2 (depends on 1)

The method of claim 1, wherein iteratively solving for the smooth rotation matrix Rnnew that minimizes the objective function according to the Riemannian manifold constrained smoothing algorithm comprises:
iteratively solving for the smooth rotation matrix Rnnew that minimizes the objective function using a steepest gradient descent method to select a direction and using an Armijo rule to select a step size.

### Claim 3 (depends on 2)

The method of claim 2, wherein dg2=(Rnold, Rnnew) is less than a preset value.

### Claim 4

An apparatus for smoothing a motion trajectory in a video, comprising:
a first acquisition module, configured for obtaining a video frame sequence of an input video, and obtaining an original rotation matrix Rold of each video frame;
a second acquisition module, configured for obtaining an original rotation matrix sequence xold=[R1old, R2old, . . . , Rnold]T of the video frame sequence that varies with time, according to the original rotation matrix of each video frame Rold;
a determination module, wherein assuming a smooth rotation matrix sequence obtained after smoothing the original rotation matrix xold is xnew=[R1new, R2new, . . . , Rnnew]T then a geodesic distance between a smooth rotation matrix Rnnew and a corresponding original rotation matrix Rnold is dg2=(Rnold, Rnnew), and a geodesic distance between two adjacent smooth rotation matrices is dg2=(Rnnew, Rn+1new); the determination module is configured for determining an objective function according to Riemann manifold architecture:
min
{
R
n
new
}
⁢
∑
n
=
1
N
⁢
1
2
⁢
d
g
2
⁡
(
R
n
old
,
R
n
new
)
+
α
⁢
∑
n
=
1
N
-
1
⁢
1
2
⁢
d
g
2
⁡
(
R
n
new
,
R
n
+
1
new
)
a third acquisition module, configured for iteratively solving for the smooth rotation matrix Rnnew that minimizes the objective function according to Riemannian manifold constrained smoothing algorithm, and further obtaining the smooth rotation matrix sequence xnew; and
a processing module, configured for performing image stabilization on the video frame sequence based on the smooth rotation matrix sequence xnew.

### Claim 5 (depends on 4)

The apparatus of claim 4, wherein the third acquisition module is configured for iteratively solving for the smooth rotation matrix Rnnew that minimizes the objective function using a steepest gradient descent method to select a direction and using an Armijo rule to select a step size.

### Claim 6 (depends on 5)

The apparatus of claim 5, wherein dg2=(Rnold, Rnnew) is less than a preset value.

### Claim 7

A non-transitory computer readable storage medium, storing at least one instruction, which when executed on one or more processors is configured to perform a method for smoothing a motion trajectory in a video, the method comprising the following operations:
obtaining a video frame sequence of an input video, and obtaining an original rotation matrix Rold of each video frame;
obtaining an original rotation matrix sequence xold=[R1old, R2old, . . . , Rnold]T of the video frame sequence that varies with time, according to the original rotation matrix of each video frame Rold;
assuming a smooth rotation matrix sequence obtained after smoothing the original rotation matrix xold is xnew=[R1new, R2new, . . . , Rnnew]T, then a geodesic distance between a smooth rotation matrix Rnnew and a corresponding original rotation matrix Rnold is dg2(Rnold, Rnnew), and a geodesic distance between two adjacent smooth, rotation matrices is dg2(Rnnew, Rn+1new);
determining an objective function based on Riemann manifold architecture:
min
{
R
n
new
}
⁢
∑
n
=
1
N
⁢
1
2
⁢
d
g
2
⁡
(
R
n
old
,
R
n
new
)
+
α
⁢
∑
n
=
1
N
-
1
⁢
1
2
⁢
d
g
2
⁡
(
R
n
new
,
R
n
+
1
new
)
iteratively solving for the smooth rotation matrix Rnnew that minimizes the objective function according to Riemannian manifold constrained smoothing algorithm, and further obtaining the smooth rotation matrix sequence xnew; and
performing image stabilization on the video frame sequence based on the smooth rotation matrix sequence xnew.

### Claim 8 (depends on 7)

The non-transitory computer readable storage medium of claim 7, wherein iteratively solving for the smooth rotation matrix Rnnew that minimizes the objective function according to the Riemannian manifold constrained smoothing algorithm comprises:
iteratively that solving for the smooth rotation matrix Rnnew that minimizes the objective function using a steepest gradient descent method to select a direction and using an Armijo rule to select a step size.

### Claim 9 (depends on 8)

The non-transitory computer readable storage medium of claim 8, wherein dg2=(Rnold, Rnnew) is less than a preset value.