Anti-shake method and apparatus for panoramic video, and portable terminal

A non-transitory computer-readable medium having stored thereon, a set of computer-executable instructions for causing a processor to perform steps comprising:

• acquiring, in real time, a current state timestamp, an accelerometer data and an angular velocity of a gyroscope in a portable terminal;
• estimating a rotation from the portable terminal to a world coordinate system using Extended Kalman Filtering combined with both the accelerometer data and the angular velocity;
• synchronizing the timestamp of the gyroscope with a timestamp of a panoramic video frame;
• performing quaternion interpolation in a rotation from the portable terminal to a world coordinate system to obtain a rotation matrix corresponding to the panoramic video frame;
• and
• rotating a panoramic image according to the current rotation matrix to generate a stable video frame;
• wherein the step of estimating a rotation from the portable terminal to a world coordinate system using Extended Kalman Filtering combined with both the accelerometer data and the angular velocity, specifically comprises steps of:
• S1021, calculating an initial-state rotation q0+ and an initial process covariance matrix
• P
• 0
• +
• ,
• q
• 0
• +
• =
• exp
• ⁡
• [
• -
• d
• 0
• ×
• g
• 
• d
• 0
• ×
• g
• 
• *
• a
• ⁢
• cos
• ⁡
• (
• d
• 0
• ·
• g
• )
• ]
• ×
• ,
• where, d0 is a measured initial accelerometer data, g is the gravity vector of the world coordinate system;
• an initial process covariance matrix
• P
• 0
• +
• =
• [
• 2
• ⁢
• 5
• .
• 0
• 0
• 0
• 0
• 2
• ⁢
• 5
• .
• 0
• 0
• 0
• 0
• 2
• ⁢
• 5
• .
• 0
• ]
• ;
• S1022, calculating a State Transition Matrix Φ(ωk) at time K using the angular velocity ωk;
• Φ(ωk)=exp(−[ωk·Δt]x), where, ωk is an angular velocity at time K, Δt represents a sampling time interval of the gyroscope;
• S1023, calculating a State Noise Covariance Matrix Qk, updating an a priori estimate for a state rotation qk− and an a priori estimate for a process covariance matrix pk−;
• Q
• k
• =
• [
• Δ
• ⁢
• ⁢
• t
• 2
• Δ
• ⁢
• ⁢
• t
• 2
• Δ
• ⁢
• ⁢
• t
• 2
• ]
• ,
• where Qk is a State Noise Covariance Matrix;
• qk−=Φ(ωk)*qk−1+, where, qk−1+ is an a posteriori estimation for a state rotation at time K−1;
• Pk−=qk−·Pk−1+·qk−+Qk, where, Pk−1+ is an a posteriori estimation for a process covariance matrix at time K−1;
• S1024, updating an Observation Noise Covariance Matrix Vk from the accelerometer data dk, calculating an Observation Transformation Jacobian Matrix Hk, and calculating an Observation Error ek between the current observation and estimated observation;
• V
• k
• =
• [
• δ
• k
• 2
• δ
• k
• 2
• δ
• k
• 2
• ]
• ,
• where, δk=δkvar+δkmean, δkvar=α∥dk|−|dk−1∥+(1−α)δk−1var,
• δ
• k
• mean
• =
• β
• ⁢
• 
• 
• d
• k
• 
• +
• 1
• 
• d
• k
• 
• -
• 2
• 
• ,
• α is a smoothing factor for an acceleration change, and β is an impact factor for an acceleration module;
• H
• k
• =
• ∂
• h
• ∂
• q
• 
• q
• k
• -
• ,
• where, h is an observation function, h(q,v)=q·g+Ik, g is the gravity vector of the world coordinate system, q is a state rotation, that is, a rotation from the world coordinate system to the gyroscope coordinate system;
• Ik is a measurement noise;
• ek=dk×h(qk−,0);
• S1025, updating an optimal Kalman Gain Matrix Kk at time k;
• Kk=Pk−·HkT(Hk·Pk−·HkT+Vk)−1;
• and
• S1026, updating the a posteriori estimation for a rotation qk+ from the portable terminal to the world coordinate system, and the a posteriori estimation for the process covariance matrix pk+ according to the optimal Kalman Gain Matrix Kk and the Observation Error ek;
• q
• k
• +
• =
• exp
• ⁢
• ⁢
• (
• [
• K
• k
• ·
• e
• k
• ]
• x
• )
• ·
• q
• k
• -
• ,
• ⁢
• P
• k
• +
• =
• exp
• ⁢
• ⁢
• (
• [
• K
• k
• ·
• e
• k
• ]
• x
• )
• ·
• P
• k
• -
• ·
• exp
• ⁢
• ⁢
• (
• [
• K
• k
• ·
• e
• k
• ]
• x
• )
• .

A portable terminal, comprising:

• one or more processors;
• a memory;
• and
• one or more computer programs, wherein the one or more computer programs are stored in the memory and are configured to be executed by the one or more processors to perform steps comprising:
• acquiring, in real time, a current state timestamp, an accelerometer data and an angular velocity of a gyroscope in a portable terminal;
• estimating a rotation from the portable terminal to a world coordinate system using Extended Kalman Filtering combined with both the accelerometer data and the angular velocity;
• synchronizing the timestamp of the gyroscope with a timestamp of a panoramic video frame;
• performing quaternion interpolation in a rotation from the portable terminal to a world coordinate system to obtain a rotation matrix corresponding to the panoramic video frame;
• and
• rotating a panoramic image according to the current rotation matrix to generate a stable video frame;
• wherein, after the step of acquiring, in real time, a current state timestamp, an accelerometer data and an angular velocity of a gyroscope in a portable terminal, further comprising a step of:
• reducing noise from the accelerometer data using a low-pass filtering, which further comprises:
• reducing noise from the accelerometer data using a low-pass filtering through Formula d′i=α·di+(1−α)·Ri·d′i−1, where d′i represents the accelerometer data after low-pass filtering at time i, di represents the accelerometer data at time i, and Ri is a relative rotation of the i frame video of the gyroscope;
• Ri=exp[−ωi·Δt]x, where ωi represents the angular velocity at time i, d′i−1 represents the accelerometer data after low-pass filtering at time i−1, and α represents a smoothing factor,
• α
• =
• Δ
• ⁢
• t
• R
• ⁢
• c
• +
• Δ
• ⁢
• t
• ,
• ⁢
• Rc
• =
• 1
• 2
• ⁢
• ⁢
• π
• ⁢
• ⁢
• f
• c
• ,
• where fc represents a cut-off frequency for a low-pass filtering, Rc represents a time constant, and Δt represents a sampling time interval of the gyroscope.