US 9,854,168 B2Granted

One-pass video stabilization

AssigneeFuturewei Technologies, Inc.

InventorsZhenyu Wu, Wei Jiang, John Wus, Hong Heather Yu

Life of this patent

Estimating the term…

Loading patent…

Abstract

A device is disclosed comprising a memory configured for holding video and a processor coupled to the memory. The memory contains computer-executable instructions that, when executed by the processor, cause the device to perform operations to stabilize the video, the operations comprising buffering consecutive original video frames, determining transformation matrices from subsets of the original video frames, wherein the transformation matrices represent estimates of stable camera motion, using the transformation matrices to warp the original video frames and generate video that is stabilized relative to the original video frames, and adjusting a size of a subset of original video frames in response to detecting a condition.

Claims22

Dependent claims

A device, comprising:

a memory configured for storing a buffer of video frames and computer-executable instructions;
and
a processor coupled to the memory, wherein the computer-executable instructions are executed by the processor to cause the device to perform operations to stabilize the video, wherein the operations comprise:
receiving a plurality of consecutive video frames, wherein the plurality of consecutive video frames comprises a current video frame and a plurality of previous video frames;
storing the plurality of consecutive video frames in the buffer;
calculating a global motion estimate for the current video frame, wherein the global motion estimate is a 3×3 transformation matrix, wherein the global motion estimate describes a camera's relative motion between the current video frame and an adjacent video frame, and wherein the adjacent video frame is a video frame that was received before the current video frame;
calculating a long-term camera motion estimate for the current video frame, wherein the long-term camera motion estimate is a 3×3 transformation matrix, wherein the long-term camera motion estimate is a geometric mean of an accumulation of global motion estimates, and wherein the accumulation of global motion estimates comprises the global motion estimate for the current video frame and a calculated global motion estimate for each of the previous video frames in the buffer;
calculating a smoothed long-term camera motion estimate for the current video frame by applying a Kalman filter to the long-term camera motion estimate, wherein the smoothed long-term camera motion estimate is a 3×3 transformation matrix;
and
warping the current video frame according to the smoothed long-term camera motion estimate.

7 more elements

A method of stabilizing video, the method comprising:

receiving a plurality of consecutive video frames, wherein the plurality of consecutive video frames comprises a current video frame and a plurality of previous video frames;
storing the plurality of consecutive video frames in a buffer;
calculating a global motion estimate for the current video frame, wherein the global motion estimate is a 3×3 transformation matrix, wherein the global motion estimate describes a camera's relative motion between the current video frame and an adjacent video frame, and wherein the adjacent video frame is a video frame that was received before the current video frame;
calculating a long-term camera motion estimate for the current video frame, wherein the long-term camera motion estimate is a 3×3 transformation matrix, wherein the long-term camera motion estimate is a geometric mean of an accumulation of global motion estimates, and wherein the accumulation of global motion estimates comprises the global motion estimate for the current video frame and a calculated global motion estimate for each of the previous video frames in the buffer;
calculating a smoothed long-term camera motion estimate for the current video frame by applying a Kalman filter to the long-term camera motion estimate, wherein the smoothed long-term camera motion estimate is a 3×3 transformation matrix;
warping the current video frame according to the smoothed long-term camera motion estimate.

3 more elements

A non-transitory computer-readable storage medium storing programming for a one-pass video stabilization method for execution by a device comprising a memory and a processor, the programming including instructions to:

access a plurality of consecutive video frames stored in a buffer, wherein the plurality of consecutive video frames comprises a current video frame and a plurality of previous video frames;
calculate a global motion estimate for the current video frame, wherein the global motion estimate is a 3×3 transformation matrix, wherein the global motion estimate describes a camera's relative motion between the current video frame and an adjacent video frame, and wherein the adjacent video frame is a video frame that was received before the current video frame;
calculate a long-term camera motion estimate for the current video frame, wherein the long-term camera motion estimate is a 3×3 transformation matrix, wherein the long-term camera motion estimate is the geometric mean of an accumulation of global motion estimates, and wherein the accumulation of global motion estimates comprises the global motion estimate for the current video frame and a calculated global motion estimate for each of the previous video frames in the buffer;
calculate a smoothed long-term camera motion estimate for the current video frame by applying a Kalman filter to the long-term camera motion estimate, wherein the smoothed long-term camera motion estimate is a 3×3 transformation matrix;
and
warp the current video frame according to the smoothed long-term camera motion estimate.

3 more elements

Dependencies are inferred from the claim text.

Facts

Application: 14/642,469Filed 9 Mar 2015 · us provisional application
Priority: 7 Mar 2014
Published: US 2015/0256755 A110 Sep 2015
Granted: 26 Dec 2017
Term ends: Not estimatedThe source records don’t support an estimate.

Classification8 codes

H04N 23/683
Vibration or motion blur correction: performed by a processor, e.g. controlling the readout of an image memory
G06T 7/246
Analysis of motion: using feature-based methods, e.g. the tracking of corners or segments
G06T 7/277
Analysis of motion: involving stochastic approaches, e.g. using Kalman filters
H04N 23/681
For stable pick-up of the scene, e.g. compensating for camera body vibrations: motion detection