Joint video stabilization and motion deblurring
AssigneeGoogle Inc.
InventorsFuhao Shi, Mauricio Delbracio, Chia-Kai Liang, Damien Martin Kelly, Peyman Milanfar
Life of this patent
Estimating the term…
Abstract
Systems and methods for real-time image deblur and stabilization can utilize sensor data for estimating motion blur without the high computational cost of image analysis techniques. The estimated motion blur can then be utilized to generate a motion blur kernel for image correction. The systems and methods can further refine the correction by processing the motion blur kernel with a polynomial filter to generate a sharpening kernel. The systems and methods can provide for real-time correction even with minimal to no stabilization masking.
Claims19
A computer-implemented method for improved deblurring of an image, the method comprising:
- generating, by a computing system comprising one or more processors and one or more sensors, sensor data, wherein the sensor data is descriptive of motion occurring during an exposure time of the image;
- determining, by the computing system, an estimated motion blur for the image based on the sensor data;
- determining, by the computing system, masking strength for motion blur masking based on the estimated motion blur;
- determining, by the computing system, a motion blur kernel to correct motion blur for the image captured during the exposure time based on the estimated motion blur;
- processing, by the computing system, the motion blur kernel with one or more polynomial filters to generate a sharpening kernel, wherein the sharpening kernel comprises one or more terms for controlling at least one of frequency amplification or noise amplification;
- and
- generating, by the computing system, an augmented image by applying the sharpening kernel and the motion blur masking to the image.
A computing system, the system comprising:
- a plurality of sensors comprising one or more motion sensors and one or more image sensors;
- one or more processors;
- and
- one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
- receiving a user input to capture an image with the one or more image sensors;
- generating image data during an exposure time in response to the user input;
- generating sensor data with the one or more motion sensors;
- processing the sensor data with one or more machine-learned models to generate a projection matrix and a virtual camera trajectory;
- determining an estimated motion blur based on the projection matrix and the virtual camera trajectory;
- generating a motion blur kernel based at least in part on the estimated motion blur;
- generating a joint kernel comprising the motion blur kernel to reduce judder effect and a sharpening kernel for camera motions along directions other than a main panning direction;
- adjusting motion blur masking strength based at least in part on the projection matrix;
- and
- generating augmented image data based on the motion blur masking strength and by applying the joint kernel to the image data.
One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations comprising:
- obtaining image data and motion data, wherein the image data comprises an image captured during an exposure time, and wherein the motion data is descriptive of a camera motion during the exposure time;
- processing the image data and the motion data with an image correction model to generate augmented image data, wherein the augmented image data comprises an augmented image, wherein the image correction model is trained to:
- generate an estimated motion blur masking based on a stabilized virtual camera pose;
- generate an estimated frame deblur based on a motion blur kernel to generate a sharpening kernel using one or more polynomial filters, wherein the sharpening kernel comprises at least one of a frequency amplifier or a noise amplifier for frequencies that are attenuated by blur;
- and
- correct an image based at least in part on the estimated motion blur masking and the sharpening kernel;
- and
- providing the augmented images to a user.
Dependencies are inferred from the claim text.