US 8,786,716 B2 · Rolling shutter reduction based on motion sensors

Abstract

This disclosure pertains to devices, methods, and computer readable media for reducing rolling shutter distortion effects in captured video frames based on timestamped positional information obtained from positional sensors in communication with an image capture device. In general, rolling shutter reduction techniques are described for generating and applying image segment-specific perspective transforms to already-captured segments (i.e., portions) of images in a video sequence, so as to counter or compensate for unwanted distortions that occurred during the read out of the image sensor. Such distortions may be due to, for example, the use of CMOS sensors combined with the rapid movement of the image capture device. In contrast to the prior art, rolling shutter reduction techniques described herein may be applied to captured images in real-time or near real-time using positional sensor information and without intensive image processing that would require an analysis of the content of the underlying image data.

Claims25

Dependent claims

A motion sensor-based rolling shutter reduction method, comprising:

obtaining, by a device comprising an image sensor, a sequence of images, wherein each image in the sequence comprises a plurality of sequentially read out rows, and wherein the device has one or more image capture parameters;
selecting a first plurality of rows from a first image in the sequence;
for each of the first plurality of rows:
obtaining motion information corresponding to motion of the device at approximately the time the row was obtained;
and
determining a motion estimate for the row based, at least in part, on the obtained motion information;
defining a first plurality of segments, the first plurality of segments comprising the first image;
for each of the first plurality of segments:
generating a perspective transformation for the segment based, at least in part, on the motion estimates for one or more of the first plurality of rows;
and
applying the generated perspective transformation to the segment to generate a corrected image segment;
assembling the plurality of corrected image segments into a first corrected mage;
and
storing the first corrected image in a memory.

11 more elements

An electronic device, comprising:

an image sensor;
a positional sensor;
a memory communicatively coupled to the image sensor;
a programmable control device communicatively coupled to the memory and the positional sensor,
wherein the memory has computer program code stored thereon for causing the programmable control device to:
receive a plurality of sequential images captured by the image sensor, the electronic device having image capture parameters, wherein each of the plurality of sequential images is associated with values corresponding to the image capture parameters at the time each of the images was captured;
obtain motion information from the positional sensor for each of the plurality of sequential images, wherein the motion information for each image in the plurality of sequential images is obtained at approximately the same time as each image was captured, and wherein the motion information comprises a plurality of motion samples;
divide each image from the plurality of sequential images into a plurality of segments;
generate a perspective transformation for each segment of each image based, at least in part, on one or more motion samples corresponding to the segment and the one or more image capture parameters associated with the image;
apply the generated perspective transformations independently to each segment of each of the plurality of sequential images to substantially remove rolling shutter distortions;
and
store each of the perspective transformed plurality of sequential images in the memory.

9 more elements

Dependencies are inferred from the claim text.

Rolling shutter reduction based on motion sensors

Life of this patent

Rolling shutter reduction based on motion sensors

Life of this patent

Abstract

Claims25