US 9,866,820 B1Granted

Online calibration of cameras

AssigneeAmazon Technologies, Inc.

InventorsAmit Kumar Agrawal, Ilya Vladimirovich Brailovskiy, Sharadh Ramaswamy, Ambrish Tyagi

Life of this patent

Estimating the term…

Loading patent…

Abstract

An electronic device can have two or more pairs of cameras capable of performing three-dimensional imaging. In order to provide accurate disparity information, these cameras should be sufficiently calibrated. Automatic calibration can be performed by periodically capturing images with a pair of front-facing cameras and locating matching facial or other feature points in corresponding images captured by those cameras. Correspondences can be detected between feature points and the corresponding feature points can be normalized and outlier feature points can be rejected. A transformation matrix can be determined using at least a portion of remaining feature points and can be used to determine rotation and translation parameters to correct for misalignment between the cameras. The calibration parameters can be refined or otherwise adjusted, and can be used or stored for use in correcting images subsequently captured by those cameras.

Claims18

Dependent claims

A computing device, comprising:

a computing device processor;
a first camera having a first field of view;
a second camera separated a distance from the first camera, the second camera having a second field of view that at least partially overlaps the first field of view;
a memory device including instructions that, when executed by the computing device processor, cause the computing device to:
capture first image data by the first camera and second image data by the second camera, the first image data and the second image data being captured concurrently;
cause the first image data and the second image data to be analyzed to locate a first representation of an head portion corresponding to a head of a user of the computing device in the first image data and a second representation corresponding to the head portion in the second image data;
analyze the first representation and the second representation using a facial recognition algorithm to determine substantially a same facial feature represented in both the first representation and in the second representation;
determine, based at least in part on camera geometry between the first camera and the second camera, a mapping between the facial feature in the first representation and the facial feature in the second representation to determine corresponding facial features between the first representation and the second representation;
normalize the facial feature in the first representation, to at least partially remove an intrinsic characteristic associated with the first camera;
normalize the facial feature in the second representation, to at least partially remove an intrinsic characteristic associated with the second camera;
generate a calibration model characterizing a rotation and translation between the first camera and the second camera based at least in part on the mapping;
generate an updated calibration model by applying at least a quaternion averaging function to a rotation parameter of the calibration model and by applying at least an infinite impulse response (IIR) function to a translation parameter of the calibration model;
and
apply the updated calibration model to one or more subsequent image data captured by at least one of the first camera or the second camera to reduce misalignment effects in the one or more subsequent image data.

11 more elements

A method, comprising:

receiving first image data and second image data, the first image data from a first camera and the second image data from a second camera, the first camera and the second camera having partially overlapping fields of view;
determining a first location of a feature point represented in the first image data;
normalizing the first location of the feature point represented in the first image data, to at least partially remove an intrinsic characteristic associated with the first camera;
determining a second location of the feature point represented in second image data;
normalizing the second location of the feature point represented in the second image data, to at least partially remove an intrinsic characteristic associated with the second camera;
determining, using at least the first location and the second location, calibration parameters to correct for misalignment between the first camera and the second camera in at least one dimension;
generating smoothed calibration parameters by applying a first smoothing function and a second smoothing function to the calibration parameters, wherein the first smoothing function includes at least a quaternion averaging function, and the second smoothing function includes at least an infinite impulse response (IIR) function;
determining that the first camera and the second camera are misaligned by a threshold amount;
and
generating updated calibration parameters based at least in part on the smoothed calibration parameters to correct for misalignment errors of at least one subsequent image captured by at least one of the first camera or the second camera.

7 more elements

A computing device, comprising:

a computing device processor;
a memory device including instructions that, when executed by the computing device processor, cause the computing device to:
receive first image data and second image data, the first image data from a first camera and the second image data from a second camera, the first camera and the second camera having partially overlapping fields of view;
determine a first location of a feature point represented in the first image data;
normalize the first location of the feature point represented in the first image data, to at least partially remove an intrinsic characteristic associated with the first camera;
determine a second location of the feature point represented in second image data;
normalize the second location of the feature point represented in the second image data, to at least partially remove an intrinsic characteristic associated with the second camera;
determine, using at least the first location and the second location, calibration parameters to correct for misalignment between the first camera and the second camera in at least one dimension;
apply a first smoothing function and a second smoothing function to the calibration parameters to generate updated calibration parameters, wherein the first smoothing function includes at least a quaternion averaging function, and the second smoothing function includes at least an infinite impulse response (IIR) function;
determine that the first camera and the second camera are misaligned by a threshold amount;
and
use the updated calibration parameters to correct for misalignment errors of at least one subsequent image captured by at least one of the first camera or the second camera.

9 more elements

Dependencies are inferred from the claim text.