Image processing apparatus, image processing method, non-transitory computer-readable storage medium for obtaining an amount of shake in an image

An image processing apparatus comprising:

• one or more processors;
• and
• at least one memory coupled to the one or more processors storing instructions that, when executed by the one or more processors, cause the one or more processors to function as:
• a first obtaining unit configured to obtain, based on a first disparity that is based on an initial start point of a motion vector in one image of a stereo image and a point that corresponds to the initial start point in the other image of that the stereo image and a second disparity that is based on an initial end point of the motion vector in the one image and a point that corresponds to the initial end point in the other image, a final start point and a final end point;
• a second obtaining unit configured to obtain, based on the final start point and the final end point, an amount of shake in the stereo image;
• a third obtaining unit configured to obtain, based on a plurality of feature points in the one image and a plurality of feature points in the other image, a calibration value for correcting a vertical shift between the one image and the other image,
• wherein the first obtaining unit obtains the first three-dimensional point, further using the calibration value,
• wherein the first obtaining unit obtains a disparity based on the initial start point of a motion vector in the one image and a point that corresponds to the initial start point in the other image and obtains the first three-dimensional point based on the initial start point and the disparity,
• wherein the first obtaining unit obtains a disparity based on the initial end point of a motion vector in the one image and a point that corresponds to the initial end point in the other image and obtains the second three-dimensional point based on the initial end point and the disparity, and
• wherein the second obtaining unit obtains the second three-dimensional point, further using the calibration value, and
• wherein the second obtaining unit obtains, as the amount of shake, a parameter that minimizes a value that is based on a difference between a three-dimensional point that corresponds to the initial start point that has been obtained by the first obtaining unit and a three-dimensional point for which a three-dimensional point that corresponds to the initial end point that has been obtained by the first obtaining unit has been transformed using that parameter.

An image processing method that is performed by an image processing apparatus, the method comprising:

• obtaining, based on an initial start point of a motion vector in one image of a stereo image and a point that corresponds to the initial start point in the other image of the stereo image, a first three-dimensional point that corresponds to the initial start point;
• obtaining, based on an initial end point of the motion vector in the one image and a point that corresponds to the initial end point in the other image, a second three-dimensional point that corresponds to the initial end point;
• obtaining, based on the first three-dimensional point and the second three-dimensional point, a third three-dimensional point for which a camerawork motion component has been removed from the second three-dimensional point and obtain, based on the third three-dimensional point and the first three-dimensional point, an amount of shake in the stereo image;
• and
• obtaining, based on a plurality of feature points in the one image and a plurality of feature points in the other image, a calibration value for correcting a vertical shift between the one image and the other image,
• where the obtaining obtains the first three-dimensional point, further using the calibration value,
• wherein the obtaining obtains a disparity based on the initial start point of a motion vector in the one image and a point that corresponds to the initial start point in the other image and obtains the first three-dimensional point based on the initial start point and the disparity,
• wherein the obtaining obtains a disparity based on the initial end point of a motion vector in the one image and a point that corresponds to the initial end point in the other image and obtains the second three-dimensional point based on the initial end point and the disparity,
• wherein the obtaining obtains the second three-dimensional point, further using the calibration value, and
• wherein the obtaining obtains, as the amount of shake, a parameter that minimizes a value that is based on a difference between a three-dimensional point for which the first three-dimensional point has been normalized and a three-dimensional point for which a three-dimensional point for which the third three-dimensional point has been normalized has been transformed using that parameter.

An image processing method that is performed by an image processing apparatus, the method comprising:

• obtaining, based on a first disparity that is based on an initial start point of a motion vector in one image of a stereo image and a point that corresponds to the initial start point in the other image of the stereo image and a second disparity that is based on an initial end point of the motion vector in one image and a point that corresponds to the initial end point in the other image, a final start point and a final end point;
• and
• obtaining, based on the obtained start point and end point, an amount of shake in the stereo image;
• obtaining, based on a plurality of feature points in the one image and a plurality of feature points in the other image, a calibration value for correcting a vertical shift between the one image and the other image,
• wherein the obtaining obtains the first three-dimensional point, further using the calibration value,
• wherein the obtaining obtains a disparity based on the initial start point of a motion vector in the one image and a point that corresponds to the initial start point in the other image and obtains the first three-dimensional point based on the initial start point and the disparity,
• wherein the obtaining obtains a disparity based on the initial end point of a motion vector in the one image and a point that corresponds to the initial end point in the other image and obtains the second three-dimensional point based on the initial end point and the disparity,
• wherein the obtaining obtains the second three-dimensional point, further using the calibration value, and
• wherein the obtaining obtains, as the amount of shake, a parameter that minimizes a value that is based on a difference between a three-dimensional point for which a first three-dimensional point has been normalized and a three-dimensional point for which a three-dimensional point for which a third three-dimensional point has been normalized has been transformed using that parameter.

A non-transitory computer-readable storage medium storing a computer program for causing a computer to function as:

• a first obtaining unit configured to obtain, based on an initial start point of a motion vector in one image of a stereo image and a point that corresponds to the initial start point in the other image of the stereo image, a first three-dimensional point that corresponds to the initial start point;
• a second obtaining unit configured to obtain, based on an initial end point of the motion vector in the one image and a point that corresponds to that end point in the other image, a second three-dimensional point that corresponds to the initial end point;
• a third obtaining unit configured to obtain, based on the first three-dimensional point and the second three-dimensional point, a third three-dimensional point for which a camerawork motion component has been removed from the second three-dimensional point and obtain, based on the third three-dimensional point and the first three-dimensional point, an amount of shake in the stereo image;
• and
• a fourth obtaining unit configured to obtain, based on a plurality of feature points in the one image and a plurality of feature points in the other image, a calibration value for correcting a vertical shift between the one image and the other image,
• wherein the first obtaining unit obtains the first three-dimensional point, further using the calibration value,
• wherein the first obtaining unit obtains a disparity based on the initial start point of a motion vector in the one image and a point that corresponds to the initial start point in the other image and obtains the first three-dimensional point based on the initial start point and the disparity,
• wherein the first obtaining unit obtains a disparity based on the initial end point of a motion vector in the one image and a point that corresponds to the initial end point in the other image and obtains the second three-dimensional point based on the initial end point and the disparity,
• wherein the second obtaining unit obtains the second three-dimensional point, further using the calibration value, and
• wherein the third obtaining unit obtains, as the amount of shake, a parameter that minimizes a value that is based on a difference between a three-dimensional point for which the first three-dimensional point has been normalized and a three-dimensional point for which a three-dimensional point for which the third three-dimensional point has been normalized has been transformed using that parameter.

A non-transitory computer-readable storage medium storing a computer program for causing a computer to function as:

• a first obtaining unit configured to obtain, based on a first disparity that is based on an initial start point of a motion vector in one image of a stereo image and a point that corresponds to the initial start point in the other image of the stereo image and a second disparity that is based on an initial end point of the motion vector in one image and a point that corresponds to the initial end point in the other image, a final start point and a final end point;
• a second obtaining unit configured to obtain, based on the final start point and the final end point, an amount of shake in the stereo image;
• and
• a third obtaining unit configured to obtain, based on a plurality of feature points in the one image and a plurality of feature points in the other image, a calibration value for correcting a vertical shift between the one image and the other image,
• wherein the first obtaining unit obtains the first three-dimensional point, further using the calibration value,
• wherein the first obtaining unit obtains a disparity based on the initial start point of a motion vector in the one image and a point that corresponds to the initial start point in the other image and obtains the first three-dimensional point based on the initial start point and the disparity,
• wherein the first obtaining unit obtains a disparity based on the initial end point of a motion vector in the one image and a point that corresponds to the initial end point in the other image and obtains the second three-dimensional point based on the initial end point and the disparity,
• wherein the second obtaining unit obtains the second three-dimensional point, further using the calibration value, and
• wherein the second obtaining unit obtains, as the amount of shake, a parameter that minimizes a value that is based on a difference between a three-dimensional point for which a first three-dimensional point has been normalized and a three-dimensional point for which a three-dimensional point for which a third three-dimensional point has been normalized has been transformed using that parameter.