Motion vector calculation method and hand-movement correction device, imaging device and moving picture generation device

A motion vector calculation method in which multiple pieces of image data are input to calculate motion vectors between images, the method comprising:

• processing for inputting specification information relating to feature points of a reference image;
• processing for calculating motion vector mesh data representing correspondence relationships among points, lines and areas between the reference image and a target image based on input specification information;
• processing for dispersedly extracting multiple feature points of the reference image from an area excluding the area range of the input specification information and calculating, as motion vector mesh data, correspondence relationships among points, lines and areas between the reference image and the target image based on the extracted feature points;
• processing for dividing the area of the motion vector mesh data into one or more triangles, correcting the motion vector data to a predetermined resolution level L and calculating input motion vector mesh data;
• processing for calculating and storing multi-resolution data up to the resolution level L of the input multiple pieces of image data;
• and
• processing for estimating motion vectors per resolution by using the reference image, target image and input motion vector mesh data of the resolution level L, which are calculated by the processings above to perform:
• a step (1) of defining an energy function on each starting point coordinate of the input motion vector mesh data of an area excluding the area range of the input specification information by handling coordinates on the reference image and coordinates on the target image as arguments, and temporarily calculating the motion vector mesh data within predetermined range from the input motion vector mesh data based on the energy function;
• a step (2) of temporarily calculating reliability of the motion vector mesh data based on an amount of strength of an edge or a corner angle detected from a color space at each feature point of the reference image;
• a step (3) of calculating correlation by using a color difference between feature points of the reference image and a difference between the motion vector mesh data temporarily calculated in the step (1);
• and
• a step (4) of finally calculating motion vector mesh data of a the resolution level L by smoothing each of the motion vector mesh data temporarily calculated in the step (1) by weighted averaging by the product of the reliability and the correlation where the smoothing utilizes the reliability temporarily calculated in the step (2) smoothed by weighted averaging by the correlation,
• wherein the motion vector mesh data between the reference image and the target image are calculated by using the finally calculated motion vector mesh data of the resolution level L as input motion vector mesh data of a resolution level L-1, which is a higher resolution, and by repeating the processing for estimating motion vectors per resolution on image data having higher resolutions sequentially.

A computer program product comprising a tangible non-transitory computer readable medium storing a computer-executable program that inputs multiple pieces of image data and calculating motion vectors between the images, the program comprising:

• processing for calculating and storing multi-resolution data up to a predetermined resolution level L (where L is a natural number) of each of the input multiple pieces of image data;
• processing for calculating the input motion vectors for the resolution level L of the reference image;
• and
• processing for estimating motion vectors per resolution by using the reference image, a target image and the input motion vectors of the resolution level L, which are calculated by the processings above, to perform:
• a step (1) of defining an energy function on each starting point coordinate of the input motion vectors by handling a coordinate on the reference image and a coordinate on the target image as arguments, and temporarily calculating motion vectors, within predetermined range from the input motion vectors, based on the energy function;
• a step (2) of temporarily calculating reliability of the motion vectors based on an amount of strength of an edge or a corner angle detected from a color space of the reference image;
• a step (3) of calculating correlation by using a color difference between feature points of the reference image and a difference between the motion vectors temporarily calculated in the step (1);
• and
• a step (4) of finally calculating motion vectors of the resolution level L by smoothing each of the motion vectors temporarily calculated in the step (1) by weighted averaging by the product of the reliability and the correlation where the smoothing utilizes the reliability temporarily calculated in the step (2) smoothed by weighted averaging by the correlation,
• wherein the motion vectors between the reference image and the target image are calculated by using the finally calculated motion vectors of the resolution level L as input motion vectors of a resolution level L-1, which is a higher resolution, and by repeating the processing for estimating motion vectors per resolution on image data having higher resolutions sequentially.