Method and device for video image processing, calculating the similarity between video frames, and acquiring a synthesized frame by synthesizing a plurality of contiguous sampled frames

A video image synthesis method comprising the steps of:

• sampling three or more contiguous frames from a video image;
• placing a reference patch comprising one or a plurality of rectangular areas on one of said three or more frames which is used as a reference frame, then respectively placing on the others of said three or more frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames coincides with an image within said reference patch, and respectively estimating correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch;
• acquiring a plurality of first interpolated frames whose resolution is higher than each of said frames, by performing interpolation either on the image within the patch of each of said other frames or on the image within the patch of each of said other frames and image within said reference patch of said reference frame, based on said correspondent relationships;
• acquiring one or a plurality of second interpolated frames whose resolution is higher than each of said frames and which are correlated with said plurality of first interpolated frames, by performing interpolation on the image within said reference patch of said reference frame;
• acquiring a plurality of coordinate-transformed frames by transforming coordinates of the images within the patches of said other frames to a coordinate space of said reference frame, based on said correspondent relationships;
• computing correlation values that represent a correlation between the image within the patch of each of said coordinate-transformed frames and the image within said reference patch of said reference frame;
• acquiring weighting coefficients that make a weight of each of said first interpolated frame greater as said correlation becomes greater, when synthesizing said each of said first interpolated frame and said second interpolated frame, based on said correlation values;
• and
• acquiring intermediate synthesized frames by weighting and synthesizing said each of said first interpolated frames and said second interpolated frames that correspond to each other on the basis of said weighting coefficients, and acquiring a synthesized frame by synthesizing said intermediate synthesized frames.

A video image synthesizer comprising:

• sampling means for sampling two contiguous frames from a video image;
• correspondent relationship estimation means for placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;
• first interpolation means for acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relationship;
• second interpolation means for acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;
• coordinate transformation means for acquiring a coordinate-transformed frame by transforming coordinates of the image within said second patch of said other frame to a coordinate space of said reference frame, based on said correspondent relationship;
• correlation-value computation means for computing a correlation value that represents a correlation between the image within the patch of said coordinate-transformed frame and the image within said reference patch of said reference frame;
• weighting-coefficient acquisition means for acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said correlation becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said correlation value;
• and
• synthesis means for acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient;
• wherein a correlation value is calculated for each of pixels and/or each local region that constitute the other frame, and
• said synthesizer further comprises means for filtering said correlation values to compute filtered correlation values, and said weighting-coefficient acquisition means acquires said weighting coefficient, based on said filtered correlation values.

A video image synthesizer comprising:

• sampling means for sampling three or more contiguous frames from a video image;
• correspondent relationship estimation means for placing a reference patch comprising one or a plurality of rectangular areas on one of said three or more frames which is used as a reference frame, then respectively placing on the others of said three or more frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames coincides with an image within said reference patch, and respectively estimating correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch;
• first interpolation means for acquiring a plurality of first interpolated frames whose resolution is higher than each of said frames, by performing interpolation either on the image within the patch of each of said other frames or on the image within the patch of each of said other frames and image within said reference patch of said reference frame, based on said correspondent relationships;
• second interpolation means for acquiring one or a plurality of second interpolated frames whose resolution is higher than each of said frames and which are correlated with said plurality of first interpolated frames, by performing interpolation on the image within said reference patch of said reference frame;
• coordinate transformation means for acquiring a plurality of coordinate-transformed frames by transforming coordinates of the images within the patches of said other frames to a coordinate space of said reference frame, based on said correspondent relationships;
• correlation-value computation means for computing correlation values that represent a correlation between the image within the patch of each of said coordinate-transformed frames and the image within said reference patch of said reference frame;
• weighting-coefficient acquisition means for acquiring weighting coefficients that make a weight of each of said first interpolated frame greater as said correlation becomes greater, when synthesizing said each of said first interpolated frame and said second interpolated frame, based on said correlation values;
• and
• synthesis means for acquiring intermediate synthesized frames by weighting and synthesizing said each of said first interpolated frames and said second interpolated frame that correspond to each other on the basis of said weighting coefficients, and acquiring a synthesized frame by synthesizing said intermediate synthesized frames.

A non-transitory computer readable medium storing a computer program, which when executed by a computer, causes the computer to execute a video image synthesis method comprising:

• a procedure of sampling two contiguous frames from a video image;
• a procedure of placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;
• a procedure of acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relationship;
• a procedure of acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;
• a procedure of acquiring a coordinate-transformed frame by transforming coordinates of the image within said second patch of said other frame to a coordinate space of said reference frame, based on said correspondent relationship;
• a procedure of computing a correlation value that represents a correlation between the image within the patch of said coordinate-transformed frame and the image within said reference patch of said reference frame;
• a procedure of acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said correlation becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said correlation value;
• and
• a procedure of acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient;
• wherein a correlation value is calculated for each pixel and/or each local region that constitutes the other frame, and
• said method further comprises a procedure of filtering said correlation values to compute filtered correlation values, and said weighting-coefficient acquisition procedure is a procedure of acquiring said weighting coefficient, based on said filtered correlation values.

A non-transitory computer readable medium storing a computer program, which when executed by a computer, causes the computer to execute a video image synthesis method comprising:

• a procedure of sampling three or more contiguous frames from a video image;
• a procedure of placing a reference patch comprising one or a plurality of rectangular areas on one of said three or more frames which is used as a reference frame, then respectively placing on the others of said three or more frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames coincides with an image within said reference patch, and respectively estimating correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch;
• a procedure of acquiring a plurality of first interpolated frames whose resolution is higher than each of said frames, by performing interpolation either on the image within the patch of each of said other frames or on the image within the patch of each of said other frames and image within said reference patch of said reference frame, based on said correspondent relationships;
• a procedure of acquiring one or a plurality of second interpolated frames whose resolution is higher than each of said frames and which are correlated with said plurality of first interpolated frames, by performing interpolation on the image within said reference patch of said reference frame;
• a procedure of acquiring a plurality of coordinate-transformed frames by transforming coordinates of the images within the patches of said other frames to a coordinate space of said reference frame, based on said correspondent relationships;
• a procedure of computing correlation values that represent a correlation between the image within the patch of each of said coordinate-transformed frames and the image within said reference patch of said reference frame;
• a procedure of acquiring weighting coefficients that make a weight of each of said first interpolated frame greater as said correlation becomes greater, when synthesizing said each of said first interpolated frame and said second interpolated frame, based on said correlation values;
• and
• a procedure of acquiring intermediate synthesized frames by weighting and synthesizing said each of said first interpolated frames and said second interpolated frame that correspond to each other on the basis of said weighting coefficients, and acquiring a synthesized frame by synthesizing said intermediate synthesized frames.

A video image synthesis method comprising the steps of:

• sampling two contiguous frames from a video image;
• placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;
• acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relationship;
• acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;
• acquiring a coordinate-transformed frame by transforming coordinates of the image within said second patch of said other frame to a coordinate space of said reference frame, based on said correspondent relationship;
• computing a correlation value that represents a correlation between the image within the patch of said coordinate-transformed frame and the image within said reference patch of said reference frame;
• acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said correlation becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said correlation value;
• and
• acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient;
• wherein a correlation value is calculated for each of the pixels and/or each local region that constitute the other frame, and
• a weighting coefficient is interpolated for each correlation value to acquire weighting coefficients for all pixels that constitute said first and second interpolated frames.

A video image synthesizer comprising:

• sampling means for sampling two contiguous frames from a video image;
• correspondent relationship estimation means for placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;
• first interpolation means for acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relationship;
• second interpolation means for acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;
• coordinate transformation means for acquiring a coordinate-transformed frame by transforming coordinates of the image within said second patch of said other frame to a coordinate space of said reference frame, based on said correspondent relationship;
• correlation-value computation means for computing a correlation value that represents a correlation between the image within the patch of said coordinate-transformed frame and the image within said reference patch of said reference frame;
• weighting-coefficient acquisition means for acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said correlation becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said correlation value;
• and
• synthesis means for acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient;
• wherein a correlation value is calculated for each of the pixels and/or each local region that constitute the other frame, and
• said weighting-coefficient acquisition means performs interpolation on a weighting coefficient for each correlation value, thereby acquiring weighting coefficients for all pixels that constitute said first and second interpolated frames.

A non-transitory computer readable medium storing a computer program, which when executed by a computer, causes the computer to execute a video image synthesis method comprising:

• a procedure of sampling two contiguous frames from a video image;
• a procedure of placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;
• a procedure of acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relationship;
• a procedure of acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;
• a procedure of acquiring a coordinate-transformed frame by transforming coordinates of the image within said second patch of said other frame to a coordinate space of said reference frame, based on said correspondent relationship;
• a procedure of computing a correlation value that represents a correlation between the image within the patch of said coordinate-transformed frame and the image within said reference patch of said reference frame;
• a procedure of acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said correlation becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said correlation value;
• and
• a procedure of acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient;
• wherein a correlation value is calculated for each of the pixels and/or each local region that constitute the other frame, and
• said weighting-coefficient acquisition procedure is a procedure of performing interpolation on a weighting coefficient for each correlation value and acquiring weighting coefficients for all pixels that constitute said first and second interpolated frames.

A video image synthesis method comprising the steps of:

• sampling two contiguous frames from a video image;
• placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;
• acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relationship;
• acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;
• acquiring a coordinate-transformed frame by transforming coordinates of the image within said second patch of said other frame to a coordinate space of said reference frame, based on said correspondent relationship;
• computing a correlation value that represents a correlation between the image within the patch of said coordinate-transformed frame and the image within said reference patch of said reference frame;
• acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said correlation becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said correlation value;
• and
• acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient;
• wherein said weighting coefficient is acquired by referring to a nonlinear graph in which said correlation value is represented in the horizontal axis and said weighting coefficient in the vertical axis.

A video image synthesizer comprising:

• sampling means for sampling two contiguous frames from a video image;
• correspondent relationship estimation means for placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;
• first interpolation means for acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relationship;
• second interpolation means for acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;
• coordinate transformation means for acquiring a coordinate-transformed frame by transforming coordinates of the image within said second patch of said other frame to a coordinate space of said reference frame, based on said correspondent relationship;
• correlation-value computation means for computing a correlation value that represents a correlation between the image within the patch of said coordinate-transformed frame and the image within said reference patch of said reference frame;
• weighting-coefficient acquisition means for acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said correlation becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said correlation value;
• and
• synthesis means for acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient;
• wherein said weighting-coefficient acquisition means acquires said weighting coefficient by referring to a nonlinear graph in which said correlation value is represented in the horizontal axis and said weighting coefficient in the vertical axis.

A non-transitory computer readable medium storing a computer program, which when executed by a computer, causes the computer to execute a video image synthesis method comprising:

• a procedure of sampling two contiguous frames from a video image;
• a procedure of placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;
• a procedure of acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relationship;
• a procedure of acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;
• a procedure of acquiring a coordinate-transformed frame by transforming coordinates of the image within said second patch of said other frame to a coordinate space of said reference frame, based on said correspondent relationship;
• a procedure of computing a correlation value that represents a correlation between the image within the patch of said coordinate-transformed frame and the image within said reference patch of said reference frame;
• a procedure of acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said correlation becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said correlation value;
• and
• a procedure of acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient;
• wherein said weighting-coefficient acquisition procedure is a procedure of acquiring said weighting coefficient by referring to a nonlinear graph in which said correlation value is represented in the horizontal axis and said weighting coefficient in the vertical axis.