Free-viewpoint video generating method and free-viewpoint video generating system

A free-viewpoint video generating method comprising:

• obtaining a multi-viewpoint video including N videos of N viewpoints, N being an integer greater than 1, the N videos having been captured by N cameras;
• calculating N camera parameters of the N cameras based on the N videos;
• and
• generating a free-viewpoint video of a virtual viewpoint based on the multi-viewpoint video such that the free-viewpoint video shows target areas and has a first resolution higher than each of resolutions of the N videos, the virtual viewpoint being different from the N viewpoints, for each target area, in order to achieve the first resolution higher than each of the resolutions of the N videos, the generating comprising:
• converting the N videos to N parallelized videos based on the N camera parameters such that the N parailelized videos have view field directions that are parallel to each other;
• performing block matching on the N parallelized videos to specify N reference areas that fit to each other in the N parallelized videos, the N reference areas each corresponding to the each target area;
• calculating components of pixels in the free-viewpoint video to represent the each target area based on components of pixels of the N reference areas;
• and
• converting the each target area in frequency to perform Phase Only Correlation (POC) when the N reference areas are specified with sub-pixel accuracy.

A free-viewpoint video generating system comprising:

• a memory;
• and
• a circuit configured to perform:
• obtaining a multi-viewpoint video including N videos of N viewpoints, N being an integer greater than 1;
• and
• generating a free-viewpoint video of a virtual viewpoint based on the multi-viewpoint video such that the free-viewpoint video shows target areas and has a first resolution higher than each of resolutions of the N videos, the virtual viewpoint being different from the N viewpoints, for each target area, in order to achieve the first resolution higher than each of the resolutions of the N videos, the generating comprising:
• referring to a three-dimensional model generated based on the N videos to specify N reference areas which are shown by the N videos and which correspond to the each target area in the free-viewpoint video;
• and
• calculating components of pixels to represent the each target area in the free-viewpoint video, based on components of pixels to represent the N reference areas in the N videos, wherein for each target area, the generating further comprises:
• specifying a three-dimensional area in the three-dimensional model, the three-dimensional area corresponding to the each target area;
• specifying the N reference areas with sub-pixel accuracy such that the N reference areas each correspond to the three-dimensional area, wherein, in the generating, the N videos included in the multi-viewpoint video are parallelized by converting the N videos included in the multi-viewpoint video into N videos with parallel view field directions, the N reference areas corresponding to the each target area are specified in the N parallelized videos, the components of the pixels to represent the each target area are calculated based on the components of the pixels to represent the N reference areas in the N parallelized videos, and the N videos included in the multi-viewpoint video are parallelized by converting the N videos included in the multi-viewpoint video into N videos with parallel view field directions with respect to a view field direction of the virtual viewpoint;
• and
• converting the each target area in frequency to perform Phase Only Correlation (POC) when the N reference areas are specified with sub-pixel accuracy.

A free-viewpoint video generating system comprising:

• a memory;
• and
• a circuit configured to perform:
• obtaining a multi-viewpoint video including N videos of N viewpoints, N being an integer greater than 1, the N videos having been captured by N cameras;
• calculating N camera parameters of the N cameras based on the N videos;
• and
• generating a free-viewpoint video of a virtual viewpoint based on the multi-viewpoint video such that the free-viewpoint video shows target areas and has a first resolution higher than each of resolutions of the N videos, the virtual viewpoint being different from the N viewpoints, for each target area, in order to achieve the first resolution higher than each of the resolutions of the N videos, the generating comprising:
• converting the N videos to N parailelized videos based on the N camera parameters such that the N parallelized videos have view field directions that are parallel to each other;
• performing block matching on the N parallelized videos to specify N reference areas that fit to each other in the N parallelized videos, the N reference areas each corresponding to the each target area;
• calculating components of pixels in the free-viewpoint video to represent the each target area based on components of pixels of the N reference areas;
• and
• converting the each target area in frequency to perform Phase Only Correlation (POC) when the N reference areas are specified with sub-pixel accuracy.