Free viewpoint video generation and interaction method based on deep convolutional neural network

A Free Viewpoint Video (FVV) generation and interaction method based on a deep Convolutional Neural Network (CNN), comprising:

• S1:
• rectifying a pose and a color of each camera in an array of N camera in an acquisition system, wherein the array of N cameras disposed uniformly about a circular arc at a same height, the pose of each camera is rectified based on a reference object at a center of the circular arc, and the position of each camera remains unchanged after rectification;
• and the color of each camera is rectified by a white balance algorithm based on Gray World;
• S2:
• shooting a target scene object in synchronous video sequences by the array of N cameras, and selecting video frames at a certain moment to rectify baselines of N−1 groups of adjacent viewpoints in sequence, to obtain N−1 image affine transformation matrices Mi, i=1,2, . . . ,n;
• S3:
• rectifying a baseline of each frame data of N−1 groups of adjacent sub-viewpoints in sequence using the affine transformation matrices Mi;
• S4:
• pre-processing binocular datasets through the baseline rectification according to S3, the color rectification according to S1, and a displacement threshold screening based on an optical flow calculation, and then training a virtual viewpoint generation ability of the deep CNN;
• S5:
• inputting data of the baseline rectified in S3 into the deep CNN trained in S4, and outputting 2D images of virtual viewpoints based on a number of reconstructed virtual viewpoints;
• S6:
• constructing an image matrix by stitching together physical viewpoints and the virtual viewpoints in sequence according to positions thereof in a physical space, and labeling Block_Indexes of all viewpoints in the image matrix in sequence;
• and
• S7:
• synthesizing the stitched image matrix obtained in S6 into an FVV at a shooting frame rate of the array of N cameras.