Image processing method, neural network training method, three-dimensional image display method, image processing system, neural network training system, and three-dimensional image display system

A computer-implemented method of training a neural network which infers residual data based on a plurality of captured images, the residual data representing a difference between a bowl-shaped predefined projection surface and a projection surface reflecting three-dimensional data of one or more three-dimensional objects imaged in the plurality of captured images, the method comprising:

• preparing the plurality of captured images taken by a plurality of respective cameras;
• preparing the three-dimensional data;
• recreating a three-dimensional image from the plurality of captured images based on the three-dimensional data to generate a training image that serves as a free viewpoint image for training based on viewpoint data given as an input;
• and
• inputting the plurality of captured images and the viewpoint data into a neural network to produce learning residual data, generating a learning image that serves as a free viewpoint image for learning by mapping the plurality of captured images onto a display projection surface using the learning residual data, information about the predefined projection surface, and the viewpoint data, and training the neural network such that a difference between the training image and the learning image becomes smaller.

A computer-implemented method of training a neural network which infers residual data based on a plurality of captured images, the residual data representing a difference between a bowl-shaped predefined projection surface and a projection surface reflecting three-dimensional data of one or more three-dimensional objects imaged in the plurality of captured images, the method comprising:

• preparing the plurality of captured images taken by a plurality of respective cameras;
• mapping the plurality of captured images onto the predefined projection surface to generate an uncorrected image that serves as an uncorrected free viewpoint image based on viewpoint data received as an input;
• preparing the three-dimensional data;
• recreating a three-dimensional image from the plurality of captured images based on the three-dimensional data to generate a training image that serves as a training free viewpoint image based on the viewpoint data;
• comparing the uncorrected free viewpoint image and the training image to prepare the residual data;
• and
• inputting the plurality of captured images and the viewpoint data into the neural network to train the neural network using the residual data prepared as training data.

A computer-implemented three-dimensional image display method of synthesizing a free viewpoint image from a plurality of captured images and displaying the free viewpoint image on a display projection surface, the method comprising:

• acquiring the plurality of captured images with a plurality of respective cameras;
• acquiring three-dimensional data of one or more three-dimensional objects imaged in the plurality of captured images;
• estimating projection surface residual data by machine learning using the plurality of captured images and viewpoint data as inputs, the projection surface residual data representing a difference between a bowl-shaped predefined projection surface and the display projection surface;
• acquiring the free viewpoint image by mapping the plurality of captured images onto the display projection surface using information about the predefined projection surface, the projection surface residual data, and the viewpoint data;
• transmitting the plurality of captured images and the three-dimensional data to a remote processing part;
• receiving, from the remote processing part, a three-dimensional image recreated at the remote processing part from the plurality of captured images based on the three-dimensional data;
• and
• displaying the free viewpoint image on a display part before receiving the three-dimensional image, and displaying the three-dimensional image on the display part after receiving the three-dimensional image.

An image processing system for synthesizing a free viewpoint image on a display projection surface from a plurality of captured images, the system comprising:

• a processor;
• and
• a memory coupled to the processor and storing instructions that, when executed, cause the processor to:
• acquire the plurality of captured images with a plurality of respective cameras;
• estimate projection surface residual data by machine learning using the plurality of captured images and viewpoint data as inputs, the projection surface residual data representing a difference between a bowl-shaped predefined projection surface and the display projection surface;
• and
• acquire the free viewpoint image by mapping the plurality of captured images onto the display projection surface using information about the predefined projection surface, the projection surface residual data, and the viewpoint data.

A system for training a neural network which infers residual data of a projection surface based on a plurality of captured images, the residual data representing a difference between a bowl-shaped predefined projection surface and a projection surface reflecting three-dimensional data of one or more three-dimensional objects imaged in the plurality of captured images, the system comprising:

• a processor;
• and
• a memory coupled to the processor and storing instructions that, when executed, cause the processor to:
• prepare the plurality of captured images taken by a plurality of respective cameras;
• prepare the three-dimensional data;
• and
• recreate a three-dimensional image from the plurality of captured images based on the three-dimensional data to generate a training image that serves as a free viewpoint image for training based on viewpoint data given as an input;
• and
• input the plurality of captured images and the viewpoint data in a neural network to produce learning residual data, generate a learning image that serves as a free viewpoint image for learning by mapping the plurality of captured images onto a display projection surface using the learning residual data, information about the predefined projection surface, and the viewpoint data, and train the neural network such that a difference between the training image and the learning image becomes smaller.