Image processing apparatus, image processing method, and storage medium

An image processing apparatus comprising:

• one or more hardware processors;
• and
• one or more memories storing one or more programs configured to be executed by the one or more hardware processors, the one or more programs including instructions for:
• obtaining image capturing parameters of each of a plurality of imaging apparatuses arranged at positions different from one another;
• obtaining data of a captured image obtained by image capturing by each of the plurality of imaging apparatuses;
• obtaining virtual viewpoint information including at least one of information indicating a position of a virtual viewpoint and information indicating a viewing direction from the virtual viewpoint;
• determining a learning condition of a learning model estimating radiance fields corresponding to an object existing in an image capturing area of the plurality of imaging apparatuses based on the virtual viewpoint information;
• performing learning of the learning model based on the learning condition, the image capturing parameters, and data of the captured image;
• estimating a three-dimensional shape of the object by using at least part of obtained data of a plurality of the captured images;
• and
• determining a degree of importance of the object by using at least one of information indicating a volume of the object obtained based on an estimated three-dimensional shape of the object and information indicating a distance from a position of the virtual viewpoint obtained based on an estimated three-dimensional shape of the object to the object,
• wherein the determining of the learning condition is performed based on the determined degree of importance of the object.

An image processing apparatus comprising:

• one or more hardware processors;
• and
• one or more memories storing one or more programs configured to be executed by the one or more hardware processors, the one or more programs including instructions for:
• obtaining image capturing parameters of each of a plurality of imaging apparatuses arranged at positions different from one another;
• obtaining data of a captured image obtained by image capturing by each of the plurality of imaging apparatuses;
• obtaining virtual viewpoint information including at least one of information indicating a position of a virtual viewpoint and information indicating a viewing direction from the virtual viewpoint;
• determining a learning condition of a learning model estimating radiance fields corresponding to an object existing in an image capturing area of the plurality of imaging apparatuses based on the virtual viewpoint information;
• performing learning of the learning model based on the learning condition, the image capturing parameters, and data of the captured image;
• and
• determining a degree of importance of the object in a case where the learning of the learning model is performed based on the virtual viewpoint information and information indicating a type of the object,
• wherein the determining of the learning condition is performed based on the determined degree of importance of the object.

An image processing method comprising the steps of:

• obtaining image capturing parameters of each of a plurality of imaging apparatuses arranged at positions different from one another;
• obtaining data of a captured image obtained by image capturing by each of the plurality of imaging apparatuses;
• obtaining virtual viewpoint information including at least one of information indicating a position of a virtual viewpoint and information indicating a viewing direction from the virtual viewpoint;
• determining a learning condition of a learning model estimating radiance fields corresponding to an object existing in an image capturing area of the plurality of imaging apparatuses based on the virtual viewpoint information;
• performing learning of the learning model based on the learning condition, the image capturing parameters, and data of the captured image;
• obtaining a virtual viewpoint image corresponding to an appearance from the virtual viewpoint based on the virtual viewpoint information;
• and
• determining a degree of importance of the object in a case where the learning of the learning model is performed based on an object area, which is an image area corresponding to the object in the obtained virtual viewpoint image,
• wherein the determining of the learning condition is performed based on the determined degree of importance of the object.

A non-transitory computer readable storage medium storing a program for causing a computer to perform a control method of an image processing apparatus, the control method comprising the steps of:

• obtaining image capturing parameters of each of a plurality of imaging apparatuses arranged at positions different from one another;
• obtaining data of a captured image obtained by image capturing by each of the plurality of imaging apparatuses;
• obtaining virtual viewpoint information including at least one of information indicating a position of a virtual viewpoint and information indicating a viewing direction from the virtual viewpoint;
• determining a learning condition of a learning model estimating radiance fields corresponding to an object existing in an image capturing area of the plurality of imaging apparatuses based on the virtual viewpoint information;
• performing learning of the learning model based on the learning condition, the image capturing parameters, and data of the captured image;
• obtaining a virtual viewpoint image corresponding to an appearance from the virtual viewpoint based on the virtual viewpoint information;
• and
• determining a degree of importance of the object in a case where the learning of the learning model is performed based on an object area, which is an image area corresponding to the object in the obtained virtual viewpoint image,
• wherein the determining of the learning condition is performed based on the determined degree of importance of the object.