Image processing apparatus and method, and storage medium

A method for achieving high definition of an image of a second image group by using a first image group, the second image group having less high-frequency components in a corresponding frame than the first image group, the method comprising:

• selecting, based on a high-definition target image selected from the second image group, a pair of supervisory data to be used for learning from among candidate pairs registered in a candidate database in which frame pairs consisting of a frame of the first image group and a frame of the second image group having an image-capturing time difference equal to or less than a threshold are registered, the threshold being set to a display period of one frame of the first image group;
• generating a learning model;
• inferring high-frequency components;
• and
• generating a high-definition image based on the high-definition target image and the high-frequency components inferred in the inferring;
• cutting out one or more local regions to be inferred from the high-definition target image, the one or more local regions being non-overlapping square blocks of N×N pixels obtained by dividing the high-definition target image into equal-sized blocks, and
• selecting a pair of local regions to be used for the learning from the pair of supervisory data obtained in the selecting, based on one or more local images cut out in the cutting out,
• wherein in the learning model, for each said block, generating, a local region learning model is generated immediately after the pair of local regions for the block is selected and before inference of the block, and to discard the generated local region learning model after the inference of the block,
• wherein in the inferring, local region high-frequency components of the one or more local images cut out from the high-definition target image are inferred by using the local region learning model, and
• wherein in the generating, a high-definition image of the one or more local regions is generated and combined by using the local region high-frequency components and the one or more local images of the high-definition target image sequentially for each of the blocks.

A non-transitory computer-readable storage medium storing a program for causing a computer to function as each unit of an image processing apparatus configured to achieve high definition of an image of a second image group by using a first image group, the second image group having less high-frequency components in a corresponding frame than the first image group, the apparatus comprising:

• a selection unit configured to select, based on a high-definition target image selected from the second image group, a pair of supervisory data to be used for learning from among candidate pairs registered in a candidate database in which frame pairs consisting of a frame of the first image group and a frame of the second image group having an image-capturing time difference equal to or less than a threshold are registered, the threshold being set to a display period of one frame of the first image group;
• a learning model generation unit configured to generate a learning model;
• an inference unit configured to infer high-frequency components;
• and
• an image generation unit configured to generate a high-definition image based on the high-definition target image and the high-frequency components inferred by the inference unit;
• cutting out one or more local regions to be inferred from the high-definition target image, the one or more local regions being non-overlapping square blocks of N×N pixels obtained by dividing the high-definition target image into equal-sized blocks, and
• selecting a pair of local regions to be used for the learning from the pair of supervisory data obtained in the selecting, based on one or more local images cut out in the cutting out,
• wherein in the learning model generating, for each said block, a local region learning model is generated immediately after the pair of local regions for the block is selected and before inference of the block, and to discard the generated local region learning model after the inference of the block,
• wherein in the inferring, local region high-frequency components of the one or more local images cut out from the high-definition target image are inferred by using the local region learning model, and
• wherein in the generating, a high-definition image of the one or more local regions is generated and combined by using the local region high-frequency components and the one or more local images of the high-definition target image sequentially for each of the blocks.