US 2024/0177273 A1Published application

System and a method for obtaining a processed output image having quality index selectable by an user

AssigneeSamsung Electronics Co., Ltd.

InventorsAleksei Aleksandrovich IVAKHNENKO, Polina Vladimirovna KARPIKOVA, Anastasiia Sergeevna IASHCHENKO, Andrei Nikolaevich SPIRIDONOV, Ekaterina Yurievna RADIONOVA, Riccardo FABBRICATORE, Leonid Igorevich KOSTIUSHKO

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

A system for obtaining output images having a quality index selectable by a user includes an electronic device including at least one processor and a memory storing input images and a set of generative artificial neural networks (GANs), the at least one processor being configured to perform artificial neural network operations. Each GAN is selectable by the user from the set of GANs stored in the memory, for obtaining an image with predefined quality index, each GAN is pre-trained and includes a plurality of Earlier Exit branches each of which is connected after each calculating module, each Earlier Exit branch contains as many calculating modules as remain in the backbone from a connection point of that Earlier Exit branch to a backbone exit, and each calculating module of each Earlier Exit branch performs a same function as a corresponding remaining calculating module in the backbone.

Claims20

Dependent claims

A system for obtaining output images having a quality index selectable by a user, the system comprising:

an electronic device comprising at least one processor and a memory operably connected to the processor and storing input images, a plurality of generative artificial neural networks (GANs) and a plurality of predictors, the at least one processor being configured to implement the GANs and the predictors to perform artificial neural network operations;
wherein each GAN being selectable by the user from the plurality of GANs stored in the memory, for obtaining an image with predefined quality index, each GAN is pre-trained, and each GAN comprises:
a plurality of calculating modules forming a backbone, and
a plurality of Earlier Exit branches each of which is connected after each calculating module, except for a last calculating module of the backbone, each Earlier Exit branch containing as many calculating modules as remain in the backbone from a connection point of that Earlier Exit branch to a backbone exit, each calculating module of each Earlier Exit branch performing a same function as a corresponding remaining calculating module in the backbone, and a computational budget of each Earlier Exit branch being less than a computational budget of corresponding remaining calculating modules of the backbone to the backbone exit, and
wherein each of the plurality of predictors is an artificial neural network, each predictor is generated and pre-trained for one of the plurality of GANs stored in the memory, and each predictor is configured to predict a quality index of a processed image for output of each Earlier Exit branch of the particular GAN based on an input image, which the user intends to apply to an input of the particular GAN,
wherein the closer a particular Earlier Exit branch is located to the backbone exit, the higher a computational budget for obtaining a particular output image generated by the particular Earlier Exit branch,
wherein quality indexes of output images generated by different Earlier Exit branches are different from each other, and
wherein each GAN is configured to output the output images from one of the plurality of Earlier Exit branches, which generates the output image having the quality index most matching the quality index selected by the user.

5 more elements

A method for obtaining an output image with a quality index selected by a user, the method comprising:

selecting, from a memory by the user, an input image, a pre-trained generative artificial neural network (GAN), and pre-trained predictor corresponding to the pre-trained GAN, the pre-trained GAN comprising a plurality of calculating modules forming a backbone, a plurality of Earlier Exit branches each of which is connected after each calculating module, except for a last calculating module of the backbone, each Earlier Exit branch containing as many calculating modules as remain in the backbone from a connection point of that Earlier Exit branch to a backbone exit, each calculating module of each Earlier Exit branch performing a same function as a corresponding remaining calculating module in the backbone, and a computational budget of each Earlier Exit branch being less than a computational budget of corresponding remaining calculating modules of the backbone to the backbone exit;
selecting, by the user, the quality index for the output image;
feeding the input image converted to predictor input data to be processed by the pre-trained predictor, to the pre-trained predictor;
predicting based on the input data, by the pre-trained predictor, predicted quality indexes for potential output images, which would be generated by each of the plurality of Earlier Exit branches of the pre-trained GAN;
selecting one Earlier Exit branch whose output image has a potential quality index most matching the quality index selected by the user;
converting the input image into GAN input data to be processed by the pre-trained GAN;
feeding the GAN input data to the pre-trained GAN with the one Earlier Exit branch for processing;
processing the GAN input data by the pre-trained GAN with the one Earlier Exit branch;
and
obtaining, on exit of the one Earlier Exit branch, the output image.

7 more elements

A method for obtaining an output image with a quality index selected by a user, the method comprising:

selecting, from a memory by the user, an input image, a pre-trained generative artificial neural network (GAN), and pre-trained predictor corresponding to the pre-trained GAN, the pre-trained GAN comprising a plurality of calculating modules forming a backbone, a plurality of Earlier Exit branches each of which is connected after each calculating module, except for a last calculating module of the backbone, each Earlier Exit branch containing as many calculating modules as remain in the backbone from a connection point of that Earlier Exit branch to a backbone exit, each calculating module of each Earlier Exit branch performing a same function as a corresponding remaining calculating module in the backbone, and a computational budget of each Earlier Exit branch being less than a computational budget of corresponding remaining calculating modules of the backbone to the backbone exit;
selecting, by the user, the quality index for the output image;
feeding the input image converted to predictor input data to be processed by the predictor, to the pre-trained predictor;
predicting based on the input image, by the pre-trained predictor, predicted quality indexes for potential output images, which would be generated by each of the plurality of Earlier Exit branches of the pre-trained GAN;
selecting one Earlier Exit branch whose output image has a potential quality index most matching the quality index selected by the user;
converting the input image into GAN input data to be processed by the pre-trained GAN;
feeding the GAN input data to the pre-trained GAN with the one Earlier Exit branch for processing;
processing the GAN input data by the pre-trained GAN with the one Earlier Exit branch, and during the processing:
fetching, from a database storing guide data, fetched guide data corresponding to the input image,
concatenating the fetched guide data with data output from one of the calculating modules preceding the one Earlier Exit branch to generate concatenated data, and
feeding the concatenated data into the one Earlier Exit branch or into the backbone for further processing;
and
obtaining, on exit of the one Earlier Exit branch, the output image.

10 more elements

Dependencies are inferred from the claim text.