End-to-end relighting of a foreground object of an image

A non-transitory computer-readable medium with instructions stored thereon that, when executed by a processor, cause the processor to perform operations comprising:

• generating a segmentation mask of an image, the segmentation mask defining a boundary of a human object in the image;
• transforming a masked image into a feature vector, the masked image including the image and the segmentation mask;
• concatenating the feature vector with spherical harmonic coefficients of a target illumination for the image;
• and
• disentangling an albedo decoder and a light transport decoder separately by:
• generating an albedo map influenced by the target illumination from the feature vector concatenated with the spherical harmonic coefficients of the target illumination utilizing the albedo decoder;
• generating a light transport matrix for the target illumination from the feature vector concatenated with the spherical harmonic coefficients of the target illumination utilizing the light transport decoder;
• predicting illumination coefficients of the image by concatenating a residual output of the albedo decoder and a residual output of the light transport decoder with the feature vector;
• obtaining a shading map of the image by multiplying the light transport matrix and illumination coefficients;
• processing a shaded image with a refinement block to thereby produce a refined image with a sharpened detail, the shaded image including the image and the shading map;
• scaling the refined image with the albedo map influenced by the target illumination and the shading map to produce a relight image.

A computing system comprising:

• a processor;
• and
• memory containing instructions that, when executed by the processor, cause the computing system to:
• automatically detect a boundary of a foreground object in an image;
• convert the image with the boundary into a feature vector;
• predict an illumination coefficient of the image by concatenating the feature vector with spherical harmonic coefficients of a target illumination;
• and
• disentangle an albedo decoder and a light transport decoder separately by:
• generating an albedo map influenced by the target illumination from the feature vector concatenated with the spherical harmonic coefficients of the target illumination utilizing the albedo decoder;
• generating a light transport matrix for the target illumination from the feature vector concatenated with the spherical harmonic coefficients of the target illumination utilizing the light transport decoder;
• obtain a shading map based on the light transport matrix and the predicted illumination coefficient;
• generate a relight image of the image based on the albedo map influenced by the target illumination and the shading map.