Differentiable emulation of non-differentiable image processing for adjustable and explainable non-destructive image and video editing

One or more computer storage media having a plurality of executable instructions embodied thereon, which, when executed by one or more processors, cause the one or more processors to perform a method comprising:

• training an emulator to predict corresponding data structures from a plurality of non-differentiable image processing control parameters, the corresponding data structures being differentiable with respect to each control parameter of the plurality of non-differentiable image processing control parameters, training the emulator by:
• predicting initial data structures based on an initial set of control parameters;
• generating a set of predicted training images based on applying the initial data structures to a set of training images;
• generating a set of ground truth images by applying the initial set of control parameters to the set of training images using non-differentiable image processing;
• and
• backpropagating a corresponding loss function based on the set of predicted training images and the set of ground truth images into a neural network of the emulator;
• training a translator to predict a predicted set of control parameters for a task from the plurality of non-differentiable image processing control parameters by:
• receiving, by the translator, a set of unadjusted training images for the task;
• predicting, by the translator, an initial predicted set of control parameters based on the set of unadjusted training images for the task;
• generating, by the emulator, an initial set of adjusted training images for the task by applying predicted corresponding data structures based on the initial predicted set of control parameters to the set of unadjusted training images;
• and
• backpropagating a loss function into the translator to determine the predicted set of control parameters, the loss function determined based on the initial set of adjusted training images for the task and a set of training ground truth images for the task;
• generating an updated image by applying the predicted set of control parameters to an input image;
• and
• causing display of the updated image and the predicted set of control parameters.

A computing system comprising:

• a processor;
• and
• a non-transitory computer-readable medium having stored thereon instructions that when executed by the processor, cause the processor to perform operations including:
• predicting, by a translator, a predicted set of control parameters from a plurality of non-differentiable image processing control parameters for a style of a reference image by:
• receiving, by the translator, the reference image;
• predicting, by the translator, an initial predicted set of control parameters based on at least one of an input image or the reference image;
• generating, by an emulator, an initial updated image by applying predicted corresponding data structures based on the initial predicted set of control parameters to the input image, the emulator trained to predict corresponding data structures from the plurality of non-differentiable image processing control parameters, the corresponding data structures being differentiable with respect to each control parameter of the plurality of non-differentiable image processing control parameters, training the emulator by:
• predicting initial data structures based on an initial set of control parameters;
• generating a set of predicted training images based on applying the initial data structures to a set of training images;
• generating a set of ground truth images by applying the initial set of control parameters to the set of training images using non-differentiable image processing;
• and
• backpropagating a corresponding loss function based on the set of predicted training images and the set of ground truth images into a neural network of the emulator;
• and
• backpropagating a loss function into the translator to determine the predicted set of control parameters, the loss function determined based on the initial updated image and the reference image;
• generating an updated image by applying the predicted set of control parameters to the input image;
• and
• causing display of the updated image and the predicted set of control parameters.