# US 12,602,750 B2 · Differentiable emulation of non-differentiable image processing for adjustable and explainable non-destructive image and video editing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,750 B2 - Document ID: US12602750B2 - Status: Granted - Kind: B2 - Application number: 18/528,488 - Filed: 2023-12-04 - Published: 2025-06-05 - Granted: 2026-04-14 - Earliest claimed: 2023-12-04 - Assignees: Adobe Inc. - Applicants: Adobe Inc. - Inventors: Yifei Fan; Peter Merrill - Web page: https://patentgrep.com/patent/US12602750B2 ## CPC - G06T 5/60: using machine learning, e.g. neural networks - G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403) - G06T 3/4046: using neural networks - G06T 11/00: Two-dimensional [2D] image generation - G06T 2207/10016: Video; Image sequence - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Methods and systems are provided for differentiable emulation of non-differentiable image processing for adjustable and explainable non-destructive image and video editing. In embodiments described herein, an emulator is trained to predict corresponding data structures from non-differentiable image processing control parameters, the data structures being differentiable with respect to each control parameter of the non-differentiable image processing control parameters. A translator is trained to predict a predicted set of control parameters from the non-differentiable image processing control parameters. An updated image is generated by applying the predicted set of control parameters from the translator to an input image. The updated image is subsequently displayed with the predicted set of control parameters. ## Claims ### Claim 1 A computer-implemented method comprising: training an emulator to predict corresponding data structures from a plurality of non-differentiable image processing control parameters, the corresponding data structures being differentiable with respect to each control parameter of the plurality of non-differentiable image processing control parameters, training the emulator by: predicting initial data structures based on an initial set of control parameters; generating a set of predicted training images based on applying the initial data structures to a set of training images; generating a set of ground truth images by applying the initial set of control parameters to the set of training images using non-differentiable image processing; and backpropagating a loss function based on the set of predicted training images and the set of ground truth images into a neural network of the emulator; training a translator to predict a predicted set of control parameters from the plurality of non-differentiable image processing control parameters using the corresponding data structures predicted by the emulator; generating an updated image by applying the predicted set of control parameters to an input image; and causing display of the updated image and the predicted set of control parameters. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the input image and the updated image are frames of a video. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein generating the updated image by applying the predicted set of control parameters to the input image further comprises: applying, by the emulator, predicted corresponding data structures from the predicted set of control parameters to the input image; and causing display of the updated image and the predicted set of control parameters through the emulator. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein generating the updated image by applying the predicted set of control parameters to the input image further comprises: applying the predicted set of control parameters by a non-differentiable image processing application to the input image; and causing display of the updated image and the predicted set of control parameters through the non-differentiable image processing application. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein each of the corresponding data structures comprise at least one of a 3D look-up table (3D LUT) or a bilateral grid. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the emulator comprises a transformer model. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the initial set of control parameters are randomly selected, the initial data structures correspond to an initial 3D look-up table (3D LUT), the set of predicted training images are generated by applying tri-linear interpolation to the set of training images using the initial 3D LUT, and the set of training images are adjusted to be low resolution. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the initial set of control parameters are randomly selected, the initial data structures correspond to an initial bilateral grid, the set of predicted training images are generated by applying bilateral filtering to the set of training images using the initial bilateral grid, and the set of training images are adjusted to be low resolution. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the translator is trained to predict the predicted set of control parameters for a task and the task is at least one of auto-tone, auto-color-recommendation, harmonization, relighting, exemplar-based style transfer, language-driven editing, and or shadow removal. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the translator is trained to predict the predicted set of control parameters for a style of a reference image. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein the predicted set of control parameters comprises a first set of predicted control parameters for a corresponding predicted mask and a second set of predicted control parameters for an area outside of the corresponding predicted mask. ### Claim 12 One or more computer storage media having a plurality of executable instructions embodied thereon, which, when executed by one or more processors, cause the one or more processors to perform a method comprising: training an emulator to predict corresponding data structures from a plurality of non-differentiable image processing control parameters, the corresponding data structures being differentiable with respect to each control parameter of the plurality of non-differentiable image processing control parameters, training the emulator by: predicting initial data structures based on an initial set of control parameters; generating a set of predicted training images based on applying the initial data structures to a set of training images; generating a set of ground truth images by applying the initial set of control parameters to the set of training images using non-differentiable image processing; and backpropagating a corresponding loss function based on the set of predicted training images and the set of ground truth images into a neural network of the emulator; training a translator to predict a predicted set of control parameters for a task from the plurality of non-differentiable image processing control parameters by: receiving, by the translator, a set of unadjusted training images for the task; predicting, by the translator, an initial predicted set of control parameters based on the set of unadjusted training images for the task; generating, by the emulator, an initial set of adjusted training images for the task by applying predicted corresponding data structures based on the initial predicted set of control parameters to the set of unadjusted training images; and backpropagating a loss function into the translator to determine the predicted set of control parameters, the loss function determined based on the initial set of adjusted training images for the task and a set of training ground truth images for the task; generating an updated image by applying the predicted set of control parameters to an input image; and causing display of the updated image and the predicted set of control parameters. ### Claim 13 (depends on 12) The media of claim 12, wherein the set of unadjusted training images and the set of training ground truth images are adjusted to be low resolution. ### Claim 14 (depends on 12) The media of claim 12, wherein the task is at least one of auto-tone, auto-color-recommendation, harmonization, relighting, exemplar-based style transfer, language-driven editing, or shadow removal. ### Claim 15 (depends on 12) The media of claim 12, wherein the predicted set of control parameters comprises a first set of predicted control parameters for a corresponding predicted mask and a second set of predicted control parameters for an area outside of the corresponding predicted mask. ### Claim 16 (depends on 12) The media of claim 12, wherein the corresponding data structures comprise at least one of a 3D look-up table (3D LUT) or a bilateral grid. ### Claim 17 A computing system comprising: a processor; and a non-transitory computer-readable medium having stored thereon instructions that when executed by the processor, cause the processor to perform operations including: predicting, by a translator, a predicted set of control parameters from a plurality of non-differentiable image processing control parameters for a style of a reference image by: receiving, by the translator, the reference image; predicting, by the translator, an initial predicted set of control parameters based on at least one of an input image or the reference image; generating, by an emulator, an initial updated image by applying predicted corresponding data structures based on the initial predicted set of control parameters to the input image, the emulator trained to predict corresponding data structures from the plurality of non-differentiable image processing control parameters, the corresponding data structures being differentiable with respect to each control parameter of the plurality of non-differentiable image processing control parameters, training the emulator by: predicting initial data structures based on an initial set of control parameters; generating a set of predicted training images based on applying the initial data structures to a set of training images; generating a set of ground truth images by applying the initial set of control parameters to the set of training images using non-differentiable image processing; and backpropagating a corresponding loss function based on the set of predicted training images and the set of ground truth images into a neural network of the emulator; and backpropagating a loss function into the translator to determine the predicted set of control parameters, the loss function determined based on the initial updated image and the reference image; generating an updated image by applying the predicted set of control parameters to the input image; and causing display of the updated image and the predicted set of control parameters. ### Claim 18 (depends on 17) The system of claim 17, wherein the reference image and the input image are adjusted to be low resolution during training of the translator. ### Claim 19 (depends on 17) The system of claim 17, wherein the reference image is generated by a language model. ### Claim 20 (depends on 17) The system of claim 17, wherein the predicted set of control parameters comprises a first set of predicted control parameters for a corresponding predicted mask and a second set of predicted control parameters for an area outside of the corresponding predicted mask.