# US 2024/0202989 A1 · Neural photofinisher digital content stylization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202989 A1
- Document ID: US20240202989A1
- Status: Published application
- Kind: A1
- Application number: 18/067,878
- Filed: 2022-12-19
- Published: 2024-06-20
- Earliest claimed: 2022-12-19
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Ethan Tseng; Zhihao Xia; Yifei Fan; Xuaner Zhang; Peter Merrill; Lars Jebe; Jiawen Chen
- Web page: https://patentgrep.com/patent/US20240202989A1

## CPC

- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77)
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/82: using neural networks
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 2201/10: Recognition assisted with metadata
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Digital content stylization techniques are described that leverage a neural photofinisher to generate stylized digital images. In one example, the neural photofinisher is implemented as part of a stylization system to train a neural network to perform digital image style transfer operations using reference digital content as training data. The training includes calculating a style loss term that identifies a particular visual style of the reference digital content. Once trained, the stylization system receives a digital image and generates a feature map of a scene depicted by the digital image. Based on the feature map as well as the style loss, the stylization system determines visual parameter values to apply to the digital image to incorporate a visual appearance of the particular visual style. The stylization system generates the stylized digital image by applying the visual parameter values to the digital image automatically and without user intervention.

## Claims

### Claim 1

A system comprising:
an automatic style transfer system implemented by one or more processing devices to perform operations including:
training a neural network to perform digital image style transfer operations using reference digital content with a particular visual style as training data, the training including calculating a style loss for the particular visual style;
receiving input data that includes a digital image depicting a scene;
determining, by an encoder of the neural network and based on the input data, a feature map of the scene depicted by the digital image;
determining, by the neural network, visual parameter values to apply to the digital image to incorporate a visual appearance of the particular visual style based on the feature map and the style loss; and
generating, by a neural photofinisher of the automatic style transfer system, a stylized digital image by applying the determined visual parameter values to the digital image automatically and without user intervention.

### Claim 2 (depends on 1)

The system as described in claim 1, wherein the reference digital content includes a plurality of frames from a digital video that represents the particular visual style, the input data includes an unprocessed raw image file obtained from one or more sensors of an image capture device, and the stylized digital image is an sRGB image.

### Claim 3 (depends on 1)

The system as described in claim 1, wherein the neural photofinisher includes a plurality of neural proxies, each respective neural proxy being trained to perform an image processing transformation directly to the input data independent of other neural proxies of the plurality of neural proxies.

### Claim 4 (depends on 1)

The system as described in claim 1, wherein the style loss includes a luma loss term to identify a perceptual brightness of the particular visual style, a chroma loss term to determine a color palette of the particular visual style, and a gram loss term that identifies non-content features of the particular visual style.

### Claim 5 (depends on 1)

The system as described in claim 1, wherein the automatic style transfer system is operable to augment the feature map with image statistics including histograms based on the input data.

### Claim 6 (depends on 1)

The system as described in claim 1, wherein the visual parameter values define relative amounts of visual attributes including temperature, tint, exposure, contrast, and saturation.

### Claim 7 (depends on 1)

The system as described in claim 1, wherein the automatic style transfer system is trained to perform style transfer operations for a plurality of visual styles, and wherein the operations include receiving an input to select one of the plurality of visual styles and the stylized digital image incorporates a visual appearance of the selected visual style.

### Claim 8 (depends on 1)

The system as described in claim 1, wherein the input data includes a digital video including a sequence of frames and metadata associated with the digital video, and the automatic style transfer system is further configured to perform operations including:
identifying at least two keyframes of the digital video based on the metadata;
determining visual parameter values to apply to the at least two keyframes to incorporate the visual appearance of the particular visual style; and
generating a stylized digital video by applying the determined visual parameter values to the at least two keyframes and interpolating visual parameter values to be applied to frames between the at least two keyframes based on the determined visual parameter values.

### Claim 9

A method comprising:
training, by a processing device, a plurality of neural proxies of a differentiable neural photofinisher to perform photofinishing operations using tapout training data that includes intermediate images extracted from a reference photofinishing pipeline;
receiving, by the processing device, input data that includes a digital image and one or more parameter values that define relative amounts of respective visual attributes to apply to the digital image; and
generating, by the processing device a stylized digital image by applying order independent transformations to the digital image using the plurality of neural proxies based on the one or more parameter values.

### Claim 10 (depends on 9)

The method as described in claim 9, wherein each respective neural proxy models an individual image processing operation of the reference photofinishing pipeline.

### Claim 11 (depends on 9)

The method as described in claim 9, wherein the plurality of neural proxies includes a neural proxy to model a saturation transformation, a neural proxy to model a vibrance transformation, and a neural proxy to model RGB toning transformation, the neural proxies to model the saturation, vibrance, and RGB toning transformations including a multilayer perceptron that operates at a pixel level.

### Claim 12 (depends on 9)

The method as described in claim 9, wherein the plurality of neural proxies includes a neural proxy to model a tone mapping transformation and a neural proxy to model a texture transformation, the neural proxies to model the tone mapping and texture transformations having a network architecture that applies convolutional filters per-channel to preserve a color space of the stylized digital image.

### Claim 13 (depends on 9)

The method as described in claim 9, wherein the plurality of neural proxies includes a differentiable program to model a color conversion transformation and a differentiable program to model a gamma correction transformation, the differentiable program to model the color conversion and gamma correction transformations based on metadata associated with the digital image.

### Claim 14 (depends on 9)

The method as described in claim 9, wherein the order independent transformations applied by the plurality of neural proxies are based in part of image statistics associated with the input data.

### Claim 15 (depends on 9)

The method as described in claim 9, wherein the digital image is a short exposure Bayer image, and wherein the plurality of neural proxies includes a neural proxy to model a demosaicing operation, the neural proxy to model a demosaicing operation configured to apply a transformation to the digital image to generate the stylized digital image with a visual appearance of a long exposure image.

### Claim 16 (depends on 9)

The method as described in claim 9, wherein the input data includes a first set of parameter values to generate a first stylized digital image based on the digital image and a second set of parameter values to generate a second stylized digital image based on the digital image, and wherein the differentiable neural photofinisher is configured to generate a perturbation to apply to the digital image such that an image classifier incorrectly classifies the first stylized digital image and correctly classifies the second stylized digital image.

### Claim 17

A non-transitory computer-readable storage medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
receiving input data that includes a first digital image and a corresponding edited digital image with a particular visual style that has been processed by a reference photofinishing pipeline;
predicting, by a neural photofinisher, visual parameter values to generate the corresponding edited digital image with the particular visual style imparted by the reference photofinishing pipeline;
receiving a second digital image; and
generating a stylized digital image with the particular visual style by applying the predicted visual parameter values to the second digital image.

### Claim 18 (depends on 17)

The non-transitory computer-readable storage medium as described in claim 17, wherein the reference photofinishing pipeline is an image signal processing (ISP) pipeline of a camera of a mobile device.

### Claim 19 (depends on 18)

The non-transitory computer-readable storage medium as described in claim 18, wherein predicting the visual parameter values is based in part on camera metadata associated with the camera of the mobile device.

### Claim 20 (depends on 17)

The non-transitory computer-readable storage medium as described in claim 17, wherein the neural photofinisher implements a first-order gradient descent to predict the visual parameter values.
