# US 10,482,639 B2 · Deep high-resolution style synthesis > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,639 B2 - Document ID: US10482639B2 - Status: Granted - Kind: B2 - Application number: 15/438,147 - Filed: 2017-02-21 - Published: 2018-08-23 - Granted: 2019-11-19 - Earliest claimed: 2017-02-21 - Assignees: Adobe Inc. - Applicants: Adobe Inc. - Inventors: Yijun Li; Chen Fang; Jimei Yang; Zhaowen Wang; Xin Lu - Web page: https://patentgrep.com/patent/US10482639B2 ## CPC - G06T 11/60: Creating or editing images; Combining images with text - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) - G06V 10/40: Extraction of image or video features - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/82: using neural networks - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract In some embodiments, techniques for synthesizing an image style based on a plurality of neural networks are described. A computer system selects a style image based on user input that identifies the style image. The computer system generates an image based on a generator neural network and a loss neural network. The generator neural network outputs the synthesized image based on a noise vector and the style image and is trained based on style features generated from the loss neural network. The loss neural network outputs the style features based on a training image. The training image and the style image have a same resolution. The style features are generated at different resolutions of the training image. The computer system provides the synthesized image to a user device in response to the user input. ## Claims ### Claim 1 A computer-implemented method for synthesizing an image style based on a plurality of neural networks, the computer-implemented method comprising: selecting, by a computer system, a style image based on user input that identifies the style image; generating, by a generator neural network, a synthesized image based on the style image, wherein the generator neural network is a feed-forward generator neural network and is trained using a loss neural network by: generating, by different layers of the generator neural network and based on a training image, a first output image at a first resolution and a second output image at a second resolution, wherein the first resolution is smaller than the second resolution, the training image and the style image having a same resolution; applying the loss neural network to the first output image to generate a first set of output style features; applying the loss neural network to the second output image to generate a second set of output style features; generating a first set of training image style features and a second set of training image style features based on the training image and the loss neural network; determining, by the computer system, a first loss based on the first set of output style features and the first set of training image style features, and a second loss based on the second set of output style features and the second set of training image style features; and adjusting, by the computer system, parameters of different layers of the generator neural network based, at least in part, upon the first and the second losses; and providing, by the computer system, the synthesized image to a user device in response to the user input. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the synthesized image and the style image have the same resolution, and wherein the synthesized image comprises global style properties and local style properties of the style image. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the second output image at the same resolution of the training image. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the generator neural network comprises a first set of layers that output the first output image and a second set of layers that output the second output image, the second set of layers comprising the first set of layers, and the parameters of the first set of layers of the generator neural network are updated based on the first loss and the second loss and the parameters of the second set of layers of the generator neural network are updated based on the second loss. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the generator neural network transfers style properties of the style image to the synthesized image. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein the style image comprises an image of an artwork at a resolution greater than or equal to a 1024 pixel resolution, and wherein the style properties define visual aesthetics of the artwork. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the generator neural network further generates the synthesized image based on additional user input that identifies a content image, wherein the synthesized image is generated by at least transferring style properties of the style image to the synthesized image and transferring content from the content image to the synthesized image. ### Claim 8 (depends on 7) The computer-implemented method of claim 7, wherein the generator neural network is trained by at least computing style losses and content losses at a lower resolution and a same resolution of the training image, and wherein the style losses and content losses are computed based on the sets of output style features generated by the loss neural network for the output images at different resolutions. ### Claim 9 (depends on 8) The computer-implemented method of claim 8, wherein the style losses and the content losses are weighted to generate a first total loss at the lower resolution and a second total loss at the resolution of the training image, and wherein the generator neural network is trained based on a backpropagation according to the first total loss and a backpropagation according to the second total loss. ### Claim 10 A computer system for synthesizing an image style based on a plurality of neural networks, the computer system comprising: means for selecting a style image based on user input that identifies the style image; means for generating a synthesized image based on a generator neural network and a noise vector of the style image, wherein the generator neural network is a feed-forward generator neural network and is trained using a loss neural network by: generating, by different layers of the generator neural network and based on a training image, a first output image at a first resolution and a second output image at a second resolution, wherein the first resolution is smaller than the second resolution, applying the loss neural network to the first output image to generate a first set of output style features; applying the loss neural network to the second output image to generate a second set of output style features; generating a first set of training image style features and a second set of training image style features based on the training image and the loss neural network; determining, by the computer system, a first loss based on the first set of output style features and the first set of training image style features, and a second loss based on the second set of output style features and the second set of training image style features; and adjusting parameters of different layers of the generator neural network based, at least in part, upon the first and second losses; and means for providing the synthesized image to a user device in response to the user input. ### Claim 11 (depends on 10) The computer system of claim 10, wherein the generator neural network comprises a multi-layer neural network trained to generate output images at different resolutions, wherein the multi-layer neural network comprises sets of up-sampling, convolutional, and data transformation layers. ### Claim 12 (depends on 11) The computer system of claim 11, wherein the loss neural network comprises a multi-layer neural network trained to generate style features based on training images. ### Claim 13 (depends on 10) The computer system of claim 10, wherein the style image and the training image have a 1024 pixel resolution, wherein the first set of output style features is generated at a resolution greater than or equal to a 256 pixel resolution, wherein the second set of output style features is generated at a resolution greater than or equal to a 512 pixel resolution, and wherein a third set of output style features is generated at a resolution greater than or equal to the 1024 pixel resolution. ### Claim 14 (depends on 10) The computer system of claim 10, wherein the noise vector comprises a random vector that includes a random subset of features of the style image. ### Claim 15 A non-transitory computer-readable storage medium storing instructions that, upon execution by a processor of a computer system, configure the computer system to perform operations comprising: selecting a style image based on user input that identifies the style image; generating a synthesized image based on a generator neural network and the style image, wherein the generator neural network is a feed-forward generator neural network and is trained using a loss neural network by: generating, by different layers of the generator neural network and based on a training image, a first output image at a first resolution and a second output image at a second resolution, wherein the first resolution is smaller than the second resolution, the training image and the style image having a same resolution; applying the loss neural network to the first output image to generate a first set of output style features; applying the loss neural network to the second output image to generate a second set of output style features; generating a first set of training image style features and a second set of training image style features based on the training image and the loss neural network; determining, by the computer system, a first loss based on the first set of output style features and the first set of training image style features, and a second loss based on the second set of output style features and the second set of training image style features; and adjusting parameters of different layers of the generator neural network based, at least in part, upon the first and the second losses; and providing the synthesized image to a user device in response to the user input. ### Claim 16 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein the style image is selected from a data store of style images, and wherein the style images are available to the user device based on a user account associated with the user device. ### Claim 17 (depends on 1) The computer-implemented method of claim 1, wherein generating a first set of training image style features and a second set of training image style features comprises: generating, based on the training image, a first image having a same resolution as the first output image; generating, based on the training image, a second image having a same resolution as the second output image; applying the loss neural network to the first image to generate the first set of training image style features; and applying the loss neural network to the second image to generate the second set of training image style features. ### Claim 18 (depends on 17) The computer-implemented method of claim 17, wherein the first image and the second image is generated by down-sampling the training image to respective resolutions of the first and second images. ### Claim 19 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein the generator neural network comprises a first set of layers that output the first output image and a second set of layers that output the second output image, the second set of layers comprising the first set of layers. ### Claim 20 (depends on 19) The non-transitory computer-readable storage medium of claim 19, wherein the parameters of the first set of layers of the generator neural network are updated based on the first loss and the second loss and the parameters of the second set of layers of the generator neural network are updated based on the second loss.