# US 10,872,399 B2 · Photorealistic image stylization using a neural network model > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,872,399 B2 - Document ID: US10872399B2 - Status: Granted - Kind: B2 - Application number: 16/246,375 - Filed: 2019-01-11 - Published: 2019-08-08 - Granted: 2020-12-22 - Earliest claimed: 2018-02-02 - Assignees: NVIDIA Corporation - Applicants: NVIDIA Corporation - Inventors: Yijun Li; Ming-Yu Liu; Ming-Hsuan Yang; Jan Kautz - Web page: https://patentgrep.com/patent/US10872399B2 ## CPC - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06T 5/60: using machine learning, e.g. neural networks - G06T 5/70: Denoising; Smoothing - G06T 7/11: Region-based segmentation - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Photorealistic image stylization concerns transferring style of a reference photo to a content photo with the constraint that the stylized photo should remain photorealistic. Examples of styles include seasons (summer, winter, etc.), weather (sunny, rainy, foggy, etc.), lighting (daytime, nighttime, etc.). A photorealistic image stylization process includes a stylization step and a smoothing step. The stylization step transfers the style of the reference photo to the content photo. A photo style transfer neural network model receives a photorealistic content image and a photorealistic style image and generates an intermediate stylized photorealistic image that includes the content of the content image modified according to the style image. A smoothing function receives the intermediate stylized photorealistic image and pixel similarity data and generates the stylized photorealistic image, ensuring spatially consistent stylizations. ## Claims ### Claim 1 A computer-implemented method, comprising: processing a photorealistic content image by a sequence of processing layers within an encoder portion of a photo style transfer neural network model to extract content features from the photorealistic content image, wherein each processing layer includes a max pooling layer that generates location data identifying a location of a maximum value of the content features output by the processing layer; processing an output of the encoder portion based on style features extracted from a photorealistic style image to produce a transformed feature map; and processing the transformed feature map by a sequence of decoder processing layers within a decoder portion of the photo style transfer neural network model to produce a stylized photorealistic image including content from the photorealistic content image that is modified according to the photorealistic style image, wherein an unpooling layer within each decoder processing layer uses the location data generated by a corresponding one of the max pooling layers to process the transformed feature map. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising processing the stylized photorealistic image according to pixel similarity data for the photorealistic content image to produce a smoothed stylized photorealistic image. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein the pixel similarity data identifies pixels in the stylized photorealistic image that are consistent in color with adjacent pixels. ### Claim 4 (depends on 2) The computer-implemented method of claim 2, wherein a quadratic function with a closed-form solution is solved to produce the smoothed stylized photorealistic image. ### Claim 5 (depends on 2) The computer-implemented method of claim 2, wherein the pixel similarity data comprises pixel affinity values. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, further comprising processing the photorealistic style image by a second photo style transfer neural network model to produce an improved photorealistic style image, wherein the second photo style transfer neural network model has lower feature representation compared with the photo style transfer neural network model. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the photorealistic style image is segmented into a first style region and a second style region and the photo style transfer neural network model processes the first style region and a corresponding first content region of the photorealistic content image to produce a first region of the photorealistic style image. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the photorealistic content image is segmented into a first content region and a second content region and the photo style transfer neural network model processes the first content region and a corresponding first style region of the photorealistic style image to produce a first region of the photorealistic style image. ### Claim 9 A system, comprising a processor configured to implement a photo style transfer neural network model comprising an encoder portion and a decoder portion configured to: process a photorealistic content image by a sequence of processing layers within the encoder portion to extract content features from the photorealistic content image, wherein each processing layer includes a max pooling layer that generates location data identifying a location of a maximum value of the content features output by the processing layer; process an output of the encoder portion based on style features extracted from a photorealistic style image to produce a transformed feature map; and processing the transformed feature map by a sequence of decoder processing layers within the decoder portion to produce a stylized photorealistic image including content from the photorealistic content image that is modified according to the photorealistic style image, wherein an unpooling layer within each decoder processing layer uses the location data generated by a corresponding one of the max pooling layers to process the transformed feature map. ### Claim 10 (depends on 9) The system of claim 9, further comprising a smoothing function module configured to process the stylized photorealistic image according to pixel similarity data for the photorealistic content image to produce a smoothed stylized photorealistic image. ### Claim 11 (depends on 10) The system of claim 10, wherein the pixel similarity data identifies pixels in the stylized photorealistic image that are consistent in color with adjacent pixels. ### Claim 12 (depends on 10) The system of claim 10, wherein a quadratic function with a closed-form solution is solved to produce the smoothed stylized photorealistic image. ### Claim 13 (depends on 10) The system of claim 10, wherein the pixel similarity data comprises pixel affinity values. ### Claim 14 (depends on 9) The system of claim 9, further comprising a second photo style transfer neural network model configured to process the photorealistic style image to produce an improved photorealistic style image, wherein the second photo style transfer neural network model has lower feature representation compared with the photo style transfer neural network model. ### Claim 15 (depends on 9) The system of claim 9, wherein the photorealistic style image is segmented into a first style region and a second style region and the photo style transfer neural network model processes the first style region and a corresponding first content region of the photorealistic content image to produce a first region of the photorealistic style image. ### Claim 16 (depends on 9) The system of claim 9, wherein the photorealistic content image is segmented into a first content region and a second content region and the photo style transfer neural network model processes the first content region and a corresponding first style region of the photorealistic style image to produce a first region of the photorealistic style image. ### Claim 17 A non-transitory, computer-readable storage medium storing instructions that, when executed by a processing unit, cause the processing unit to: process a photorealistic content image by a sequence of processing layers within an encoder portion of a photo style transfer neural network model to extract content features from the photorealistic content image, wherein each processing layer includes a max pooling layer that generates location data identifying a location of a maximum value of the content features output by the processing layer; process an output of the encoder portion based on style features extracted from a photorealistic style image to produce a transformed feature map; and processing the transformed feature map by a sequence of decoder processing layers within the decoder portion of the photo style transfer neural network model to produce a stylized photorealistic image including content from the photorealistic content image that is modified according to the photorealistic style image, wherein an unpooling layer within each decoder processing layer uses the location data generated by a corresponding one of the max pooling layers to process the transformed feature map.