# US 2019/0244060 A1 · Domain Stylization Using a Neural Network Model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0244060 A1
- Document ID: US20190244060A1
- Status: Published application
- Kind: A1
- Application number: 16/265,725
- Filed: 2019-02-01
- Published: 2019-08-08
- Earliest claimed: 2018-02-02
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Aysegul Dundar; Ming-Yu Liu; Ting-Chun Wang; John Zedlewski; Jan Kautz
- Web page: https://patentgrep.com/patent/US20190244060A1

## CPC

- G06V 10/82: using neural networks
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/24: Classification techniques
- G06F 18/24133: {Distances to prototypes}
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 3/10: Selection of transformation methods according to the characteristics of the input images
- G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215)
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 15/00: Three-dimensional [3D] image rendering
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/955: {using specific electronic processors}
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A style transfer neural network may be used to generate stylized synthetic images, where real images provide the style (e.g., seasons, weather, lighting) for transfer to synthetic images. The stylized synthetic images may then be used to train a recognition neural network. In turn, the trained neural network may be used to predict semantic labels for the real images, providing recognition data for the real images. Finally, the real training dataset (real images and predicted recognition data) and the synthetic training dataset are used by the style transfer neural network to generate stylized synthetic images. The training of the neural network, prediction of recognition data for the real images, and stylizing of the synthetic images may be repeated for a number of iterations. The stylization operation more closely aligns a covariate of the synthetic images to the covariate of the real images, improving accuracy of the recognition neural network.

## Claims

### Claim 1

A computer-implemented method, comprising:
processing a photorealistic image by a recognition neural network model to produce predicted recognition data for the photorealistic image; and
processing the photorealistic image, the predicted recognition data for the photorealistic image, a synthetic image, and ground truth recognition data for the synthetic image by a style transfer neural network model to produce a stylized synthetic image including content from the synthetic image that is modified according to the photorealistic image.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising training the recognition neural network model using the stylized synthetic image and the ground truth recognition data for the synthetic image.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, further comprising, after the training, processing the photorealistic image by the recognition neural network model to produce second predicted recognition data for the photorealistic image.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, further comprising processing the photorealistic image, the second predicted recognition data, the synthetic image, and the ground truth recognition data by the style transfer neural network model to produce a second stylized synthetic image including content from the synthetic image that is modified according to the photorealistic image.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, further comprising processing additional photorealistic images, the predicted recognition data for the additional photorealistic images, the synthetic image, and the ground truth recognition data for the synthetic image by the style transfer neural network model to produce additional stylized synthetic image, each including content from the synthetic image that is modified according a different one of the additional photorealistic images.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, prior to processing the photorealistic image by the recognition neural network model, processing a set of photorealistic images without recognition data for the set of photorealistic images, a set of synthetic images, and ground truth recognition data for the set of synthetic images by the style transfer neural network model to produce a set of stylized synthetic images.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the ground truth recognition data for the synthetic image identifies a first region associated with a first label and the first region of the synthetic image is stylized based on a first region of the photorealistic image that, according to the predicted recognition data for the photorealistic image, is associated with the first label.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein the ground truth recognition data for the synthetic image identifies a second region associated with a second label and the second region of the synthetic image is unchanged in the stylized synthetic image when the predicted recognition data for the photorealistic image does not identify a region of the photorealistic image that is associated with the second label.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the recognition neural network model is a classification neural network model.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the recognition neural network model is a segmentation neural network model.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, wherein the recognition neural network model is an object detection neural network model.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, wherein the recognition neural network model is an instance segmentation neural network model.

### Claim 13 (depends on 1)

The computer-implemented method of claim 1, wherein the recognition neural network model is a panoptic segmentation neural network model.

### Claim 14

A computer-implemented method, comprising:
training a recognition neural network model using stylized synthetic images and ground truth recognition data for the synthetic images, wherein the stylized synthetic images include content from the synthetic images that is modified according to photorealistic images; and
processing a photorealistic image by the recognition neural network model to produce predicted recognition data for the photorealistic image.

### Claim 15 (depends on 14)

The computer-implemented method of claim 14, wherein the photorealistic image is included as one of the photorealistic images.

### Claim 16

A system, comprising:
a recognition neural network model configured to process a photorealistic image to produce predicted recognition data for the photorealistic image; and
a style transfer neural network model configured to process the photorealistic image, the predicted recognition data for the photorealistic image, a synthetic image, and ground truth recognition data for the synthetic image to produce a stylized synthetic image including content from the synthetic image that is modified according to the photorealistic image.

### Claim 17 (depends on 16)

The system of claim 16, further comprising training the recognition neural network model using the stylized synthetic image and the ground truth recognition data for the synthetic image.

### Claim 18 (depends on 17)

The system of claim 17, wherein, after the training, the recognition neural network model processes the photorealistic image by to produce second predicted recognition data for the photorealistic image.

### Claim 19 (depends on 18)

The system of claim 18, wherein the style transfer neural network model processes the photorealistic image, the second predicted recognition data, the synthetic image, and the ground truth recognition data to produce a second stylized synthetic image including content from the synthetic image that is modified according to the photorealistic image.

### Claim 20

A non-transitory, computer-readable storage medium storing instructions that, when executed by a processing unit, cause the processing unit to:
process a photorealistic image by a recognition neural network model to produce predicted recognition data for the photorealistic image; and
process the photorealistic image, the predicted recognition data for the photorealistic image, a synthetic image, and ground truth recognition data for the synthetic image by a style transfer neural network model to produce a stylized synthetic image including content from the synthetic image that is modified according to the photorealistic image.
