US 10,504,267 B2 · Generating a stylized image or stylized animation by matching semantic features via an appearance guide, a segmentation guide, and/or a temporal guide

Abstract

Certain embodiments involve generating an appearance guide, a segmentation guide, and a positional guide and using one or more of the guides to synthesize a stylized image or animation. For example, a system obtains data indicating a target and a style exemplar image and generates a segmentation guide for segmenting the target image and the style exemplar image and identifying a feature of the target image and a corresponding feature of the style exemplar image. The system generates a positional guide for determining positions of the target feature and style feature relative to a common grid system. The system generates an appearance guide for modifying intensity levels and contrast values in the target image based on the style exemplar image. The system uses one or more of the guides to transfer a texture of the style feature to the corresponding target feature.

Claims20

Dependent claims

A method for generating a stylized image, the method comprising:

obtaining, by a processor, data indicating a target image and a style exemplar image, the target image including a first character and the style exemplar image including a second character and wherein the style exemplar image includes a style or texture;
generating, by the processor, a segmentation guide for the target image and the style exemplar image by:
creating a first head soft mask of the first character and a second head soft mask of the second character;
identifying a plurality of target features of the target image and a plurality of style features of the style exemplar image;
and
creating a soft mask for a target feature of the plurality of target features and a soft mask for a style feature of the plurality of style features;
creating, by the processor and using the segmentation guide, the stylized image by applying the texture or style from the style exemplar image to the target image;
and
outputting, by the processor, the stylized image for display via a user interface or for storage.

6 more elements

A method for generating a stylized image, the method comprising:

obtaining, by a processor, data indicating a target image and a style exemplar image, the target image including a first character and the style exemplar image including a second character and wherein the style exemplar image includes a style or texture;
generating, by the processor, an appearance guide for the style exemplar image and the target image by:
generating a converted target image and a converted style exemplar image by converting a copy of the style exemplar image and a copy of the target image into grayscale;
matching a global intensity level of the converted target image to a global intensity level of the converted style exemplar image;
matching a local and global contrast of the converted target image to a local and global contrast of the converted style exemplar image;
identifying a set of pixels in the converted target image having an intensity level that matches an intensity level of a set of pixels in the converted style exemplar image;
and
transferring the style or texture of the style exemplar image to the target image using the identified set of pixels in the converted style exemplar image;
creating, by the processor and using the appearance guide, the stylized image by applying the texture or style from the style exemplar image to the target image;
and
outputting, by the processor, the stylized image for display via a user interface or for storage.

8 more elements

A method comprising:

obtaining, by a processor, data indicating a target image and a style exemplar image, the target image including a first character and the style exemplar image including a second character and wherein the style exemplar image includes a style or texture;
generating, by the processor, a temporal guide for generating a stylized video, the stylized video comprising a plurality of stylized images and the temporal guide usable for determining an amount of temporal flickering in the stylized video, wherein generating the temporal guide comprises advecting, by the processor and using an underlying motion field, a stylized image of the plurality of stylized images;
controlling, by the processor, the amount of temporal flickering in the stylized video by varying an amount of blur of the advected stylized image and the style exemplar image;
and
outputting, by the processor, the stylized video for display via a user interface or for storage.

2 more elements

A system comprising:

one or more processing devices;
and
one or more non-transitory computer-readable media communicatively coupled to the one or more processing devices and storing program code,
wherein the one or more processing devices is configured for executing the program code and thereby performing operations comprising:
obtaining data indicating a target image and a style exemplar image;
and performing one or more of (1) a first set of stylization operations, (2) a second set of stylization operations, and (3) a third set of stylization operations,
wherein the first set of stylization operations comprises:
(a) generating a segmentation guide for the target image and the style exemplar image by at least:
(i) creating a first head soft mask of a first character depicted in the target image and a second head soft mask of a second character depicted in the style exemplar image,
(ii) identifying a target feature of the target image and a style feature of the style exemplar image, and
(iii) creating a soft mask for the target feature and a soft mask for the style feature, and
(b) creating a stylized image by using the segmentation guide to apply a texture or style from the style exemplar image to the target image;
wherein the second set of stylization operations comprises:
(a) generating an appearance guide for the style exemplar image and the target image by at least:
(i) generating a converted target image and a converted style exemplar image by converting a copy of the style exemplar image and a copy of the target image into grayscale,
(ii) matching a global intensity level of the converted target image to a global intensity level of the converted style exemplar image,
(iii) matching a local and global contrast of the converted target image to a local and global contrast of the converted style exemplar image,
(iv) identifying a set of pixels in the converted target image having an intensity level that matches an intensity level of a set of pixels in the converted style exemplar image, and
(v) transferring the style or texture of the style exemplar image to the target image using the identified set of pixels in the converted style exemplar image, and
(b) creating the stylized image or a different stylized image by using the appearance guide to apply the texture or style from the style exemplar image to the target image;
and
wherein the third set of stylization operations comprises:
(a) generating a temporal guide usable for determining an amount of temporal flickering in a stylized video, wherein generating the temporal guide comprises advecting a particular stylized image from a plurality of stylized images that are included in the stylized video,
(b) controlling the amount of temporal flickering in the stylized video by varying an amount of blur of the advected stylized image and the style exemplar image, and
(c) outputting the stylized video for display via a user interface or for storage.

22 more elements

Dependencies are inferred from the claim text.

Generating a stylized image or stylized animation by matching semantic features via an appearance guide, a segmentation guide, and/or a temporal guide

Life of this patent

Generating a stylized image or stylized animation by matching semantic features via an appearance guide, a segmentation guide, and/or a temporal guide

Life of this patent

Abstract

Claims20