Real-time intelligent image manipulation system

A method of image manipulation based on a user's desired content and style, comprising:

• obtaining, by a central processing unit (CPU), a content image representative of the user's desired content;
• obtaining, by the CPU, a style image representative of the user's desired style, wherein the user's desired style comprises desired colors, textures, and patterns of a stylized output image;
• obtaining, by the CPU, a choice of either a quick processing model or a professional processing model;
• generating, by the CPU, a stylized feature vector, wherein the stylized feature vector is generated using the whitening and coloring transformations;
• decoding, by the CPU, the stylized feature vector into an output image having the user's desired content and style;
• and
• displaying, by the CPU, the output image;
• wherein, upon choosing the quick processing model, generating, by the CPU, the stylized feature vector comprises:
• encoding, by the CPU, the content image into a content feature vector by applying a series of non-linear transformations, wherein the content feature vector is generated by extracting content features from the content image by generating an initial content image vector and performing one or more whitening transformations on the initial content image vector;
• encoding, by the CPU, the style image into a style feature vector by applying a series of non-linear transformations;
• and
• generating, by the CPU, the stylized feature vector based on the content feature vector and the style feature vector;
• and wherein, upon choosing the professional processing model, generating, by the CPU, the stylized feature vector comprises:
• determining, by the CPU, an optimization parameter;
• encoding, by the CPU, the content image into a tentative feature vector;
• encoding, by the CPU, the style image into a tentative style feature vector,
• generating, by the CPU, a tentative stylized feature vector based on the tentative content feature vector and tentative style feature vector,
• applying, by the CPU, the tentative stylized feature vector to generate a tentative output image;
• repeating, by the CPU, an optimization operation for a number of optimization iterations determined by the optimization parameter, the optimization operation comprising:
• encoding, by the CPU, content elements of the tentative output image into a refined content feature vector,
• encoding, by the CPU, style elements of the tentative output image into a refined style feature vector,
• determining, by the CPU, a content loss parameter by comparing the tentative content feature vector to the refined content feature vector,
• determining, by the CPU, a style loss parameter by comparing the tentative style feature vector to the refined style feature vector,
• optimizing, by the CPU, the refined content feature vector and refined style feature vector based on the content loss parameter and style loss parameter,
• generating, by the CPU, a refined stylized feature vector based on the refined content feature vector and refined style feature vector,
• wherein in case the refined stylized feature vector is not sufficiently optimized according to the optimization parameter, decoding, by the CPU, the refined stylized feature vector into a refined output image, saving the refined output image as the tentative output image, and repeating the optimization operation, and
• wherein in case the refined stylized feature vector is sufficiently optimized according to the optimization parameter, saving, by the CPU, the refined stylized feature vector as the stylized feature vector, wherein the professional processing model performs iterative optimization directly on pixel values of the output image to minimize both content loss and style loss, wherein the professional processing model minimizes an Euclidean distance between neural features of the content image and the output image, and minimizes the Euclidean distance between covariance matrixes of the neural features of the style image and the output image.

A non-transitory computer-readable medium including instructions, which when executed by one or more processors, cause the processors to perform a method, comprising:

• obtaining a content image representative of a user's desired content;
• obtaining a style image representative of the user's desired style, wherein the user's desired style comprises desired colors, textures, and patterns of a stylized output image;
• obtaining a choice of either a quick processing model or a professional processing model;
• generating a stylized feature vector, wherein the stylized feature vector is generated using the whitening and coloring transformations;
• applying the stylized feature vector to generate an output image having the user's desired content and style;
• and displaying the output image;
• wherein, upon choosing the quick processing model, generating the stylized feature vector comprises:
• encoding the content image into a content feature vector by applying a series of non-linear transformations, wherein the content feature vector is generated by extracting content features from the content image by generating an initial content image vector and performing one or more whitening transformations on the initial content image vector;
• encoding the style image into a style feature vector by applying a series of non-linear transformations;
• and
• generating the stylized feature vector based on the content feature vector and the style feature vector;
• and wherein, upon choosing the professional processing model, generating the stylized feature vector comprises:
• determining an optimization parameter;
• encoding the content image into a tentative feature vector;
• encoding the style image into a tentative style feature vector,
• generating a tentative stylized feature vector based on the tentative content feature vector and tentative style feature vector,
• applying the tentative stylized feature vector to generate a tentative output image;
• repeating an optimization operation for a number of optimization iterations determined by the optimization parameter, the optimization operation comprising:
• encoding content elements of the tentative output image into a refined content feature vector,
• encoding style elements of the tentative output image into a refined style feature vector,
• determining a content loss parameter by comparing the tentative content feature vector to the refined content feature vector,
• determining a style loss parameter by comparing the tentative style feature vector to the refined style feature vector,
• optimizing the refined content feature vector and refined style feature vector based on the content loss parameter and style loss parameter,
• generating a refined stylized feature vector based on the refined content feature vector and refined style feature vector, wherein in case the refined stylized feature vector is not sufficiently optimized according to the optimization parameter, applying the refined stylized feature vector to generate a refined output image,
• saving the refined output image as the tentative output image, and repeating the optimization operation, and wherein in case the refined stylized feature vector is sufficiently optimized according to the optimization parameter, saving the refined stylized feature vector as the stylized feature vector, wherein the professional processing model performs iterative optimization directly on pixel values of the output image to minimize both content loss and style loss, wherein the professional processing model minimizes an Euclidean distance between neural features of the content image and the output image, and minimizes the Euclidean distance between covariance matrixes of the neural features of the style image and the output image.