Image colorization using machine learning
AssigneeGoogle Inc.
InventorNori Kanazawa
Life of this patent
Estimating the term…
Abstract
Implementations described herein relate to methods, systems, and computer-readable media to train and use a machine-learning model to colorize a grayscale image that depicts a person. In some implementations, a computer-implemented method includes receiving the grayscale image. The method further includes generating a colorized image based on the grayscale image as output of a trained convolutional neural network (CNN) by providing the grayscale image as input to the trained CNN. In some implementations, the trained CNN performs part segmentation to detect one or more parts of the person and colorizes the grayscale image.
Claims20
A computer-implemented method to train a convolutional neural network to colorize grayscale images, the method comprising:
- providing training data comprising a plurality of training color images that each depict one or more persons, corresponding grayscale training images, and part annotations for the one or more persons in each training color image;
- and
- for each grayscale training image,
- providing the grayscale training image as input to the convolutional neural network;
- obtaining, as output of the convolutional neural network,
- an output feature vector that is descriptive of a colorized image based on the grayscale training image, and
- part annotations for the one or more persons in the grayscale training image;
- obtaining a perceptual loss value from a pixel discriminator based on a label prediction, by the pixel discriminator, of whether the colorized image is an original color image or a predicted color image;
- and
- modifying a parameter of the convolutional neural network based on the perceptual loss value.
A system, comprising:
- a memory with instructions stored thereon;
- and
- a processing device, coupled to the memory, wherein the processing device is configured to access the memory and execute the instructions, and wherein the instructions cause the processing device to perform or control performance of operations comprising:
- providing training data comprising a plurality of training color images that each depict one or more persons, corresponding grayscale training images, and part annotations for the one or more persons in each training color image;
- and
- for each grayscale training image,
- providing the grayscale training image as input to a convolutional neural network;
- obtaining, as output of the convolutional neural network,
- an output feature vector that is descriptive of a colorized image based on the grayscale training image, and
- part annotations for the one or more persons in the grayscale training image;
- obtaining a perceptual loss value from a pixel discriminator based on a label prediction, by the pixel discriminator, of whether the colorized image is an original color image or a predicted color image;
- and
- modifying a parameter of the convolutional neural network based on the perceptual loss value.
A non-transitory computer-readable medium with instructions stored thereon that, responsive to execution by a processing device, causes the processing device to perform or control performance of operations comprising:
- providing training data comprising a plurality of training color images that each depict one or more persons, corresponding grayscale training images, and part annotations for the one or more persons in each training color image;
- and
- for each grayscale training image,
- providing the grayscale training image as input to a convolutional neural network;
- obtaining, as output of the convolutional neural network,
- an output feature vector that is descriptive of a colorized image based on the grayscale training image, and
- part annotations for the one or more persons in the grayscale training image;
- obtaining a perceptual loss value from a pixel discriminator based on a label prediction, by the pixel discriminator, of whether the colorized image is an original color image or a predicted color image;
- and
- modifying a parameter of the convolutional neural network based on the perceptual loss value.
Dependencies are inferred from the claim text.