US 12,621,417 B1Granted

Face-aware relighting of live video content

AssigneeAmazon Technologies, Inc.

InventorsPrerit Jaiswal, Mehmet Umut Isik, Srikanth Venkata Tenneti, Samuel J Wilson, Amritpal Singh Saini, Parisa Rahimzadeh, Amalavoyal Chari, Michael Mark Goodwin

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

Systems and methods are provided for modifying video content to improve lighting of a person's face depicted within the video content. Pixels depicting skin may be detected in a first frame of the video content. Transformation parameters may then be determined based on intensity values of the pixels depicting skin, where the transformation parameters represent adjusted pixel intensity values determined to improve at least one of brightness or contrast of the pixels depicting skin. Based on the transformation parameters, intensity association data may be generated and stored that associates each possible input pixel intensity value in at least one channel with a corresponding adjusted pixel intensity value. The stored intensity association data as determined with respect to the first frame may then be reused to modify intensity values for a series of frames of the video content.

Claims18

Dependent claims

A system comprising:

memory;
and
at least one computing device configured with computer-executable instructions that, when executed, cause the at least one computing device to:
obtain a first frame of video content, wherein the video content depicts at least a human face as captured by a camera, wherein the camera is part of the system or in local communication with the system;
determine a bounding region within the first frame that includes depiction of the human face, wherein the bounding region is determined at least in part by providing the first frame as input to a machine learning model trained to detect depiction of human faces within an input image;
apply segmentation to image data within the bounding region of the first frame to identify pixels depicting skin;
determine transformation parameters based on intensity values of the pixels depicting skin, wherein the transformation parameters represent at least adjusted pixel intensity values determined by the system to improve at least one of brightness or contrast of the pixels depicting skin, wherein the transformation parameters are determined with respect to at least one channel of a color space;
store a lookup table based on the transformation parameters, wherein the lookup table comprises, for each possible input pixel intensity value in the at least one channel of the color space, a corresponding adjusted pixel intensity value determined by the system to improve the at least one of brightness or contrast of the pixels depicting skin;
generate a relit first frame of the video content, wherein the relit first frame of the video content is generated at least in part by changing an intensity value, for the at least one channel, of each pixel in the first frame to a corresponding adjusted pixel intensity value identified in the lookup table;
obtain a second frame of the video content;
determine, based at least in part on a comparison of lighting in the second frame relative to the first frame, to reuse the lookup table as stored with respect to the first frame;
and
based on the determination to reuse the lookup table, generate a relit second frame of the video content, wherein the relit second frame of the video content is generated at least in part by changing an intensity value, for the at least one channel, of each pixel in the second frame to a corresponding adjusted pixel intensity value identified in the lookup table, wherein the relit second frame of the video content is generated without detecting the human face within the second frame and without applying segmentation to the image data of the second frame.

10 more elements

A computer-implemented method comprising:

obtaining a first frame of video content, wherein the video content depicts at least a human face as captured by a camera;
applying segmentation to image data of the first frame to identify pixels depicting skin;
determining transformation parameters based at least in part on intensity values of the pixels depicting skin, wherein the transformation parameters represent at least adjusted pixel intensity values determined to improve at least one of brightness or contrast of the pixels depicting skin, wherein the transformation parameters are determined with respect to at least one channel of a color space;
based on the transformation parameters, storing intensity association data that associates each possible input pixel intensity value in the at least one channel of the color space with a corresponding adjusted pixel intensity value;
generating a modified first frame of the video content, wherein the modified first frame of the video content is generated at least in part by changing an intensity value, for the at least one channel, of each pixel in the first frame to a corresponding adjusted pixel intensity value identified in the stored intensity association data;
obtaining a second frame of the video content;
and
generating a modified second frame of the video content, wherein the modified second frame of the video content is generated at least in part by changing an intensity value, for the at least one channel, of each pixel in the second frame to a corresponding adjusted pixel intensity value identified in the stored intensity association data, wherein the modified second frame of the video content is generated without applying segmentation to the image data of the second frame.

5 more elements

One or more non-transitory computer readable media including computer-executable instructions that, when executed by a computing system, cause the computing system to perform operations comprising:

obtaining a first frame of video content, wherein the video content depicts at least a human face as captured by a camera;
identifying, from among a plurality of pixels within the first frame, pixels depicting skin;
determining transformation parameters based at least in part on intensity values of the pixels depicting skin, wherein the transformation parameters are determined with respect to at least one channel of a color space;
based on the transformation parameters, storing intensity association data that associates each possible input pixel intensity value in the at least one channel of the color space with a corresponding adjusted pixel intensity value;
generating a modified first frame of the video content, wherein the modified first frame of the video content is generated at least in part by changing an intensity value, for the at least one channel, of each pixel in the first frame to a corresponding adjusted pixel intensity value identified in the stored intensity association data;
obtaining a second frame of the video content;
and
generating a modified second frame of the video content, wherein the modified second frame of the video content is generated at least in part by changing an intensity value, for the at least one channel, of each pixel in the second frame to a corresponding adjusted pixel intensity value identified in the stored intensity association data.

5 more elements

Dependencies are inferred from the claim text.