Generating 3d facial models & animations using computer vision architectures
AssigneeVidalign Inc.
InventorsAmrish Baskaran, Adheesh Chatterjee, James Tristan Guerrera-Sapone, Sammie Kim
Life of this patent
Estimating the term…
Abstract
This disclosure relates to improved techniques for generating three-dimensional (3D) facial models and animations from two-dimensional (2D) electronic media files. Other embodiments are disclosed herein as well.
Claims18
A method implemented for generating a three-dimensional (3D) facial model from two-dimensional (2D) electronic media files via execution of computing instructions by one or more processing devices and stored on one or more non-transitory computer-readable media, the method comprising:
- receiving, at a computer vision system, an electronic video file comprising 2D video content;
- extracting, using an extraction layer of the computer vision system, facial object data corresponding to a facial object captured in the electronic video file, the facial object data at least comprising facial landmark data, edge data, and facial segmentation data corresponding to the at least one facial object;
- executing a fitting model configured to:
- derive shape parameters for the facial object based, at least in part, on an analysis of the facial object data extracted across multiple frames of the electronic video file;
- derive expression parameters corresponding to the facial object for each of the multiple frames of the electronic video file;
- derive localization data for the facial object that determines a location of the facial object in each of the multiple frames of the electronic video file;
- and
- derive camera parameters indicating a position, an orientation, and a camera pose for a camera that captured in the electronic video file and the facial object;
- generating, using a 3D morphable face model of the computer vision system, a 3D facial model corresponding to the facial object based, at least in part on, the shape parameters, the expression parameters, the localization data, and the camera parameters derived by the fitting model.
A system for generating a three-dimensional (3D) facial model from two-dimensional (2D) electronic media files, wherein the system includes one or more computing devices comprising one or more processing devices and one or more non-transitory storage devices that store instructions, wherein execution of the instructions by the one or more processing devices causes the one or more computing devices to:
- receive, at a computer vision system, an electronic video file comprising 2D video content;
- extract, using an extraction layer of the computer vision system, facial object data corresponding to a facial object captured in the electronic video file, the facial object data at least comprising facial landmark data, edge data, and facial segmentation data corresponding to the at least one facial object;
- execute a fitting model configured to:
- derive shape parameters for the facial object based, at least in part, on an analysis of the facial object data extracted across multiple frames of the electronic video file;
- derive expression parameters corresponding to the facial object for each of the multiple frames of the electronic video file;
- derive localization data for the facial object that determines a location of the facial object in each of the multiple frames of the electronic video file;
- and
- derive camera parameters indicating a position, an orientation, and a camera pose for a camera that captured in the electronic video file and the facial object;
- generate, using a 3D morphable face model of the computer vision system, a 3D facial model corresponding to the facial object based, at least in part on, the shape parameters, the expression parameters, the localization data, and the camera parameters derived by the fitting model.
A computer program product, the computer program product comprising a non-transitory computer-readable medium including instructions for causing a computing device to:
- receive, at a computer vision system, an electronic video file comprising two-dimensional (2D) video content;
- extract, using an extraction layer of the computer vision system, facial object data corresponding to a facial object captured in the electronic video file, the facial object data at least comprising facial landmark data, edge data, and facial segmentation data corresponding to the at least one facial object;
- execute a fitting model configured to:
- derive shape parameters for the facial object based, at least in part, on an analysis of the facial object data extracted across multiple frames of the electronic video file;
- derive expression parameters corresponding to the facial object for each of the multiple frames of the electronic video file;
- derive localization data for the facial object that determines a location of the facial object in each of the multiple frames of the electronic video file;
- and
- derive camera parameters indicating a position, an orientation, and a camera pose for a camera that captured in the electronic video file and the facial object;
- generate, using a three-dimensional (3D) morphable face model of the computer vision system, a 3D facial model corresponding to the facial object based, at least in part on, the shape parameters, the expression parameters, the localization data, and the camera parameters derived by the fitting model.
Dependencies are inferred from the claim text.