# US 9,519,998 B2 · Three dimensional montage generation system and method based on two dimensional single image > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,519,998 B2 - Document ID: US9519998B2 - Status: Granted - Kind: B2 - Application number: 13/790,122 - Filed: 2013-03-08 - Published: 2013-09-12 - Granted: 2016-12-13 - Earliest claimed: 2012-03-09 - Assignees: Korea Institute of Science and Technology - Applicants: Korea Institute of Science and Technology - Inventors: Ig Jae Kim; Yu Jin Hong - Web page: https://patentgrep.com/patent/US9519998B2 ## CPC - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06F 18/28: Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries - G06V 10/772: Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries - G06V 20/647: {by matching two-dimensional images to three-dimensional objects} - G06V 40/16: Human faces, e.g. facial parts, sketches or expressions - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The present disclosure relates to a three-dimensional montage generation system and method based on a two-dimensional single image. An embodiment of the present disclosure may generate a three-dimensional montage in an easy, fast and accurate way by using a two-dimensional front face image data, and estimate face portions, which cannot be restored by using a single photograph, in a statistic way by using a previously prepared face database. Accordingly, an embodiment of the present disclosure may generate a three-dimensional personal model from a single two-dimensional front face photograph, and depth information such as nose height, lip protrusion and eye contour may be effectively estimated by means of statistical distribution and correlation of data. ## Claims ### Claim 1 A three-dimensional montage generation system based on a single two-dimensional image, comprising: a feature point extraction unit configured to extract a feature point of an input two-dimensional face image; a database configured to store three-dimensional face image information; a depth estimation unit configured to obtain a correlation between the three-dimensional face image information and the feature point, and estimate a depth of each portion of the two-dimensional face image based on the correlation; a model transformation unit configured to project the two-dimensional face image to a three-dimensional template model and transform the three-dimensional template model based on the feature point and the estimated depth information of each portion of the two-dimensional face image; and a montage generating unit configured to generate a three-dimensional montage by rendering the two-dimensional face image to the transformed three-dimensional template model, wherein the correlation is based on a comparison of a facial features characteristic of a predetermined facial type; and wherein the depth of each portion of the two-dimensional face image is estimated based on a comparison of an average facial shape and predetermined depth of features of the facial features characteristic of the predetermined facial type with the feature point of the two-dimensional face image. ### Claim 2 (depends on 1) The three-dimensional montage generation system based on a single two-dimensional image according to claim 1, wherein the model transformation unit calculates translations, rotations and scales of the two-dimensional face image and the three-dimensional template model, and repeats the projecting operation and the calculating operation until an error between the two-dimensional face image and the three-dimensional template model is less than a preset threshold value. ### Claim 3 (depends on 2) The three-dimensional montage generation system based on a single two-dimensional image according to claim 2, wherein the model transformation unit transforms a facial shape of the three-dimensional template model using a blend shape technique. ### Claim 4 (depends on 1) The three-dimensional montage generation system based on single a two-dimensional image according to claim 1, wherein the depth estimation unit estimates a depth of each portion of the two-dimensional face image by using a K-Nearest Neighbor (KNN) algorithm. ### Claim 5 (depends on 4) The three-dimensional montage generation system based on a single two-dimensional image according to claim 4, wherein the depth estimation unit selects three-dimensional face image information from the database using the correlation to estimate the depth of each portion of the two-dimensional face image, and wherein a difference between a feature point of the selected three-dimensional face image information and the feature point of the two-dimensional face image is within a predetermined range. ### Claim 6 (depends on 1) The three-dimensional montage generation system based on a single two-dimensional image according to claim 1, wherein the montage generating unit generates a three-dimensional montage by estimating information of an omitted portion of the two-dimensional face image based on the correlation. ### Claim 7 (depends on 1) The three-dimensional montage generation system based on a single two-dimensional image according to claim 1, wherein the depth estimation unit estimates the depth of each portion of the two-dimensional face image based further on a depth value of a three-dimensional face image included in the three-dimensional face image information. ### Claim 8 (depends on 1) The three-dimensional montage generation system based on a single two-dimensional image according to claim 1, wherein the transformation of the three-dimensional template model further comprises obtaining one of a center location, a vertex region, or a weight of each face portion. ### Claim 9 A three-dimensional montage generation method based on a single two-dimensional image, comprising: extracting a feature point of a two-dimensional face image; obtaining a correlation between the feature point and three-dimensional face image information stored in a database; estimating a depth of each portion of the two-dimensional face image based on the correlation; transforming a three-dimensional template model based on the feature point and the estimated depth information of each portion of the two-dimensional face image; and generating a three-dimensional montage by rendering the two-dimensional face image to the transformed three-dimensional template model, wherein the correlation is based on a comparison of a facial features characteristic of a predetermined facial type, as included in the three-dimensional face image information stored in the database; and wherein the depth of each portion of the two-dimensional face image is estimated based on a comparison of an average facial shape and predetermined depth of features of the facial features characteristic of the predetermined facial type with the feature point of the two-dimensional face image. ### Claim 10 (depends on 9) The three-dimensional montage generation method based on a single two-dimensional image according to claim 9, wherein the transforming comprises: projecting the two-dimensional face image to the three-dimensional template model; calculating translations, rotations and scales of the two-dimensional face image and the three-dimensional template model; and repeating the projecting and the calculating until an error between the two-dimensional face image and the three-dimensional template model is less than a preset threshold value. ### Claim 11 (depends on 9) The three-dimensional montage generation method based on a single two-dimensional image according to claim 9, wherein the transforming comprises: transforming a facial shape of the three-dimensional template model using a blend shape technique. ### Claim 12 (depends on 9) The three-dimensional montage generation method based on a single two-dimensional image according to claim 9, wherein the estimating is performed by using a K-Nearest Neighbor (KNN) algorithm. ### Claim 13 (depends on 12) The three-dimensional montage generation method based on a single two-dimensional image according to claim 12, wherein the estimating comprises selecting three-dimensional face image information from the database using the correlation to estimate the depth of each portion of the two-dimensional face image, and wherein a difference between a feature point of the selected three-dimensional face image information and the feature point of the two-dimensional face image is within a predetermined range. ### Claim 14 (depends on 9) The three-dimensional montage generation method based on a single two-dimensional image according to claim 9, wherein the generating comprises: estimating information of an omitted portion of the two-dimensional face image based on the correlation; and generating a three-dimensional montage based on the estimated information. ### Claim 15 (depends on 9) The three-dimensional montage generation method based on a single two-dimensional image according to claim 9, wherein the estimating is further based on a depth value of a three-dimensional face image included in the three-dimensional face image information. ### Claim 16 A three-dimensional montage generation system based on a single two-dimensional image, comprising: a processor configured to extract a feature point of an input two-dimensional face image; store three-dimensional face image information; obtain a correlation between the three-dimensional face image information and the feature point, and estimate a depth of each portion of the two-dimensional face image based on the correlation; project the two-dimensional face image to a three-dimensional template model and transform the three-dimensional template model based on the feature point and the estimated depth information of each portion of the two-dimensional face image; and generate a three-dimensional montage by rendering the two-dimensional face image to the transformed three-dimensional template model, wherein the correlation is based on a comparison of a facial features characteristic of a predetermined facial type, and wherein the depth of each portion of the two-dimensional face image is estimated based on a comparison of an average facial shape and predetermined depth of features of the facial features characteristic of the predetermined facial type with the feature point of the two-dimensional face image.