Video Based Facial Recognition for Customer Verification at Touchless Checkout

A method of performing facial recognition comprising:

• (a) capturing a series of video frames of an individual face using a camera;
• (b) generating a series of two-dimensional models of the individual face at various angles, each model based upon a subset of video frames;
• (c) combining the series of two-dimensional models of the individual face with a projecting matrix to create a three-dimensional model of the individual face;
• (d) rotating the three-dimensional model to a reference frame that corresponds to a frontal view of the individual face;
• (e) generating a two-dimensional mean reference map by projecting from the rotated three-dimensional model, the mean reference map made up of a series of n facial reference points, where n is an integer greater than one;
• (f) accessing a facial database to compare the mean reference map to a series of previously-stored reference maps each of the previously-stored reference maps including a set of previously-stored reference points;
• (g) identifying as a match an individual whose previously-stored reference map in a database of individuals most closely matches the mean reference map;
• (h) incrementing a confidence score for the individual identified as the match and applying a bonus to the confidence score when the individual identified as the match was previously identified as a match for another subset of the series of video frames,
• repeating generating processes (b) through (h) for each subset of the series of video frames until the confidence score for one individual from the database of individuals exceeds a confidence threshold, and
• identifying as a selected individual the individual from the database of individuals whose confidence score exceeded the confidence threshold.

A facial recognition system comprising:

• a station including at least one video camera for capturing a series of video frames of an individual face;
• and
• a face image processing server for
• (a) generating a series of two-dimensional models of the individual face at various angles, each model based upon one of the series of video frames,
• (b) combining the series of two-dimensional models of the individual face with a projecting matrix, wherein each matrix is represented by F3D=[x00 . . . xn, y00 . . . yn, z00 . . . zn], where xn is an x coordinate value of the nth reference point, yn is a y coordinate value of the nth reference point, and zn is a z coordinate value of the nth reference point, to create a three-dimensional model of the individual face, wherein the three-dimensional model is generated by multiplying the inverse of a projecting matrix M defined as M=[1 0 0, 0 1 0]*RRotation where RRotation=[cos α cos β cos α sin β sin γ−sin α cos γ cos α sin β cos γ+sin α sin γ, sin α cos β sin α sin β sin γ+cos α cos γ sin α sin β cos γ−cos α sin γ, −sin β cos β sin γ cos β cos γ] where α is yaw, β is pitch, and γ is roll, times a matrix of two-dimensional coordinates of n facial reference points.
• (c) rotating the three-dimensional model to a reference frame that corresponds to a frontal view of the individual face,
• (d) generating a two-dimensional mean reference map by projecting from the rotated three-dimensional model, the mean reference map made up of a series of n facial reference points, where n is an integer greater than one,
• (e) accessing a facial database to compare the mean reference map to a series of previously-stored reference maps, each of the previously-stored reference maps including a set of previously-stored reference points,
• (f) identifying as a match an individual whose previously-stored reference map in a database of individuals most closely matches the mean reference map,
• (g) incrementing a confidence score for the individual identified as the match and applying a bonus to the confidence score when the individual identified as the match was previously identified as a match for another of the series of video frames,
• repeating generating processes (a) through (g) for each frame of the series of video frames until the confidence score for one individual from the database of individuals exceeds a confidence threshold, and
• identifying as a selected individual the individual from the database of individuals whose confidence score exceeded the confidence threshold.