Method and system for performing video-based automatic identity verification

A system for performing video-based automatic identity verification of a first user, the system comprising a processor, wherein the processor is configured to:

• generate one or more prompts;
• capture a video, wherein the video comprises the first user speaking the one or more prompts wherein face of the first user is a first human object;
• capture an image of an identification document comprising an image of a second user wherein face of the second user is a second human object;
• generate a deepfake detection score of the video based on one or more deepfake-techniques;
• detect a correctness of a speech in the video based on the deepfake detection score, a visual cues match, and a transcription match, wherein the speech is associated with the first user speaking the one or more prompts, and wherein the visual cues match comprises one of a valid lip contour movement match and an invalid lip contour movement match;
• detect in one or more frames of the video, the first human object based on the correctness of the speech and in the image of the identification document, the second human object;
• generate a first similarity score, and a second similarity score, based on the detection of the first human object and the second human object,
• wherein the first similarity score is generated based on a similarity within a plurality of feature vectors of the first human object present in a plurality of frames of the video, and
• the second similarity score is generated based on a similarity between the plurality of feature vectors of the first human object present in the plurality of frames of the video and a feature vector of the second human object present in the image of the identification document;
• and
• automatically perform the identity verification of the first user, based on the first similarity score, and the second similarity score,
• wherein the visual cues match is performed based on:
• a reiterative performance of a set of steps until an occurrence of an end-of-sentence token, wherein the set of steps comprises:
• a detection of a set of facial landmarks on the first human object in the one or more frames of the video by a facial fiducial point detector unit;
• an extraction of one or more regions of interest (ROIs) based on the set of facial landmarks by the facial fiducial point detector unit:
• a performance of a visual speech recognition on the ROIs by a visual speech recognition (VSR) model unit,
• wherein the VSR model unit is a neural network based unit comprising a conformer encoder and the transformer decoder:
• an obtaining of a set of lip features based on the performance of the VSR on the ROIs by the conformer encoder, wherein the set of lip features comprises a corresponding vector for one or more sets of frames corresponding to a phoneme;
• and
• a prediction of a next most probable phoneme based on the set of lip features by the transformer decoder using a language model unit:
• a generation of one or more word sequence predictions based on the set of steps by the transformer decoder:
• a prediction of a list of most probable word sequence predictions based on the one or more word sequence predictions by a beam search unit, wherein the list comprises a pre-defined number of the most probable word sequence predictions;
• a mapping of each word in the list of the most probable word sequence predictions to a corresponding nearest word of interest from a pre-defined list of probable words, by the beam search unit using one or more mapping techniques;
• and
• a performance of the visual cues match based on the mapping.