# US 12,002,291 B2 · Method and system for confidence level detection from eye features > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,291 B2 - Document ID: US12002291B2 - Status: Granted - Kind: B2 - Application number: 17/453,634 - Filed: 2021-11-04 - Published: 2023-01-19 - Granted: 2024-06-04 - Earliest claimed: 2021-06-04 - Assignees: Tata Consultancy Services Limited - Applicants: Tata Consultancy Services Limited - Inventors: Sushovan Chanda; Gauri Deshpande; Sachin Patel - Web page: https://patentgrep.com/patent/US12002291B2 ## CPC - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06N 3/08: Learning methods - G06T 7/73: using feature-based methods - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/457: {by analysing connectivity, e.g. edge linking, connected component analysis or slices} - G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking - G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching} - G06V 10/757: {Matching configurations of points or features} - G06V 10/761: {Proximity, similarity or dissimilarity measures} - G06V 10/768: {using context analysis, e.g. recognition aided by known co-occurring patterns} - G06V 10/82: using neural networks - G06V 10/95: {structured as a network, e.g. client-server architectures} - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06V 40/165: {using facial parts and geometric relationships} - G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships} - G06V 40/174: {Facial expression recognition} - G06V 40/193: {Preprocessing; Feature extraction} - G06T 2207/10016: Video; Image sequence - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30201: Face - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract State of art techniques attempt in extracting insights from eye features, specifically pupil with focus on behavioral analysis than on confidence level detection. Embodiments of the present disclosure provide a method and system for confidence level detection from eye features using ML based approach. The method enables generating overall confidence level label based on the subject's performance during an interaction, wherein the interaction that is analyzed is captured as a video sequence focusing on face of the subject. For each frame facial features comprising an Eye-Aspect ratio, a mouth movement, Horizontal displacements, Vertical displacements, Horizontal Squeezes and Vertical Peaks, are computed, wherein HDs, VDs, HSs and VPs are features that are derived from points on eyebrow with reference to nose tip of the detected face. This is repeated for all frames in the window. A Bi-LSTM model is trained using the facial features to derive confidence level of the subject. ## Claims ### Claim 1 A processor implemented method for confidence level detection, the method comprising: querying, by one or more hardware processors, a plurality of subjects with a predefined set of questions; capturing, by the one or more hardware processors, a plurality of video sequences corresponding to the plurality of subjects while the plurality of subjects respond to the predefined set of questions, wherein each video sequence among the plurality of video sequences captures a face of a corresponding subject from the plurality of subjects at predefined frames per second; extracting, by the one or more hardware processors, a plurality of face features for each of the plurality of video sequences, wherein the plurality of face features comprise an Eye-Aspect ratio—EAR, a mouth movement, Horizontal Displacements—HDs, Vertical Displacements—VDs, Horizontal Squeezes—HSs, and Vertical Peaks—VPs, wherein the HDs computation for each of the video sequence comprises: identifying nose tip coordinates from the face in every frame of a video sequence among the plurality of video sequences; identifying eyebrow start coordinates and eyebrow end coordinates for eyebrows of the face in every frame with respect to the nose tip coordinates; and computing a plurality of HD values by determining a Euclidean distance between the eyebrow start coordinates and the eyebrow end coordinates of each of the eyebrows for a current frame and a previous frame of the video sequence, for all the frames in the video sequence; wherein the VDs computation for each of the video sequence comprises determining for each of the eyebrows, a first highest point of a first eyebrow and a second highest point of a second eyebrow in vertical direction with respect to the identified nose tip coordinates in every frame; determining a highest vertical point for each frame by computing statistical mean of the first highest point and the second highest point; and computing a plurality of VD values by determining the Euclidean distance between the highest vertical point of the current frame and the previous frame of the video sequence, for all the frames in the video sequence; wherein the HSs computation for each of the video sequence comprises: identifying a HD threshold value by computing a statistical mean of the plurality of HD values; selecting a first subset of values from the plurality of HD values that are above the HD threshold value representing a plurality of HS values; and determining a horizontal count corresponding to number of the plurality of HS values appearing per second; wherein the VPs computation for each of the video sequence comprises: identifying a VD threshold value by computing a statistical mean of the plurality of VD; selecting a second subset of values from the plurality of VD values that are above the VD threshold value as a plurality of VP values; and determining a vertical count corresponding to number of the plurality of VP values appearing per second; and training, by the one or more hardware processors, a Neural network (NN) model, using the extracted plurality of face features from the plurality of video sequences to predict a confidence level of the subject as one among a plurality of confidence levels. ### Claim 2 (depends on 1) The method of claim 1, wherein the method further comprises predicting the confidence level of a test subject utilizing the trained NN model executed by the one or more hardware processors, wherein the trained NN model predicts the confidence level by analyzing extracted face features from a test video sequence captured when the test subject is queried with a set of unknown questions. ### Claim 3 (depends on 1) The method of claim 1, wherein the plurality of confidence levels comprise a low level, a medium level, and a high level. ### Claim 4 (depends on 1) The method of claim 1, wherein architecture of the NN model comprises: a first Bi-LSTM layer with 16 nodes followed a second Bi-LSTM layer with 16 nodes, followed by a dense layer with 8 nodes, with ReLU as its activation function, which is followed by a dropout layer, further followed by a simple RNN layer with 4 nodes, followed by the final output layer with softmax function, wherein Sparse-categorical-cross-entropy is used as the loss function, and an Adam optimizer with a batch size of 48. ### Claim 5 A system for confidence level detection, the method comprising, the system comprising: a memory storing instructions; one or more Input/Output—I/O interfaces; and one or more hardware processors coupled to the memory via the one or more I/O interfaces, wherein the one or more hardware processors are configured by the instructions to: query a plurality of subjects with a predefined set of question; capture a plurality of video sequences corresponding to the plurality of subjects while the plurality of subjects respond to the predefined set of questions, wherein each video sequence among the plurality of video sequences captures a face of a corresponding subject from the plurality of subjects at predefined frames per second—FPS; extract a plurality of face features for each of the plurality of video sequences, wherein the plurality of face features comprise an Eye-Aspect Ratio—EAR a mouth movement, Horizontal Displacements—HDs, Vertical Displacements—VDs, Horizontal Squeezes—HSs and Vertical Peaks—VPs wherein the HDs computation for each of the video sequence comprises: identifying nose tip coordinates from the face in every frame of a video sequence among the plurality of video sequences; identifying eyebrow start coordinates and eyebrow end coordinates for eyebrows of the face in every frame with respect to the nose tip coordinates; and computing a plurality of HD values by determining a Euclidean distance between the eyebrow start coordinates and the eyebrow end coordinates of each of the eyebrows for a current frame and a previous frame of the video sequence, for all the frames in the video sequence; wherein the VDs computation for each of the video sequence comprises determining, for each of the eyebrows, a first highest point of a first eywbrow and a second highest point of a second eyebrow in vertical direction with respect to the identified nose tip coordinates in every frame; determining a highest vertical point for each frame by computing statistical mean of the first highest point and the second highest point; and computing a plurality of VD values by determining the Euclidean distance between the highest vertical point of the current frame and the previous frame of the video sequence, for all the frames in the video sequence; wherein the HSs computation for each of the video sequence comprises: identifying a HD threshold value by computing a statistical mean of the plurality of HD values; selecting a first subset of values from the plurality of HD values that are above the HD threshold value representing a plurality of HS values; and determining a horizontal count corresponding to number of the plurality of HS values appearing per second; wherein the VPs computation for each of the video sequence comprises: identifying a VD threshold value by computing a statistical mean of the plurality of VD; selecting a second subset of values from the plurality of VD values that are above the VD threshold value as a plurality of VP values; and determining a vertical count corresponding to number of the plurality of VP values appearing per second; and train a Neural Network—NN model, using the extracted plurality of face features from the plurality of video sequences to predict a confidence level of the subject as one among a plurality of confidence levels. ### Claim 6 (depends on 5) The system of claim 5, wherein the one or more hardware processors are further configured to predict the confidence level of an test subject utilizing the trained NN model executed by the one or more hardware processors, wherein the trained NN model predicts the confidence level by analyzing extracted face features from the test video sequence captured when the test subject is queried with a set of unknown questions. ### Claim 7 (depends on 5) The system of claim 5, wherein the plurality of confidence levels comprise a low level, a medium level, and a high level. ### Claim 8 (depends on 5) The system of claim 5, wherein architecture of the NN model comprises: a first Bi-LSTM layer with 16 nodes followed a second Bi-LSTM layer with 16 nodes, followed by a dense layer with 8 nodes, with ReLU as its activation function, which is followed by a dropout layer, further followed by a simple RNN layer with 4 nodes, followed by the final output layer with softmax function, wherein Sparse-categorical-cross-entropy is used as the loss function, and an Adam optimizer with a batch size of 48. ### Claim 9 One or more non-transitory machine-readable information storage mediums comprising one or more instructions, which when executed by one or more hardware processors causes a method for confidence level detection, the method comprising: querying a plurality of subjects with a predefined set of questions; capturing a plurality of video sequences corresponding to the plurality of subjects while the plurality of subjects respond to the predefined set of questions, wherein each video sequence among the plurality of video sequences captures a face of a corresponding subject from the plurality of subjects at predefined frames per second—FPS; extracting a plurality of face features for each of the plurality of video sequences, wherein the plurality of face features comprise an Eye-Aspect Ratio—EAR, a mouth movement, Horizontal Displacements—HD Vertical Displacements—VDs, Horizontal Squeezes—HSs and Vertical Peaks—VPs wherein the HDs computation for each of the video sequence comprises: identifying nose tip coordinates from the face in every frame of a video sequence among the plurality of video sequences; identifying eyebrow start coordinates and eyebrow end coordinates for eyebrows of the face in every frame with respect to the nose tip coordinates; and computing a plurality of HD values by determining a Euclidean distance between the eyebrow start coordinates and the eyebrow end coordinates of each of the eyebrows for a current frame and a previous frame of the video sequence, for all the frames in the video sequence; wherein the VDs computation for each of the video sequence comprises determining for each of the eyebrows, a first highest point of a first eyebrow and a second highest point of a second eyebrow in vertical direction with respect to the identified nose tip coordinates in every frame; determining a highest vertical point for each frame by computing statistical mean of the first highest point and the second highest point; and computing a plurality of VD values by determining the Euclidean distance between the highest vertical point of the current frame and the previous frame of the video sequence, for all the frames in the video sequence; wherein the HSs computation for each of the video sequence comprises: identifying a HD threshold value by computing a statistical mean of the plurality of HD values; selecting a first subset of values from the plurality of HD values that are above the HD threshold value representing a plurality of HS values; and determining a horizontal count corresponding to number of the plurality of HS values appearing per second; wherein the VPs computation for each of the video sequence comprises: identifying a VD threshold value by computing a statistical mean of the plurality of VD; selecting a second subset of values from the plurality of VD values that are above the VD threshold value as a plurality of VP values; and determining a vertical count corresponding to number of the plurality of VP values appearing per second; and training a Neural Network—NN model, using the extracted plurality of face features from the plurality of video sequences to predict a confidence level of the subject as one among a plurality of confidence levels. ### Claim 10 (depends on 9) The one or more non-transitory machine-readable information storage mediums of claim 9, further comprising predicting the confidence level of a test subject utilizing the trained NN model executed by the one or more hardware processors, wherein the trained NN model predicts the confidence level by analyzing extracted face features from a test video sequence captured when the test subject is queried with a set of unknown questions. ### Claim 11 (depends on 9) The one or more non-transitory machine-readable information storage mediums of claim 9, wherein the plurality of confidence levels comprise a low level, a medium level, and a high level. ### Claim 12 (depends on 9) The one or more non-transitory machine-readable information storage mediums of claim 9, wherein architecture of the NN model comprises: a first Bi-LSTM layer with 16 nodes followed a second Bi-LSTM layer with 16 nodes, followed by a dense layer with 8 nodes, with ReLU as its activation function, which is followed by a dropout layer, further followed by a simple RNN layer with 4 nodes, followed by the final output layer with softmax function, wherein Sparse-categorical-cross-entropy is used as the loss function, and an Adam optimizer with a batch size of 48.