Children visual attention abnormal screening method, involves extracting eye movement, facial expression and head movement multi-mode characteristic of children based on multimodal data learning

A method for screening mobile visual attention abnormalities in children based on multimodal data learning, comprising:

• step 1:
• setting up at least a calibration video and a testing video, such that head-face videos of m children with visual attention abnormalities and m children with normal development while watching the calibration video and the testing video on smartphones are recorded, respectively, wherein m represents an integer having a value of two or greater;
• step 2:
• constructing an eye-tracking estimation model, and subsequently estimating fixation point location from frames of the head-face videos corresponding to the testing video based on the eye-tracking estimation model, wherein eye-tracking features are extracted from fixation point jump magnitude, fixation point jump angle, and area of interest;
• step 3:
• extracting facial expression features and head movement features from the head-face videos corresponding to the testing video and subsequently training a Long Short-Term Memory (LSTM) network to fuse different modal features to establish a mapping from multimodal features to category labels;
• and
• step 4:
• in a testing stage, head-face video of a child to be classified while watching videos on smartphones is recorded, such that the multimodal features including eye-tracking, facial expressions, and head movements are extracted and input into the trained LSTM network model to determine whether the child is abnormal.