Monitoring and optimization of human-device interactions

A system for detecting an inattentive state of a person, comprising:

• a video game system including a controller to provide user input and an electronic display at which a person's eyes are directed at configured to present a video game for viewing and playing by the person;
• an image capture device positioned to capture digital video frames of the person's head and face;
• a computing device configured to receive the captured digital video frames and extract visual cues of eyes and mouth of the person's head and face from each of the captured digital video frames, the computing device further configured to
• increment a frame counter when
• an average of an aspect ratio of a left eye and a right eye of the person exceeds a predetermined threshold in a current video frame, or
• a yawn is detected from the shape of the mouth in the current video frame and the average of the aspect ratio of the left eye and the right eye of the person is less than or equal to the predetermined threshold,
• reset the frame counter when no yawn is detected in the current video frame,
• generate a first electronic signal when the frame counter reaches a predetermined number of frames and resetting the frame counter;
• the computing device is further configured to detect
• an eye gaze of the person is to the left or to the right of the screen by
• extracting the eye portions from the face in the current video frame,
• determining a first ratio of white pixels corresponding to the right side and the left side of the left iris,
• determining a second ratio of white pixels corresponding to the right side and the left side of the right iris,
• determining the eye gaze is to the right when an average of the first and second ratios is below a first eye gaze threshold, determining the eye gaze is to the left when the average is greater than a second eye gaze threshold, and determining the eye gaze is centered when the average is between the first and second eye gaze thresholds, and
• the computing device generates a second electronic signal when the eye gaze is determined as being to the right or the left,
• wherein the computing device is configured to change currently displayed content of the electronic display device in real time to display new content non-native to the video game in response to at least one of the first and the second electronic signals, indicating the inattentive state of the person.