Method and system for training a machine learning model for point of gaze prediction

A system for training a machine learning model for point of gaze prediction, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

• setting the machine learning model to include a camera calibration module and a point of gaze regression module;
• obtaining a training dataset that comprises a plurality of pair of images from various driving sessions, wherein each pair of images includes a face image of a driver captured from a first camera mounted on a vehicle and a scene image of environment in front of the vehicle captured from a second camera mounted on the vehicle, wherein the scene image comprises an annotated point of gaze of the driver;
• and
• iteratively training the machine learning model using the training dataset by at least:
• processing a first pair of images in the plurality of pair of images using the camera calibration module to generate a first embedding vector;
• processing a second pair of images in the plurality of pair of images using the camera calibration module to generate a second embedding vector wherein the first pair of images and the second pair of images are from the same driving session;
• processing a third pair of images in the plurality of pair of images using the camera calibration module to generate a third embedding vector wherein the first pair of images and the third pair of images are from different driving sessions;
• extracting a facial region of interest (ROI), a right eye ROI and a left eye ROI from a first face image in the first pair of images;
• processing the facial ROI, the right eye ROI, the left eye ROI, a first scene image in the first pair of images, and the first embedding vector using the point of gaze regression module to generate a first predicted point of gaze;
• minimizing an Euclidean distance loss that measures a distance between the first predicted point of gaze and a first annotated point of gaze corresponding to the first face image on the first scene image;
• and
• minimizing a triplet loss that measures a difference between a first distance and a second distance, wherein the first distance measures a difference between the first embedding vector and the second embedding vector, and wherein the second distance measures a difference between the first embedding vector and the third embedding vector.