Gaze target detection method and system

A method of training a neural network to generate a depth-infused saliency map for determining a target of a subject's gaze from an image of the subject within a scene, the method comprising:

• inputting a plurality of images of a plurality of scenes into a pre-processing unit;
• analyzing, using the pre-processing unit, the images and determining information associated with the images, the image information comprising:
• information associated with a head of the subject in each image;
• information associated with the location of the subject within the scene in each image;
• and
• depth information associated with the subject and one or more objects comprised in each image;
• inputting, from the pre-processing unit, the determined image information for each image into a training unit;
• generating, using the training unit, a plurality of pseudo-labels from the plurality of images;
• inputting the determined image information obtained for each image into the neural network (NN), wherein the NN is configured to generate a depth-infused saliency map for each image using the determined image information;
• determining, using the training unit, a loss function using the depth-infused saliency maps and the pseudo-labels;
• and
• updating, using the training unit, one or more parameters of the NN to reduce the loss function.

A system for determining a target of a subject's gaze from an image of the subject within a scene, the system comprising:

• a pre-processing unit configured to determine image information from an input image, the image information comprising:
• information associated with a head of the subject in the image;
• information associated with the location of the subject within the scene;
• and
• depth information associated with the subject and one or more objects comprised in the scene;
• a depth-infused saliency unit configured to generate a depth-infused saliency map using the determined image information;
• and
• multi-modal fusion module configured to determine the target of the subject's gaze using the determined image information and the depth-infused saliency map.

A system for training a neural network to generate a depth-infused saliency map for determining a target of a subject's gaze from an image of the subject within a scene, the system comprising:

• a pre-processing unit configured to:
• receive, as an input, a plurality of images of a plurality of scenes;
• analyze, the images and determine information associated with the images, the image information comprising:
• information associated with a head of the subject in each image;
• information associated with the location of the subject within the scene in each image;
• and
• depth information associated with the subject and one or more objects comprised in each image;
• a training unit configured to:
• receive the plurality of images of the plurality of scenes;
• receive the determined image information for each image from the pre-processing unit;
• generate a plurality of pseudo-labels from the plurality of images;
• input the determined image information obtained for each image into the neural network (NN), wherein the NN is configured to generate a depth-infused saliency map for each image using the determined image information;
• determine a loss function using the depth-infused saliency maps and the pseudo-labels received from the NN;
• and,
• update one or more parameters of the NN to reduce the loss function.