A computer-implemented method of generating at least one dataset associated with one or more objects of at least one image of a scene of a real environment configured for input to a computer-implemented process for annotating at least one image frame of a scene of a real environment for input to and training at least one machine learning model, the method comprising:
- receiving, by at least one computing device, at least one image of a scene of a real environment captured by an image sensor operated by a user;
- receiving, by the at least one computing device, audio data from the user detected by at least one audio sensor and gaze tracking data of the user detected by at least one human gaze tracking sensor while capturing the at least one image by the user;
- performing, by at least one computing device, an audio parsing process parsing at least part of the received audio data from the user into text format and generating a set of terms determined in the parsing process indicative of a user's articulated task or activity while operating the image sensor;
- pre-allocating objects of interest from the generated set of terms determined in the parsing process, associating a probability with each of the pre-allocated objects of interest, and generating a scene objects of interest list associated with the at least one image comprising a respective object category and associated probability for each of the pre-allocated objects of interest;
- generating a gaze heatmap from the received gaze tracking data;
- processing at least part of the at least one image through a panoptic segmentation model and generating a set of segmentation masks, each segmentation mask of the set of segmentation masks having a respective assigned object category and panoptic segmentation confidence;
- and
- generating at least one dataset comprising a set of ranked objects of interest with objects of interest ranked according to at least one aggregated score determined based on the generated scene objects of interest list, the generated heatmap and generated set of segmentation masks, configured for input to a computer-implemented process for annotating at least one image frame of a scene of a real environment for input to and training at least one machine learning model.