# US 2026/0311464 A1 · A computer-implemented method of generating at least one dataset associated with objects of at least one image of a real environment for annotating at least one image frame for training a machine learning model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0311464 A1
- Document ID: US20260311464A1
- Status: Published application
- Kind: A1
- Application number: 19/481,681
- Filed: 2024-04-29
- Published: 2026-10-08
- Earliest claimed: 2023-05-05
- Assignees: Ramblr GmbH
- Applicants: Ramblr GmbH
- Inventors: Drago GUGGIANA NILO; Philipp SCHUBERT; Jennifer KREBS; Thomas ALT
- Web page: https://patentgrep.com/patent/US20260311464A1

## CPC

- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 40/205: Parsing
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 20/20: in augmented reality scenes
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The invention relates to computer-implemented methods of generating at least one dataset associated with one or more objects of at least one image of a scene of a real environment, in particular egocentric video data. Such generated at least one dataset is configured for input to a computer-implemented process for annotating at least one image frame of a scene of a real environment, particularly for input to and training at least one image-based machine learning model. The invention also relates to computer-implemented methods of annotating at least one image frame of a scene of a real environment, in particular egocentric video data, configured for input to and training at least one image-based machine learning model, and to a corresponding computer program. The invention provides a customized computer-implemented method of generating at least one dataset associated with one or more objects of at least one image of a scene of a real environment, which enables more efficient classification of digital images or videos and training of image-based machine learning models, such as employed in augmented reality applications or autonomous driving.

## Claims

### Claim 1

A computer-implemented method of generating at least one dataset associated with one or more objects of at least one image of a scene of a real environment configured for input to a computer-implemented process for annotating at least one image frame of a scene of a real environment for input to and training at least one machine learning model, the method comprising:
receiving, by at least one computing device, at least one image of a scene of a real environment captured by an image sensor operated by a user;
receiving, by the at least one computing device, audio data from the user detected by at least one audio sensor and gaze tracking data of the user detected by at least one human gaze tracking sensor while capturing the at least one image by the user;
performing, by at least one computing device, an audio parsing process parsing at least part of the received audio data from the user into text format and generating a set of terms determined in the parsing process indicative of a user's articulated task or activity while operating the image sensor;
pre-allocating objects of interest from the generated set of terms determined in the parsing process, associating a probability with each of the pre-allocated objects of interest, and generating a scene objects of interest list associated with the at least one image comprising a respective object category and associated probability for each of the pre-allocated objects of interest;
generating a gaze heatmap from the received gaze tracking data;
processing at least part of the at least one image through a panoptic segmentation model and generating a set of segmentation masks, each segmentation mask of the set of segmentation masks having a respective assigned object category and panoptic segmentation confidence; and
generating at least one dataset comprising a set of ranked objects of interest with objects of interest ranked according to at least one aggregated score determined based on the generated scene objects of interest list, the generated heatmap and generated set of segmentation masks, configured for input to a computer-implemented process for annotating at least one image frame of a scene of a real environment for input to and training at least one machine learning model.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the set of ranked objects of interest is configured to be used in a computer-implemented process for annotating at least one image frame of a scene of a real environment for input to and training at least one machine learning model of an augmented reality application.

### Claim 3 (depends on 1)

The method according to claim 1, further comprising:
providing a list of universal objects of interest not processed from the at least one image, and augmenting the scene objects of interest list with the list of universal objects of interest.

### Claim 4 (depends on 1)

The method according to claim 1, further comprising:
generating an audio-visual ground truth comprising an object category and respective location information for at least one object of the at least one image from the received audio data and gaze tracking data of the user; and
matching the generated audio-visual ground truth with a spatially corresponding segmentation mask of the generated set of segmentation masks and its assigned object category.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the at least one aggregated score is determined based on a weighted sum of (i) the probability of the respective object of interest contained in the scene objects of interest list, (ii) the respective gaze heat for the corresponding segmentation mask having an object category corresponding to the respective object of interest, and (iii) the respective panoptic segmentation confidence of the corresponding segmentation mask having an object category corresponding to the respective object of interest.

### Claim 6 (depends on 1)

The method according to claim 1, wherein determining the at least one aggregated score comprises:
matching object categories included in the scene objects of interest list with respective object categories of the generated set of segmentation masks, and
matching of segmentation masks of the generated set of segmentation masks and the gaze heatmap, wherein optionally determining the at least one aggregated score comprises assigning a score from the scene objects of interest list to a respective object category which could not be matched to an object category of the generated segmentation masks.

### Claim 7 (depends on 6)

The method according to claim 6, further comprising adding at least one region of interest originating from the gaze heat above a predetermined threshold that was not matched to a generated segmentation mask to the scene objects of interest list, wherein the at least one region of interest is assigned an undefined object category.

### Claim 8

A computer-implemented method of generating at least one dataset associated with one or more objects of at least one image of a scene of a real environment configured for input to a computer-implemented process for annotating at least one image frame of a scene of a real environment for input to and training at least one machine learning model, the method comprising:
receiving, by at least one computing device, at least one image of a scene of a real environment captured by an image sensor operated by a user;
receiving, by the at least one computing device, audio data from the user detected by at least one audio sensor and gaze tracking data of the user detected by at least one human gaze tracking sensor while capturing the at least one image by the user;
performing, by at least one computing device, an audio parsing process parsing at least part of the received audio data from the user and generating parsed audio data indicative of a user's articulated task or activity or an object of interest of the scene while operating the image sensor;
generating a gaze heatmap from the received gaze tracking data,
processing at least part of the at least one image through a panoptic segmentation model and generating a set of segmentation masks, each segmentation mask of the set of segmentation masks having a respective assigned object category;
generating an audio-visual ground truth comprising location information and a respective object category for at least one object of the at least one image from the parsed audio data and gaze heatmap; and
matching the generated audio-visual ground truth with a spatially corresponding segmentation mask of the generated set of segmentation masks and its assigned object category, and generating at least one dataset of the audio-visual ground truth comprising at least one data label indicative of the respective object category and the matched segmentation mask, the generated at least one dataset configured for input to a computer-implemented process for annotating at least one image frame of a scene of a real environment for input to and training at least one machine learning model.

### Claim 9 (depends on 1)

The method according to claim 1, wherein the image sensor and optionally the human gaze tracking sensor are part of a wearable computing device worn by the user while looking at the scene of the real environment.

### Claim 10 (depends on 1)

A computer-implemented method of annotating at least one image frame of a scene of a real environment configured for input to and training at least one machine learning model, the method comprising:
receiving, by at least one computing device, at least one image frame of a scene of a real environment to be annotated and providing the at least one image frame to a display device for presenting to a user;
receiving, by the at least one computing device, at least a part of a set of ranked objects of interest provided according to a method of claim 1;
presenting at least a portion of the received set of ranked objects of interest on the display device while displaying the at least one image frame to be annotated on the display device; and
receiving instructions from a human annotator instructing the at least one computing device to add or refine one or more of the object categories or segmentation masks to the at least one image frame and augmenting the at least one image frame with corresponding annotation information to at least one annotated dataset configured for input to and training at least one machine learning model.

### Claim 11 (depends on 10)

The method according to claim 10, wherein the at least one portion of the set of ranked objects of interest is presented on the display device after receiving instructions from the annotator instructing the at least one computing device to confirm or correct any stored audio-visual ground truth, wherein optionally the at least one portion of the set of ranked objects of interest presented on the display device comprises only objects of interest which are not included in the confirmed or corrected audio-visual ground truth.

### Claim 12 (depends on 8)

A computer-implemented method of annotating at least one image frame of a scene of a real environment configured for input to and training at least one machine learning model, the method comprising:
receiving, by at least one computing device, at least one image frame of a scene of a real environment to be annotated and providing the at least one image frame to a display device for presenting to a user;
receiving, by the at least one computing device, at least one dataset of an audio-visual ground truth generated according to a method of one of claim 8,
receiving instructions from a human annotator instructing the at least one computing device to add or refine one or more of the respective object categories and segmentation masks to the at least one image frame and augmenting the at least one image frame with corresponding annotation information to at least one annotated dataset configured for input to and training at least one machine learning model.

### Claim 13 (depends on 12)

The method according to claim 12, further comprising
presenting one or more segmentation masks of the set of segmentation masks on the display device in an overlapping manner with the at least one image frame,
receiving instructions from the annotator instructing the at least one computing device to confirm or correct the at least one dataset of an audio-visual ground truth by confirming or refining one or more of the respective object category and presented segmentation mask.

### Claim 14 (depends on 10)

The method according to claim 10, wherein the at least one image frame corresponds to the at least one image captured by the image sensor or, if the at least one image is a part of a video stream, corresponds to an image captured by the image sensor preceding or succeeding the at least one image captured by the image sensor.

### Claim 15

(canceled)
