Using eye tracking to label computer vision datasets
AssigneeAmazon Technologies, Inc.
InventorsBenjamin Jack Barash, Yves Albers Schoenberg
Life of this patent
Estimating the term…
Abstract
A system is configured to label computer vision datasets using eye tracking of users that track objects depicted in imagery to label the datasets. The imagery may include moving images (e.g., video) or still images. By using eye tracking, users may be able to label large amounts of imagery more efficiently than by manually labeling datasets using conventional input devices. A user may be instructed to watch a particular object during a playback of the video while an imaging device determines a direction of the user's gaze which correlates with a location in the imagery. An application may then associate the location in the imagery determined from the user's gaze as a location of the object on a frame-by-frame basis, or for certain frames.
Claims20
A system comprising:
- one or more processors;
- and
- memory to store computer:
- implemented instructions that, when executed, cause the one or more processors to perform acts to:
- select imagery that is associated with a number of labels that is less than a labeling threshold to present to a user for labeling using eye tracking of eyes of the user;
- prompt the user to stare at an object in the imagery;
- prompt the user to identify a label of the object;
- receive, as user input, the label of the object;
- receive eye tracking data that indicates a direction of a gaze of the user staring at the object in the imagery;
- determine a pixel location associated with the object based at least in part on the eye tracking data;
- determining a bounding shape of the object based at least in part on the pixel location;
- perform object detection on the object;
- compare an output of the object detection with the bounding shape of the object to validate the pixel location;
- and
- storing the label of the object and the bounding shape in association with the imagery.
A computer-implemented method to label an object depicted in imagery, the method comprising:
- determining imagery that is associated with a number of labels that is less than a labeling threshold;
- causing playback of the imagery for viewing by a user;
- instructing the user to stare at the object during the playback of the imagery;
- receiving eye tracking data that provides a direction of a gaze of the user;
- associating the gaze of the user with a pixel location in the imagery;
- receiving user input data associated with a label of the object;
- associating the pixel location with the label of the object;
- determining a bounding shape of the object based at least in part on the pixel location;
- performing object detection on the object;
- and
- comparing an output of the object detection with the bounding shape of the object to validate the pixel location.
A system comprising:
- one or more processors;
- and
- memory to store computer-implemented instructions that, when executed, cause the one or more processors to perform acts to:
- determine imagery that is associated with a number of labels that is less than a labeling threshold;
- cause the imagery to be presented for labeling using eye tracking;
- prompt a user to stare at an object in the imagery;
- receive eye tracking data that indicates a pixel location associated with the object;
- receive user input data associated with a label of the object;
- associate the pixel location of the object with the label of the object;
- determine a bounding shape of the object based at least in part on the pixel location;
- perform object detection on the object;
- and
- compare an output of the object detection with the bounding shape of the object to validate the pixel location.
Dependencies are inferred from the claim text.