Gaze-driven recording of video
AssigneeApple Inc.
InventorsManeli Noorkami, Ranjit Desai
Life of this patent
Estimating the term…
Abstract
Systems and methods for gaze-driven recording of video are described. Some implementations may include accessing gaze data captured using one or more gaze-tracking sensors; applying a temporal filter to the gaze data to obtain a smoothed gaze estimate; determining a region of interest based on the smoothed gaze estimate, wherein the region of interest identifies a subset of a field of view; accessing a frame of video; recording a portion of the frame associated with the region of interest as an enhanced frame of video, wherein the portion of the frame corresponds to a smaller field of view than the frame; and storing, transmitting, or displaying the enhanced frame of video.
Claims20
A system comprising:
- one or more gaze-tracking sensors;
- one or more image sensors;
- and
- a processing apparatus configured to:
- access gaze data captured using the one or more gaze-tracking sensors;
- apply a temporal filter to the gaze data to obtain a smoothed gaze estimate;
- determine a region of interest based on the smoothed gaze estimate, wherein the region of interest identifies a subset of a field of view of the one or more image sensors;
- access a frame of video captured using the one or more image sensors, and wherein the processing apparatus is configured to determine the region of interest based on the smoothed gaze estimate by performing operations comprising applying computer vision processing to the frame of video to detect and classify an object appearing in the frame of video to obtain a location within the frame of video associated with the object, determining values of a cost function for the region of interest based on the smoothed gaze estimate and the location of the object, and determining the region of interest based on the values of the cost function;
- and
- apply signal processing to the frame of video based on the region of interest to obtain an enhanced frame of video.
A method comprising:
- accessing gaze data captured using one or more gaze-tracking sensors;
- applying a temporal filter to the gaze data to obtain a smoothed gaze estimate;
- determining a region of interest based on the smoothed gaze estimate, wherein the region of interest identifies a subset of a field of view;
- accessing a frame of video, and wherein determining the region of interest based on the smoothed gaze estimate comprises applying computer vision processing to the frame of video to detect and classify an object appearing in the frame of video to obtain a location within the frame of video associated with the object, determining values of a cost function for the region of interest based on the smoothed gaze estimate and the location of the object, and determining the region of interest based on the values of the cost function;
- applying signal processing to the frame of video based on the region of interest to obtain an enhanced frame of video;
- and
- storing, transmitting, or displaying the enhanced frame of video.
A non-transitory computer-readable storage medium comprising executable instructions that, when executed by a processor, cause performance of operations comprising:
- accessing gaze data captured using one or more gaze-tracking sensors;
- applying a temporal filter to the gaze data to obtain a smoothed gaze estimate;
- determining a region of interest based on the smoothed gaze estimate, wherein the region of interest identifies a subset of a field of view;
- accessing a frame of video;
- applying signal processing to the frame of video based on the region of interest to obtain an enhanced frame of video, wherein the executable instructions for applying signal processing to the frame of video based on the region of interest to obtain the enhanced frame of video include executable instructions that, when executed by a processor, cause performance of operations comprising applying computer vision processing to a portion of the frame associated with the region of interest to detect an object appearing in the portion of the frame and determine a classification of the object, wherein the portion of the frame corresponds to a smaller field of view than the frame, generating an annotation based on the classification;
- and superimposing the annotation on the frame of video to obtain the enhanced frame of video;
- and
- storing, transmitting, or displaying the enhanced frame of video.
Dependencies are inferred from the claim text.