Repairing missing frames in recorded video with machine learning
AssigneeAmazon Technologies, Inc.
InventorKevin Harkness
Life of this patent
Estimating the term…
Abstract
Repairing missing frames in a video includes obtaining video data from an image capture system, applying a first neural network model to the video data to detect that one or more frames are missing, where the first neural network model has been trained to detect missing frames based on training data in which an artificial gap has been introduced. In response to detecting that the one or more frames are missing, a second model is applied to the video data to generate one or more replacement frames. The one or more replacement frames are based on at least a first frame prior to the detected dropped one or more frames, and a second frame after the detected dropped one or more frames. Modified video data is generated using the plurality of frames and the replacement frames.
Claims20
A method for repairing missing frames in a video, comprising:
- obtaining video data comprising a plurality of frames from an image capture system;
- applying a first neural network model to the video data to detect that one or more frames are missing, wherein the first neural network model has been trained to detect missing frames based on training data in which an artificial gap has been introduced;
- in response to detecting that the one or more frames are missing, applying a second model to the plurality of frames to generate one or more replacement frames, wherein the one or more replacement frames are based on at least a first frame of the plurality of frames prior to the one or more frames, and a second frame of the plurality of frames after the one or more frames;
- and
- generating modified video data using the plurality of frames and the one or more replacement frames.
A non-transitory computer readable medium comprising computer readable code for repairing missing frames in recorded video data, executable by one or more processors to:
- obtain video data comprising a plurality of frames from an image capture system;
- detect that one or more frames of the video data are missing;
- in response to detecting that the one or more frames are missing, generate one or more replacement frames based on a first frame of the plurality of frames prior to the one or more frames, and a second frame of the plurality of frames after the one or more frames;
- and
- generate modified video data using the plurality of frames and the one or more replacement frames.
A system for repairing missing frames in a video, comprising:
- a display device;
- a camera;
- one or more processors;
- and
- one or more computer readable media coupled to the one or more processors and comprising computer readable code which, when executed by the one or more processors, causes the one or more processors to:
- detect that the camera is capturing video data comprising a plurality of frames;
- in response to detecting that the camera is capturing additional video data, monitor one or more system resources for the camera;
- detect, based on the monitoring of the one or more system resources, that one or more parameters for the one or more system resources satisfy a predetermined condition by applying a prediction model to determine a probability that one or more frames are likely to be dropped based on a usage of the one or more system resources;
- and
- in response to determining that the one or more parameters for the one or more system resources satisfy the predetermined condition, presenting an indication that the video data is likely to be missing frames.
Dependencies are inferred from the claim text.