Automatic composition of composite images or video with stereo foreground objects
AssigneeGoPro, Inc.
InventorsBalineedu Chowdary Adsumilli, Ryan Lustig
Life of this patent
Estimating the term…
Abstract
A processing device generates composite images from a sequence of images. The composite images may be used as frames of video. A foreground/background segmentation is performed at selected frames to extract a plurality of foreground object images depicting a foreground object at different locations as it moves across a scene. The foreground object images are stored to a foreground object list. The foreground object images in the foreground object list are overlaid onto subsequent video frames that follow the respective frames from which they were extracted, thereby generating a composite video.
Claims20
A method for generating a composite output image from an input video having a sequence of frames, the method comprising:
- receiving a sequence of stereo video frames depicting a foreground object;
- for selected frames in the sequence of stereo video frames, performing by a processing device, foreground/background segmentations to extract respective stereo foreground object images each comprising a representation of the foreground object with background pixels subtracted;
- storing the respective stereo foreground object images to a foreground object list, each of the stereo foreground object images having left and right images with a disparity between them;
- transforming the stereo foreground object images to adjust the respective disparities between the respective left and right images based on a change between a convergence depth for the respective selected frames and a convergence depth for a current frame, the transforming of the stereo foreground object images comprising:
- decreasing the disparity for a given stereo foreground object image, in response to the convergence depth for the current frame being further away than a convergence depth for a selected frame corresponding to the given stereo foreground object;
- or
- increasing the disparity for the given stereo foreground object image, in response to the convergence depth for the current frame being closer than the convergence depth for the selected frame corresponding to the given stereo foreground object;
- and
- overlaying the transformed stereo foreground object images onto the current frame to generate a composite output image.
A non-transitory computer-readable storage medium configured to store instructions for generating a composite output image from an input video having a sequence of frames, the instructions being configured to cause, when executed by a processor apparatus, the processor apparatus to perform steps comprising:
- receiving a sequence of stereo video frames depicting a foreground object;
- for selected frames in the sequence of stereo video frames, performing foreground/background segmentations to extract respective stereo foreground object images each comprising a representation of the foreground object with background pixels subtracted;
- storing the respective stereo foreground object images to a foreground object list, each of the stereo foreground object images having left and right images with a disparity between them;
- transforming the stereo foreground object images to adjust the respective disparities between the respective left and right images based on a change between a convergence depth for the respective selected frames and a convergence depth for a current frame;
- and
- overlaying the transformed stereo foreground object images onto the current frame to generate a composite output image;
- wherein the transforming of the stereo foreground object images comprises:
- decreasing the disparity for a given stereo foreground object image, in response to the convergence depth for the current frame being further away than a convergence depth for a selected frame corresponding to the given stereo foreground object;
- and
- increasing the disparity for the given stereo foreground object image, in response to the convergence depth for the current frame being closer than the convergence depth for the selected frame corresponding to the given stereo foreground object.
An apparatus configured to generate a composite output image from an input video having a sequence of frames, the apparatus comprising:
- a processor apparatus;
- a non-transitory computer-readable storage medium having a computer application, the computer application comprising a plurality of instructions configured to, when executed by the processor apparatus, cause the apparatus to:
- receive a sequence of stereo video frames depicting a foreground object;
- for selected frames in the sequence of stereo video frames, perform foreground/background segmentations to extract respective stereo foreground object images each comprising a representation of the foreground object with background pixels subtracted;
- store the respective stereo foreground object images to a foreground object list, each of the stereo foreground object images having left and right images with a disparity between them;
- transform the stereo foreground object images to adjust the respective disparities between the respective left and right images based on a change between a convergence depth for the respective selected frames and a convergence depth for a current frame;
- and
- overlay the transformed stereo foreground object images onto the current frame to generate a composite output image;
- wherein, during the transformation of the stereo foreground object images, the plurality of instructions are further configured to, when executed by the processor apparatus, cause the apparatus to:
- decrease the disparity for a given stereo foreground object image, in response to the convergence depth for the current frame being further away than a convergence depth for a selected frame corresponding to the given stereo foreground object;
- and
- increase the disparity for the given stereo foreground object image, in response to the convergence depth for the current frame being closer than the convergence depth for the selected frame corresponding to the given stereo foreground object.
Dependencies are inferred from the claim text.