Precise object segmentation with multi-modal input for realtime video application
AssigneeBlack Sesame Technologies Inc.
InventorsFangwen Tu, Bo Li
Life of this patent
Estimating the term…
Abstract
The present invention discloses a system for precise representation of object segmentation with multi-modal input for real-time video applications. The multi-modal segmentation system takes advantage of optical, temporal as well as spatial information to enhance the segmentation for AR and VR or other entrainment purpose with accurate details. The system can segment foreground objects such as human and salient objects within a video frame and allows locating object-of-interest for multiple-purposes.
Claims15
A multi-modal system for segmenting a multimedia, wherein the multi-modal system comprising:
- an electronic device for recording of a video multimedia;
- at least one sensor for detecting motion of the electronic device;
- a processor including;
- a refinement unit configured to concatenate one or More additional information to the multimedia to generate an optimized multimedia;
- a notification unit configured to send a notification for capturing a background region of the video multimedia;
- a shooting unit configured to shoot the multimedia with respect to the background region;
- a capturing unit configured to capture the background region followed by the synthesis of the background region to generate a multimedia output;
- and
- a masking unit configured to produce one or more masks based on the background region;
- and
- wherein the refinement unit
- updates the background region depending on the one or more masks;
- extras one or More features associated with the one Or more mask;
- selects one or more dominant features from the one or more features;
- and
- combines the one or more dominant features to generate a segmented multimedia.
A method for segmenting a multimedia using a multi-modal system, wherein the method comprising:
- initiating a recording of the multimedia by an electronic device;
- detecting motion of the electronic device;
- refining the multimedia by concatenating one or more additional information to the multimedia to generate an optimized multimedia;
- sending a notification for capturing a background region of the multimedia;
- shooting the multimedia with respect to the background region;
- capturing the background region followed by, the synthesis of the background region to generate a multimedia output,
- producing one Or more masks based on the background region;
- updating the background region depending on the one or more masks;
- extracting one or more features associated with the one or more masks;
- selecting one or more dominant features from the one or more features;
- and
- combining the one or more dominant features to produce a segmented multimedia.
A non-transitory computer program product comprising a non-transitory computer useable medium having computer program logic for enabling at least one processor in a computer system for segmenting a multimedia via a software platform, said computer program logic comprising:
- initiating a recording of the multimedia by an electronic device,
- detecting motion of the electronic device;
- refining the multimedia by concatenating one or more additional information to the multimedia to generate an optimized multimedia;
- sending a notification fix capturing a background region of the video multimedia;
- shooting the multimedia with respect to the background region;
- capturing the background region followed by the synthesis of the background region to generate a multimedia output,
- producing one or more masks based on the background region;
- updating the background wound region depending on the one or more masks;
- extracting one or more features associated with the one or more masks;
- selecting one or more dominant features from the one or more features;
- and
- combining the one or more dominant features to produce a segmented multimedia.
Dependencies are inferred from the claim text.