Content delivery system and method for automated video overlay insertion
AssigneeTiny Pixels Technologies Inc.
InventorsJune Yan Lin Chen, Hsiehyu Fuh
Life of this patent
Estimating the term…
Abstract
There is provided a system and method for automated overlay insertion to a video. The method includes: associating each overlay with one or more overlay categories; detecting scene categories within each frame and a confidence probability value; adjusting the confidence probability values based on preceding frame groupings; matching overlay categories to comparable adjusted scene categories; generating a selection score for each overlay category that is relative to the probability value for the matched adjusted scene category; randomly selecting an overlay from the one or more overlays, the probability for each overlay based on the respective selection score; and inserting the selected overlay into the video.
Claims20
A method for automated overlay insertion to a video, the method comprising:
- receiving the video and one or more overlays, the video comprising a plurality of individual frames grouped in frame groupings;
- associating each overlay with one or more overlay categories;
- for each frame grouping in the plurality of frames, passing the visual contents of the frame to a trained convolutional neural network, the convolutional neural network is trained to detect scene categories within each frame and a confidence probability value associated with each detected scene category;
- successively for each of the frame groupings, adjusting the confidence probability values associated with the respective scene categories for a current frame grouping based on preceding frame groupings, by:
- determining a confidence probability for a scene category as a union of the confidence probability for such scene category for the current frame grouping and the confidence probability for such scene category for the preceding frame grouping if such scene category was detected in the preceding frame grouping and detected in the current frame grouping;
- and
- multiplying the probability value for each scene category by a predetermined scale down factor if such scene category was detected in the preceding frame grouping and not detected in the current frame grouping;
- matching overlay categories to comparable adjusted scene categories;
- generating a selection score for each overlay category that is relative to the probability value for the matched adjusted scene category;
- randomly selecting an overlay from the one or more overlays, the probability for each overlay based on the respective selection score;
- and
- inserting the selected overlay into the video.
A system for automated overlay insertion to a video, the system comprising one or more processors and a data storage, the data storage storing the video and one or more overlays, the video comprising a plurality of individual frames grouped in frame groupings, the one or more processors in communication with the one or more memory units and configured to execute:
- an overlay management module to associate each overlay with one or more overlay categories;
- a storyline generator module to, for each frame grouping in the plurality of frames, pass the visual contents of the frame to a trained convolutional neural network, the convolutional neural network is trained to detect scene categories within each frame and a confidence probability value associated with each detected scene category, and to successively for each of the frame groupings, adjust the confidence probability values associated with the respective scene categories for a current frame grouping based on preceding frame groupings, by:
- determining a confidence probability for a scene category as a union of the confidence probability for such scene category for the current frame grouping and the confidence probability for such scene category for the preceding frame grouping if such scene category was detected in the preceding frame grouping and detected in the current frame grouping;
- and
- multiplying the probability value for each scene category by a predetermined scale down factor if such scene category was detected in the preceding frame grouping and not detected in the current frame grouping;
- an overlay selector module to match overlay categories to comparable adjusted scene categories, to generate a selection score for each overlay category that is relative to the probability value for the matched adjusted scene category, and to randomly select an overlay from the one or more overlays, the probability for each overlay based on the respective selection score;
- and
- an overlay insertion module to insert the selected overlay into the video.
Dependencies are inferred from the claim text.