Method and system to generate targeted captions and summarize long, continuous media files
AssigneeFuji Xerox Co., Ltd.
InventorsBor-Chun Chen, Yin-Ying Chen, Francine Chen
Life of this patent
Estimating the term…
Abstract
A method of generating a summary of a media file that comprises a plurality of media segments is provided. The method includes calculating, by a neural network, respective importance scores for each of the media segments, based on content features associated with each of the media segments and a targeting approach, selecting a media segment from the media segments, based on the calculated importance scores, generating a caption for the selected media segment based on the content features associated with the selected media segment, and generating a summary of the media file based on the caption.
Claims22
A method of generating a summary of a media file containing at least a portion representative of a triggering event, the media file comprises a plurality of temporally separated media segments, the method comprising:
- calculating, by a neural network, respective importance scores for each of the temporally separated media segments, based on content features associated with each of the media segments and a targeting approach, the importance scores representative of the media segments containing content features indicative of the triggering event, wherein the triggering event in the media file is detected based on a dynamically selected targeting approach, which defines content features and changes in content features to be detected in the media file to identify the trigger event;;
- selecting a first media segment from the temporally separated media segments, based on the calculated importance scores indicative of the triggering event;
- generating a first caption for the selected first media segment based on the content features associated with the selected media segment;
- and
- generating a summary of the media file based on the first caption.
A non-transitory computer readable medium having stored therein a program for making a computer execute a method of generating a summary of a media file containing at least a portion representative of a triggering event, the media file comprises a plurality of temporally separated media segments, the method comprising:
- calculating, by a neural network, respective importance scores for each of the temporally separated media segments, based on content features associated with each of the media segments and a targeting approach, the importance scores representative of the media segments containing content features indicative of the triggering event, wherein the triggering event in the media file is detected based on a dynamically selected targeting approach, which defines content features and changes in content features to be detected in the media file to identify the trigger event;
- selecting a first media segment from the temporally separated media segments, based on the calculated importance scores indicative of the triggering event;
- generating a first caption for the selected first media segment based on the content features associated with the selected media segment;
- selecting a second media segment from the temporarily separated media segments, based on the calculated importance scores indicative of the triggering event
- generating a second caption for the selected second media segment based on the content features associated with the selected media segment;
- and
- generating a summary of the media file based on the first and the second caption.
A media analysis device comprising:
- a communications bus configured to receive a media file containing at least a portion representative of a triggering event, the media file comprises a plurality of temporally separated media segments;
- a memory configured to store the media file;
- a processor executing a process comprising:
- calculating, by a neural network, respective importance scores for each of the temporally separated media segments, based on content features associated with each of the media segment and a targeting approach, the importance scores representative of the media segments containing content features indicative of the triggering event, wherein the triggering event in the media file is detected based on a dynamically selected targeting approach, which defines content features and changes in content features to be detected in the media file to identify the trigger event;
- selecting a first media segment from the temporally separated media segments, based on the calculated importance scores indicative of the triggering event;
- generating a first caption for the selected first media segment based on the content features associated with the selected media segment
- selecting a second media segment from the temporarily separated media segments, based on the calculated importance scores indicative of the triggering event;
- generating a second caption for the selected second media segment based on the content features associated with the selected media segment;
- and
- generating a summary of the media file based on the first and the second caption.
Dependencies are inferred from the claim text.