Automatically detecting contents expressing emotions from a video and enriching an image index

A system for detecting contents expressing emotions from a video, the system comprising:

• at least one processor;
• a storage device comprising instructions, which when executed by the at least one processor, configure the processor to perform operations to:
• divide the video into a plurality of clips based on sub-shot detection and speech recognition, wherein start points and end points for the plurality of clips are identified using the sub-shot detection and the speech recognition, wherein the speech recognition generates a transcript for speech obtained from the plurality of clips;
• extract, from a first clip and at least one second clip of the plurality of clips, features associated with the first clip, wherein the first clip and the at least one second clip share a time boundary of the video;
• determine whether the first clip expresses emotions based on an evaluation of the features associated with the first clip by an emotion classifier;
• build an index containing the first clip based on the features associated with the first clip that express emotions in a searchable content index that includes a plurality of searchable clips indexed to corresponding emotions;
• receive a search query;
• and
• return the first clip based on a match between an emotion indicated by the search query and an emotion corresponding to the first clip as indicated in the searchable context index.

A non-transitory computer readable medium comprising instructions for detecting contents expressing emotions from a video, which when executed by at least one processor, configure the processor to perform operations comprising:

• dividing the video into a plurality of clips based on sub-shot detection and speech recognition, wherein start points and end points for the plurality of clips are identified using the sub-shot detection and the speech recognition, wherein the speech recognition generates a transcript for speech obtained from the plurality of clips;
• extracting, from a first clip and at least one second clip of the plurality of clips, features associated with the first clip, wherein the first clip and the at least one second clip share a time boundary of the video;
• determining whether the first clip expresses emotions based on an evaluation of the features associated with the first clip by an emotion classifier;
• building an index containing the first clip based on the features associated with the first clip that express emotions in a searchable content index that includes a plurality of searchable clips indexed to corresponding emotions;
• receiving a search query;
• and
• returning the first clip based on a match between an emotion indicated by the search query and an emotion corresponding to the first clip as indicated in the searchable context index.