Multimedia computer system with story segmentation capability and operating program therefor including finite automation video parser

A combination receiving a multimedia signal including a video shot sequence, an audio signal and text information for parsing the multimedia signal into one of a plurality of story program categories, each of the program categories having an associated finite automaton (FA) model and keywords, at least one of which keywords being associated with a respective node of the FA model, comprising:

• first means for analyzing the video portion of the received multimedia signal to identify keyframes therein to thereby generate identified keyframes;
• second means for comparing said identified keyframes within the video shot sequence with predetermined FA characteristics to identify a pattern of appearance within the video shot sequence;
• third means constructing a finite automaton (FA) model describing the appearance of the video shot sequence to thereby generate a constructed FA model;
• fourth means for coupling neighboring video shots or similar shots with said identified keyframes when said neighboring video shots are apparently related to a story represented by said identified keyframes;
• fifth means for extracting said keywords from said text information and storing said keywords at locations associated with each node of said constructed FA model;
• sixth means for analyzing and segmenting the audio signal in the multimedia signal into identified speaker segments, music segments, and silent segments
• seventh means for attaching said identified speaker segments, music segments, and silent segments to said constructed FA model;
• eighth means for storing the identity of said constructed FA model as said final FA model along with said keywords when said constructed FA model matches a previously defined FA model;
• and
• ninth means for generating a new FA model corresponding to said constructed FA model, for storing said new FA model, and for storing the identity of said new FA model as said final FA model along with said keywords when said constructed FA model does not match a previously defined FA model.

A video story parsing method employed in the operation of a multimedia computer system receiving a multimedia signal including a video shot sequence, an associated audio signal and corresponding text information to permit a multimedia signal parsed into a predetermined category having an associated finite automaton (FA) model and keywords, at least one of the keywords being associated with a respective node of the FA model to be parsed into a number of discrete video stories, the method comprising steps for:

• (a) extracting a plurality of keywords from an input first sentence;
• (b) categorizing said first sentence into one of a plurality of categories;
• (c) determining whether a current video shot belongs to a previous category, a current category or a new category of said plurality of categories responsive to similarity between said first sentence and an immediately preceding sentence;
• (d) repeating steps (a) through (c) until all video clips and respective sentences are assigned to one of said categories.

The video story parsing method as recited in 12, wherein said step (b) comprises:

• (b) categorizing said first sentence into one of a plurality of categories by determining a measure Mki of the similarity between the keywords extracted during step (a) and a keyword set for an ith story category Ci according to the expression set:
• if Memi≢0, M k i = ( MK Nkeywords + Mem i ) / 2
• if Memi=0, M k i = MK Nkeywords
• where MK denotes a number of matched words out of a total number Nkeywords of keywords in the respective keyword set for a characteristic sentence in said category Ci, where Memi is indicative of a measure of similarity with respect to the previous sentence sequence within category Ci and wherein 0≦Mki<1.

A method for operating a multimedia computer system receiving a multimedia signal including a video shot sequence, an associated audio signal and corresponding text information to thereby generate a video story database including a plurality of discrete stories searchable by one of finite automaton (FA) model having associated keywords, at least one of which keywords is associated with a respective node of the FA model, and user selected similarity criteria, the method comprising steps for:

• (a) analyzing the video portion of the received multimedia signal to identify keyframes therein to thereby generate identified keyframes;
• (b) comparing said identified keyframes within the video shot sequence with predetermined FA characteristics to identify a pattern of appearance within the video shot sequence;
• (c) constructing a finite automaton (FA) model describing the appearance of the video shot sequence to thereby generate a constructed FA model;
• (d) coupling neighboring video shots or similar shots with said identified keyframes when said neighboring video shots are apparently related to a story represented by said identified keyframes;
• (e) extracting said keywords from said text information and storing said keywords at locations associated with each node of said constructed FA model;
• (f) analyzing and segmenting the audio signal of the multimedia signal into identified speaker segments, music segments, laughter segments, and silent segments
• (g) attaching said identified speaker segments, music segments, laughter segments, and silent segments to said constructed FA model;
• (h) when said constructed FA model matches a previously defined FA model, storing the identity of said constructed FA model as said final FA model along with said keywords;
• (i) when said constructed FA model does not match a previously defined FA model, generating a new FA model corresponding to said constructed FA model, storing said new FA model, and storing the identity of said new FA model as said final FA model along with said keywords;
• (j) when said final FA model corresponds to a predetermined program category, performing video story segmentation according to the substeps of:
• (j)(i) extracting a plurality of keywords from an input first sentence;
• (j)(ii) categorizing said first sentence into one of a plurality of video story categories;
• (j)(iii) determining whether a current video shot belongs to a previous video story category, a current video story category or a new video story category of said plurality of video story categories responsive to similarity between said first sentence and an immediately preceding sentence;
• and
• (j)(iv) repeating steps (j)(i) through (j)(iii) until all video clips and respective sentences are assigned to one of said video story categories.

The method as recited in 14, wherein said substep (j)(ii) further comprises:

• (j)(ii) categorizing said first sentence into one of a plurality of sentence categories by determining a measure Mki of the similarity between the keywords extracted during step (k)(i) and a keyword set for an ith video story category Ci according to the expression set:
• if Memi≢0, M k i = ( MK Nkeywords + Mem i ) / 2
• if Memi=0, M k i = MK Nkeywords
• where MK denotes a number of matched words out of a total number Nkeywords of keywords in the respective keyword set for a characteristic sentence in said category Ci, where Memi is indicative of a measure of similarity with respect to the previous sentence sequence within category Ci and wherein 0≦Mki<1.