Natural language processing based dominant item detection in videos
AssigneeMicrosift Technology Licensing, LLC
InventorsTom Hirshberg, Yonit Hoffman, Zvi Figov, Maayan Yedidia
Life of this patent
Estimating the term…
Abstract
Systems and method are provided for extracting and processing terms referenced in multimedia content with the use of different term extraction models to determine the relevance of categories to the referenced terms and to rank the categories by relative dominance for the multimedia content. The most dominant category for the multimedia content and/or particular segment(s) of the multimedia content can then be identified and used to link to supplemental content and/or to identify multimedia content related to topics of interest.
Claims20
A method implemented by a computer system for processing multimedia content comprising a plurality of temporally sequential segments, the method comprising:
- accessing the multimedia content;
- identifying the plurality of temporally sequential segments of the multimedia content;
- identifying terms that are represented within each segment of the plurality of temporally sequential segments based on applying a set of term extraction models to each segment;
- applying embeddings to the terms that are identified;
- categorizing each term, respectively, to a relevant category from a plurality of categories based on the embeddings applied;
- and
- identifying a most dominant category of the plurality of categories for the multimedia content based on ranking the plurality of categories for the multimedia content and by at least calculating a dominance feature relevance of terms that were identified within each segment of the plurality of temporally sequential segments of the multimedia content and that were categorized within each category of the plurality of categories.
A method implemented by a computer system for processing multimedia content, comprising:
- accessing the multimedia content;
- identifying a plurality of segments of the multimedia content;
- identifying terms that are represented within a first segment of the plurality of segments based on applying a set of term extraction models to the first segment;
- identifying terms that are represented within a second segment of the plurality of segments based on applying the set of term extraction models to the second segment;
- applying embeddings to the terms that are identified;
- determining a relative relevance of a plurality of different categories to the first segment of the plurality of segments based on the terms that are represented within the first segment;
- determining a relative relevance of a plurality of different categories to the second segment of the plurality of segments based on the terms that are represented within the second segment;
- identifying a most dominant category of the plurality of different categories for the first segment of the plurality of segments of the multimedia content based on the relative relevance of the terms in the first segment;
- and
- identifying a most dominant category of the plurality of different categories for the second segment of the plurality of segments of the multimedia content based on the relative relevance of the terms in the second segment, wherein the most dominant category of the plurality of different categories for the second segment is a different dominant category than the most dominant category of the plurality of different categories for the first segment.
A method implemented by a computer system for processing multimedia content comprising a plurality of temporally sequential segments, the method comprising:
- accessing the multimedia content;
- identifying the plurality of temporally sequential segments of the multimedia content;
- identifying terms that are represented within each segment of the plurality of temporally sequential segments based on applying a set of term extraction models to each segment;
- applying embeddings to the terms that are identified;
- for each segment of the plurality of temporally sequential segments, determine a relative relevance of the terms to a plurality of categories;
- and
- modifying the multimedia content to include supplemental content corresponding to a particular category of the plurality of categories for a segment of the plurality of temporally sequential segments in which the relative relevance for a category included in the plurality of categories meets a particular relevance threshold.
Dependencies are inferred from the claim text.