# US 9,436,875 B2 · Method and apparatus for semantic extraction and video remix creation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,436,875 B2 - Document ID: US9436875B2 - Status: Granted - Kind: B2 - Application number: 13/707,090 - Filed: 2012-12-06 - Published: 2014-06-12 - Granted: 2016-09-06 - Earliest claimed: 2012-12-06 - Assignees: Nokia Technologies Oy - Applicants: Nokia Technologies Oy - Inventors: Igor Danilo Diego Curcio; Sujeet Shyamsundar Mate; Francesco Cricri; Mikko Joonas Roininen; Sailesh Sathish - Web page: https://patentgrep.com/patent/US9436875B2 ## CPC - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06F 18/253: {of extracted features} - G06V 10/806: {of extracted features} - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method, apparatus and computer program product are provided for extracting semantic information from user-generated media content to create a video remix which is semantically enriched. An exemplary method comprises extracting media content data and sensor data from a plurality of media content, wherein the sensor data comprises a plurality of data modalities. The method may also include classifying the extracted media content data and the sensor data. The method may further include detecting predefined objects or events utilizing the sensor data to create remix video. ## Claims ### Claim 1 A method comprising: classifying recorded data according to an event type, wherein the recorded data comprises a plurality of modalities; determining an event template configuration, the event template configuration comprises one or more selected event templates, wherein determination of the event template configuration comprises selecting each event template based on the event type and a target audience, wherein each event template comprises a set of event-specific image templates; detecting one or more predefined objects or one or more predefined events in the recorded data in accordance with each event template; and creating, with a processor, a remix video utilizing the event template and the one or more predefined objects or the one or more predefined events, wherein each event template, selected based on the event type and the target audience, identifies which one or more predefined objects of a plurality of potential objects or one or more predefined events of a plurality of potential events are to be detected in the recorded data, and wherein the event template configuration identifies a weighting of each event template's inclusion in the remix video. ### Claim 2 (depends on 1) A method of claim 1, wherein the classifying of recorded data comprises: estimating a quality level of each input data modality; extracting one or more features from one or more of the plurality of data modalities; classifying the one or more extracted features; performing relation analysis for discovering correlative relations, anti-correlative relations, or a combination thereof between the one or more extracted features; and performing fusion of feature classification using relation analysis results, quality levels of the input modalities, and location information to determine the event type. ### Claim 3 (depends on 1) A method of claim 1 wherein each event template comprises information related to: assigning roles to one or more recording devices, selecting a sub-set of editing rules; selecting a sub-set of potential salient events; and selecting a sub-set of potential salient objects. ### Claim 4 (depends on 1) A method of claim 1 further comprising: assigning roles for one or more recording devices, according to the event type template, to record one or more potential salient events or one or more potential salient objects, wherein the assigning of roles is a function of one or more of a number of recording devices, a capability of the recording devices, preference information of a person associated with the event, preference information of a person associated with the remix video, or a target audience. ### Claim 5 (depends on 1) A method of claim 1 wherein detecting a predefined object in the recorded data comprises: selecting a sub-set of image templates; and performing template matching, wherein selecting of the sub-set of image templates further comprises matching the auxiliary sensor data from the recorded data with auxiliary data associated with available templates. ### Claim 6 (depends on 1) A method of according to claim 1, wherein detecting a predefined object in the recorded data comprises analyzing auxiliary sensor data captured in the recorded data; and utilizing an offline phase and an online phase, wherein the offline phase comprises collecting a set of multimodal templates, and causing image templates and associated metadata information to be stored, wherein the associated metadata information is auxiliary sensor data, and wherein the online phase comprises template matching on the recorded data with a sub-set of the multimodal templates. ### Claim 7 (depends on 6) A method of claim 6, wherein template matching further comprises: extracting features from the sub-set of image templates; extracting features from the recorded data; and matching the extracted features. ### Claim 8 An apparatus comprising: a processor and a memory including software, the memory and the software configured to, with the processor, cause the apparatus to at least: classify recorded data according to an event type, wherein the recorded data comprises a plurality of modalities; determine an event template configuration, the event template configuration comprises one or more selected event templates, wherein determination of the event template configuration comprises software configured to, with the processor, cause the apparatus to at least select each event template based on the event type and a target audience, wherein each event template comprises a set of event-specific image templates; detect one or more predefined objects or one or more predefined events in the recorded data in accordance with the event template; and create a remix video utilizing each event template and the one or more predefined objects or the one or more predefined events, wherein each event template, selected based on the event type and the target audience, identifies which one or more predefined objects of a plurality of potential objects or one or more predefined events of a plurality of potential events are to be detected in the recorded data, and wherein the event template configuration identifies a weighting of each event template's inclusion in the remix video. ### Claim 9 (depends on 8) An apparatus of claim 8 wherein the at least one memory including the computer program code is further configured to, with the at least one processor, cause the apparatus to: extract one or more features from one or more of the plurality of data modalities; estimate a quality level of each input data modality; classify the one or more extracted features; perform relation analysis for discovering correlative relations, anti-correlative relations, or a combination thereof between the one or more extracted features; and perform fusion of feature classification using relation analysis results, quality level of each modality and location information to determine the event type. ### Claim 10 (depends on 8) An apparatus of claim 8 wherein each event template comprises information related to: assigning roles to one or more recording devices, selecting a sub-set of editing rules; selecting a sub-set of potential salient events; and selecting a sub-set of potential salient objects. ### Claim 11 (depends on 8) An apparatus of claim 8 wherein the at least one memory including the computer program code is further configured to, with the at least one processor, cause the apparatus to: assign roles for one or more recording devices, according to the event type template, to record one or more potential salient events or one or more potential salient objects, wherein the at least one memory including the computer program code is further configured to, with the at least one processor, cause the apparatus to: assign roles as a function of one or more of a number of recording devices, a capability of the recording devices, preference information of a person associated with the event, preference information of a person associated with the remix video, or a target audience. ### Claim 12 (depends on 8) An apparatus of claim 8, wherein the at least one memory including the computer program code is further configured to, with the at least one processor, cause the apparatus to select the sub-set of image templates by matching the auxiliary sensor data from the recorded data with auxiliary data associated to available templates. ### Claim 13 (depends on 8) An apparatus of claim 8, wherein the at least one memory including the computer program code is further configured to, with the at least one processor, cause the apparatus to detect a predefined object in the recorded data by: analyzing auxiliary sensor data captured in the recorded data; utilizing an offline phase and an online phase, wherein the offline phase comprises collecting a set of multimodal templates, and storing image templates and associated metadata information, wherein the associated metadata information is auxiliary sensor data, and wherein the online phase comprises template matching on the recorded data with a sub-set of the multimodal templates. ### Claim 14 (depends on 13) An apparatus of claim 13, wherein the at least one memory including the computer program code is further configured to, with the at least one processor, cause the apparatus to perform template matching by: extracting features from the sub-set of image templates; extracting features from the recorded data; and matching the extracted features. ### Claim 15 A computer program product comprising at least one non-transitory computer-readable storage medium having computer-readable program instructions stored therein, the computer-readable program instructions include program instructions configured, when executed by a computer, to: classify recorded data according to an event type, wherein the recorded data may comprise a plurality of modalities; determine an event template configuration, the event template configuration comprises one or more selected event templates, wherein determination of the event template configuration comprises selecting each event template based on the event type and a target audience, wherein each event template comprises a set of event-specific image templates; detect one or more predefined objects or one or more predefined events in the recorded data in accordance with each event template; and create a remix video utilizing each event template and the one or more predefined objects or the one or more predefined events, wherein each event template, selected based on the event type and the target audience, identifies which one or more predefined objects of a plurality of potential objects or one or more predefined events of a plurality of potential events are to be detected in the recorded data, and wherein the event template configuration identifies a weighting of each event template's inclusion in the remix video. ### Claim 16 (depends on 15) A computer program product of claim 15, wherein the program instructions configured to classify recorded data also include program instructions configured, when executed by a computer, to: estimate a quality level of each input data modality; extract one or more features from one or more of the plurality of data modalities; classify the one or more extracted features; perform relation analysis for discovering correlative relations, anti-correlative relations, or a combination thereof between the one or more extracted features; and perform fusion of feature classification using relation analysis results, quality level of each input data modality and location information to determine the event type. ### Claim 17 (depends on 15) A computer program product of claim 15 wherein each event template comprises information related to: assigning roles to one or more recording devices, selecting a sub-set of editing rules; selecting a sub-set of potential salient events; and selecting a sub-set of potential salient objects. ### Claim 18 (depends on 15) A computer program product of claim 15 wherein the program instructions also include program instructions configured, when executed by a computer, to: assign roles for one or more recording devices, according to the event type template, to record one or more potential salient events or one or more potential salient objects, wherein the assigning of roles is a function of one or more of a number of recording devices, a capability of the recording devices, preference information of a person associated with the event, preference information of a person associated with the remix video, or a target audience. ### Claim 19 (depends on 15) A computer program product of claim 15 wherein the computer-readable program instructions for detecting the predefined object also include program instructions configured, when executed by a computer, to: select a sub-set of image templates; and perform template matching, wherein selecting of the sub-set of image templates further comprises matching the auxiliary sensor data from the recorded data with auxiliary data associated with available templates. ### Claim 20 (depends on 15) A computer program product of according to claim 15, wherein the computer-readable program instructions configured to detect a predefined object in the recorded data also include program instructions configured, when executed by a computer, to analyze auxiliary sensor data captured in the recorded data; and utilize an offline phase and an online phase, wherein the offline phase comprises collecting a set of multimodal templates, and causing image templates and associated metadata information to be stored, wherein the associated metadata information is auxiliary sensor data, and wherein the online phase comprises template matching on the recorded data with a sub-set of the multimodal templates. ### Claim 21 (depends on 20) A computer program product of claim 20, wherein the computer-readable program instructions configured for template matching also include program instructions configured, when executed by a computer, to: extract features from the sub-set of image templates; extract features from the recorded data; and match the extracted features.