# US 10,311,913 B1 · Summarizing video content based on memorability of the video content > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,913 B1 - Document ID: US10311913B1 - Status: Granted - Kind: B1 - Application number: 15/902,046 - Filed: 2018-02-22 - Granted: 2019-06-04 - Earliest claimed: 2018-02-22 - Assignees: Adobe Inc. - Applicants: Adobe Inc. - Inventors: Sumit Shekhar; Harvineet Singh; Dhruv Singal; Atanu R. Sinha - Web page: https://patentgrep.com/patent/US10311913B1 ## CPC - G11B 27/28: by using information signals recorded by the same method as the main recording {(G11B27/22 takes precedence)} - G06F 18/2113: by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/47: {Detecting features for summarising video content} - G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes} - G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Certain embodiments involve generating summarized versions of video content based on memorability of the video content. For example, a video summarization system accesses segments of an input video. The video summarization system identifies memorability scores for the respective segments. The video summarization system selects a subset of segments from the segments based on each computed memorability score in the subset having a threshold memorability score. The video summarization system generates visual summary content from the subset of the segments. ## Claims ### Claim 1 A method for summarizing video content based on memorability of the video content, the method performed by one or more processing devices and comprising: accessing segments of an input video; computing memorability scores for the segments, respectively, wherein computing a memorability score for a segment comprises: generating (i) a semantic feature computed from an auto-captioning operation applied to the segment and (ii) a visual feature computed from one or more of a saliency analysis operation applied to the segment, a color analysis operation applied to the segment, and a spatio-temporal analysis operation applied to the segment, computing a first component score by applying a first predictor to the semantic feature, where the first predictor is trained to determine first component memorability scores by comparing user-generated memorability values with training semantic features generated by the auto-captioning operation, computing a second component score by applying a second predictor to the semantic feature, where the second predictor is trained to determine second component memorability scores by comparing the user-generated memorability values with training visual features generated by the one or more of the saliency analysis operation, the color analysis operation, and the spatio-temporal analysis operation, and computing the memorability score from an averaging operation applied to the first component score and the second component score; selecting a subset of segments from the segments based on each computed memorability score in the subset having a threshold memorability score; and generating visual summary content from the subset of the segments. ### Claim 2 (depends on 1) The method of claim 1, further comprising partitioning the input video into the segments prior to accessing the input video. ### Claim 3 (depends on 1) The method of claim 1, wherein selecting the subset of the segments based on each computed memorability score in the subset having the threshold memorability score comprises: ranking the segments according to the computed memorability scores, wherein the ranking applies a first rank to a first segment and a second rank to a second segment; determining that the first rank is greater than the second rank; including the first segment in the subset of the segments; and excluding the second segment from the subset of the segments. ### Claim 4 (depends on 1) The method of claim 1, wherein generating the visual summary content comprises: identifying a summary length for the visual summary content; selecting, from the subset of segments, a summary subset of segments having a combined length that is less than or equal to the summary length, wherein the summary subset includes a smaller number of segments than the subset of segments. ### Claim 5 (depends on 4) The method of claim 4, wherein selecting the summary subset comprises: determining that the summary subset (i) is less than or equal to the summary length and (ii) maximizes a sum of criteria scores for respective segments in the summary subset, wherein a criteria score comprises a memorability score for a segment weighted by a memorability weight and an additional video metric weighted by an additional video metric weight, wherein the additional video metric comprises one or more of video uniformity and video representativeness; and selecting the summary subset based on determining that the summary subset maximizes the sum of criteria scores and is less than or equal to the summary length. ### Claim 6 (depends on 1) The method of claim 1, wherein generating the visual summary content comprises combining the subset of the segments into a preview video that is included in the visual summary content. ### Claim 7 (depends on 1) The method of claim 1, wherein the visual summary content comprises a set of thumbnail images, wherein generating the set of thumbnail images comprises: computing visual quality scores for the subset of the segments; extracting thumbnail images from the subset of the segments based on the visual quality scores; and selecting the extracted thumbnail images as the set of thumbnail images. ### Claim 8 A system comprising: a processing device; and a non-transitory computer-readable medium communicatively coupled to the processing device, wherein the processing device is configured to execute program stored in the non-transitory computer-readable medium and thereby perform operations comprising: identifying a summary length for a visual summary content to be generated using input video segments; determining that a summary subset of the input video segments (i) has a combined length that is less than or equal to the summary length and (ii) maximizes a sum of criteria scores for respective segments in the summary subset, wherein at least one criteria score comprises a memorability score for an input video segment weighted by a memorability weight and an additional video metric weighted by an additional video metric weight, wherein the additional video metric comprises one or more of video uniformity and video representativeness, selecting the summary-a subset of the input video segments based on determining that the summary subset maximizes the sum of criteria scores and is less than or equal to the summary length, and generating the visual summary content from the summary subset of the input video segments. ### Claim 9 (depends on 8) The system of claim 8, the operations further comprising selecting the summary subset of the input video segments based on each memorability score in the summary subset having a threshold memorability score. ### Claim 10 (depends on 9) The system of claim 9, wherein selecting the summary subset of the input video segments based on each memorability score in the summary subset having the threshold memorability score comprises: ranking the input video segments according to the memorability scores, wherein the ranking applies a first rank to a first segment and a second rank to a second segment; determining that the first rank is greater than the second rank; including the first segment in the summary subset of the input video segments; and excluding the second segment from the summary subset of the input video segments. ### Claim 11 (depends on 8) The system of claim 8, wherein generating the visual summary content comprises combining the summary subset of the input video segments into a preview video that is included in the visual summary content. ### Claim 12 A non-transitory computer-readable medium having program code that is stored thereon, the program code executable by one or more processing devices for performing operations comprising: accessing segments of an input video; computing memorability scores for the segments, respectively, wherein computing a memorability score for a segment comprises: generating (i) a semantic feature computed from an auto-captioning operation applied to the segment and (ii) a visual feature computed from one or more of a saliency analysis operation applied to the segment, a color analysis operation applied to the segment, and a spatio-temporal analysis operation applied to the segment, computing a first component score by applying a first predictor to the semantic feature, where the first predictor is trained to determine first component memorability scores by comparing user-generated memorability values with training semantic features generated by the auto-captioning operation, computing a second component score by applying a second predictor to the semantic feature, where the second predictor is trained to determine second component memorability scores by comparing the user-generated memorability values with training visual features generated by the one or more of the saliency analysis operation, the color analysis operation, and the spatio-temporal analysis operation, and computing the memorability score from an averaging operation applied to the first component score and the second component score; a step for selecting a subset of segments from the segments based on each computed memorability score in the subset having a threshold memorability score; and generating visual summary content from the subset of the segments. ### Claim 13 (depends on 12) The non-transitory computer-readable medium of claim 12, the operations further comprising partitioning the input video into the segments prior to accessing the input video. ### Claim 14 (depends on 12) The non-transitory computer-readable medium of claim 12, wherein selecting the subset of the segments based on each computed memorability score in the subset having the threshold memorability score comprises: ranking the segments according to the computed memorability scores, wherein the ranking applies a first rank to a first segment and a second rank to a second segment; determining that the first rank is greater than the second rank; including the first segment in the subset of the segments; and excluding the second segment from the subset of the segments. ### Claim 15 (depends on 12) The non-transitory computer-readable medium of claim 12, wherein generating the visual summary content comprises: identifying a summary length for the visual summary content; selecting, from the subset of segments, a summary subset of segments having a combined length that is less than or equal to the summary length, wherein the summary subset includes a smaller number of segments than the subset of segments. ### Claim 16 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein selecting the summary subset comprises: determining that the summary subset (i) is less than or equal to the summary length and (ii) maximizes a sum of criteria scores for respective segments in the summary subset, wherein a criteria score comprises a memorability score for a segment weighted by a memorability weight and an additional video metric weighted by an additional video metric weight, wherein the additional video metric comprises one or more of video uniformity and video representativeness; selecting the summary subset based on determining that the summary subset maximizes the sum of criteria scores and is less than or equal to the summary length. ### Claim 17 (depends on 12) The non-transitory computer-readable medium of claim 12, wherein the visual summary content comprises a set of thumbnail images, wherein generating the set of thumbnail images comprises: computing visual quality scores for the subset of the segments; extracting thumbnail images from the subset of the segments based on the visual quality scores; and selecting the extracted thumbnail images as the set of thumbnail images.