# US 12,010,405 B2 · Generating video summary > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,010,405 B2 - Document ID: US12010405B2 - Status: Granted - Kind: B2 - Application number: 17/457,506 - Filed: 2021-12-03 - Published: 2023-06-08 - Granted: 2024-06-11 - Earliest claimed: 2021-12-03 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Kun Yan Yin; Lei Mei; Yan Hu; Yan Feng Han; Qi Ruan - Web page: https://patentgrep.com/patent/US12010405B2 ## CPC - H04N 21/8549: Creating video summaries, e.g. movie trailer {(retrieval in video databases by using presentations in form of a video summary G06F16/739)} - G06V 10/761: {Proximity, similarity or dissimilarity measures} - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/47: {Detecting features for summarising video content} - G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/4394: {involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58)} - H04N 21/44224: {Monitoring of user activity on external systems, e.g. Internet browsing} - H04N 21/4756: {for rating content, e.g. scoring a recommended movie} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) ## Abstract A computer-implemented method includes receiving a viewer request for playing a video summary of a video, wherein the viewer request includes a length of the video summary, generating the video summary of the viewer-requested length comprising a set of frames selected from the video based on audience reviews of the video, and playing a video stream of the video summary. ## Claims ### Claim 1 A computer-implemented method comprising: receiving, by one or more processing units, a viewer request for playing a video summary of a video, wherein the viewer request includes a length of the video summary; and in response to receiving the viewer request: obtaining, by one or more processing units, audience reviews on the video; extracting images from the audience reviews, the images being derived from the video; extracting, by one or more processing units, images features of the images to generate image feature vectors, each of the image feature vectors corresponding to a respective one of the images; grouping, by one or more processing units, the image feature vectors into clusters, the clusters being ranked based on their sizes and each having respective center point image feature vectors; designating, by one or more processing units, one or more images corresponding to respective center point image feature vectors of one or more top-ranked clusters of the clusters as the one or more representative images; determining, by one or more processing units, weights of importance of the one or more representative images based on the sizes of the one or more top-ranked clusters; extracting, by one or more processing units, textual messages from the audience reviews; extracting, by one or more processing units, text features of the textual messages to generate text feature vectors, each of the text feature vectors corresponding to a respective one of the textual messages; grouping, by one or more processing units, the text feature vectors into clusters each having respective center point text feature vectors that represent different topics, the topics each having respective weights of interest measured by sizes of respective clusters of the text feature vectors; associating, by one or more processing units, each of the one or more images with one of the topics to which the representative image is most relevant; adjusting, by one or more processing units, the weights of importance of the one or more representative images by using the weights of interest of respective associated topics; identifying, by one or more processing units, candidate frames of the video that are similar to the one or more representative images, wherein the candidate frames are ranked in order of their confidence levels of similarity to the one or more representative images and weights of importance of the one or more representative images; selecting, by one or more processing units, the set of frames based on the ranked candidate frames; generating, by one or more processing units, the video summary of the viewer-requested length comprising the set of frames; and playing, by one or more processing units, a video stream of the video summary. ### Claim 2 (depends on 1) The method of claim 1, further comprising: generating, by one or more processing units, the video stream in response to receiving the viewer request; wherein the set of frames selected is based solely on audience reviews of the video. ### Claim 3 (depends on 1) The method of claim 1, wherein the identifying candidate frames of the video comprises: comparing, by one or more processing units, frames of the video with the one or more representative images to calculate their confidence levels of similarity to respective representative images; and determining, by one or more processing units, the frames that have confidence levels greater than a threshold as the candidate frames, wherein the candidate frames are divided into tiers each associated with respective representative images, the tiers being ranked in order of the weights of importance of their associated representative images, and wherein the candidate frames are further ranked in order of their confidence levels of similarity. ### Claim 4 (depends on 3) The method of claim 3, wherein the identifying candidate frames of the video comprises: determining, by one or more processing units, frames that are adjacent to the candidate frames by an amount of time, wherein the frames have confidence levels lesser than the threshold; and including the frames as candidate frames. ### Claim 5 A system comprising: one or more processors; a memory coupled to the one or more processors; and a set of computer program instructions stored in the memory and executed by the one or more processors to implement a method comprising: receiving a viewer request for playing a video summary of a video, wherein the viewer request includes a length of the video summary; and in response to receiving the viewer request: obtaining the audience reviews on the video; extracting images from the audience reviews, the images being derived from the video; extracting image features of the images to generate image feature vectors, each of the image feature vectors corresponding to a respective one of the images; grouping the image feature vectors into clusters, the clusters being ranked based on their sizes and each having respective center point image feature vectors; designating one or more images corresponding to respective center point image feature vectors of one or more top-ranked clusters of the clusters as the one or more representative images; determining weights of importance of the one or more representative images based on the sizes of the one or more top-ranked clusters; extracting textual messages from the audience reviews; extracting text features of the textual messages to generate text feature vectors, each of the text feature vectors corresponding to a respective one of the textual messages; grouping the text feature vectors into clusters each having respective center point text feature vectors that represent different topics, the topics each having respective weights of interest measured by sizes of respective clusters of the text feature vectors; associating each of the one or more images with one of the topics to which the representative image is most relevant; adjusting the weights of importance of the one or more representative images by using the weights of interest of respective associated topics; identifying candidate frames of the video that are similar to the one or more representative images, wherein the candidate frames are ranked in order of their confidence levels of similarity to the one or more representative images and weights of importance of the one or more representative images; selecting the set of frames based on the ranked candidate frames; generating the video summary of the viewer-requested length comprising the set of frames; and playing a video stream of the video summary. ### Claim 6 (depends on 5) The system of claim 5, the method further comprising: generating, by one or more processing units, the video stream in response to receiving the viewer request; wherein the set of frames selected is based solely on audience reviews of the video. ### Claim 7 (depends on 5) The system of claim 5, wherein the identifying candidate frames of the video comprises: comparing frames of the video with the one or more representative images to calculate their confidence levels of similarity to respective representative images; and determining the frames that have confidence levels greater than a threshold as the candidate frames, wherein the candidate frames are divided into tiers each associated with respective representative images, the tiers being ranked in order of the weights of importance of their associated representative images, and wherein the candidate frames are further ranked in order of their confidence levels of similarity. ### Claim 8 (depends on 7) The system of claim 7, wherein the identifying candidate frames of the video comprises: determining, by one or more processing units, frames that are adjacent to the candidate frames by an amount of time, wherein the frames have confidence levels lesser than the threshold; and including the frames as candidate frames. ### Claim 9 A computer program product, wherein the computer program product comprises a computer readable storage medium having program instructions embodied therewith, wherein the program instructions are executable by one or more processors to implement a method comprising: receiving a viewer request for playing a video summary of a video, wherein the viewer request includes a length of the video summary; and in response to receiving the viewer request: obtaining, by one or more processing units, the audience reviews on the video; extracting, by one or more processing units, images from the audience reviews, the images being derived from the video; extracting, by one or more processing units, image features of the images to generate image feature vectors, each of the image feature vectors corresponding to a respective one of the images; grouping, by one or more processing units, the image feature vectors into clusters, the clusters being ranked based on their sizes and each having respective center point image feature vectors; designating, by one or more processing units, one or more images corresponding to respective center point image feature vectors of one or more top-ranked clusters of the clusters as the one or more representative images; determining, by one or more processing units, weights of importance of the one or more representative images based on the sizes of the one or more top-ranked clusters; extracting, by one or more processing units, textual messages from the audience reviews; extracting, by one or more processing units, text features of the textual messages to generate text feature vectors, each of the text feature vectors corresponding to a respective one of the textual messages; grouping, by one or more processing units, the text feature vectors into clusters each having respective center point text feature vectors that represent different topics, the topics each having respective weights of interest measured by sizes of respective clusters of the text feature vectors; associating, by one or more processing units, each of the one or more images with one of the topics to which the representative image is most relevant; adjusting, by one or more processing units, the weights of importance of the one or more representative images by using the weights of interest of respective associated topics; identifying, by one or more processing units, candidate frames of the video that are similar to the one or more representative images, wherein the candidate frames are ranked in order of their confidence levels of similarity to the one or more representative images and weights of importance of the one or more representative images; selecting, by one or more processing units, the set of frames based on the ranked candidate frames; generating, by one or more processing units, the video summary of the viewer-requested length comprising the set of frames; and playing a video stream of the video summary. ### Claim 10 (depends on 9) The computer program product of claim 9, the method further comprising: generating, by one or more processing units, the video stream in response to receiving the viewer request; wherein the set of frames selected is based solely on audience reviews of the video. ### Claim 11 (depends on 9) The computer program product of claim 9, wherein the identifying candidate frames of the video comprises: comparing frames of the video with the one or more representative images to calculate their confidence levels of similarity to respective representative images; and determining the frames that have confidence levels greater than a threshold as the candidate frames, wherein the candidate frames are divided into tiers each associated with respective representative images, the tiers being ranked in order of the weights of importance of their associated representative images, and wherein the candidate frames are further ranked in order of their confidence levels of similarity.