# US 12,008,811 B2 · Machine learning-based selection of a representative video frame within a messaging application > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,811 B2 - Document ID: US12008811B2 - Status: Granted - Kind: B2 - Application number: 17/550,852 - Filed: 2021-12-14 - Published: 2022-06-30 - Granted: 2024-06-11 - Earliest claimed: 2020-12-30 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: Kavya Venkata Kota Kopparapu; Benjamin Dodson; Francesc Xavier Drudis Rius; Angus Kong; Richard Leider; Jian Ren; Sergey Tulyakov; Jiayao Yu - Web page: https://patentgrep.com/patent/US12008811B2 ## CPC - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06F 16/739: {in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames} - G06F 16/785: {using colour or luminescence} - G06N 20/00: Machine learning - G06T 5/70: Denoising; Smoothing - G06V 10/70: using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88) - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes} - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Aspects of the present disclosure involve a system comprising a medium storing a program and method for machine-learning based selection of a representative video frame. The program and method provide for receiving a set of video frames; determining a first subset of frames by removing frames outside of an image quality threshold; determining a second subset by removing frames outside of an image stillness threshold; computing feature data for each frame in the second subset; providing, for each frame in the second subset, the feature data to a machine learning model (MLM), the MLM being configured to output a score for each frame in the second subset of frames based on the feature data, the MLM having been trained with a first set of images labeled based on aesthetics, and with a second set of images labeled based on image quality; and selecting a frame based on output scores. ## Claims ### Claim 1 A method, comprising: receiving a set of video frames corresponding to a video; determining a first subset of video frames by removing, from the set of video frames, those video frames which are outside of an image quality threshold; determining a second subset of video frames by removing, from the first subset of video frames, those video frames which are outside of an image stillness threshold; computing feature data for each video frame in the second subset of video frames; providing, for each video frame in the second subset of video frames, the feature data of the video frame as input to a machine learning model, wherein the machine learning model is configured to output a score for each video frame in the second subset of video frames based on the feature data of the video frame, the machine learning model having been trained with a first set of images labeled based on image aesthetics, and further having been trained with second set of images labeled based on image quality, the first and second set of images being associated with different domains; and selecting, from among the second subset of video frames, a video frame to represent the set of video frames based on the scores output by the machine learning model. ### Claim 2 (depends on 1) The method of claim 1, wherein determining the first subset of video frames comprises: applying, for each video frame in the set of video frames, respective image quantifiers that relate to luminance, sharpness, uniformity and blurriness of the video frame; and removing, from the set of video frames, those video frames which are above or below a preset threshold with respect to each of the respective image quantifiers. ### Claim 3 (depends on 1) The method of claim 1, wherein determining the second subset of video frames comprises: calculating a respective stillness value, corresponding to pixel-wise frame difference, between consecutive video frames in the first subset of video frames; and removing, from the first subset of video frames, a preset amount of video frames based on the respective stillness values. ### Claim 4 (depends on 1) The method of claim 1, wherein the domain associated with the first set of images corresponds to an aesthetic visual analysis (AVA) database, wherein the video is stored in association with a messaging application, and wherein the domain associated with the second set of images corresponds to the messaging application, the second set of images corresponding to additional videos stored in association with the messaging application. ### Claim 5 (depends on 4) The method of claim 4, wherein the second set of images corresponds to randomly-selected video frames in the additional videos stored in association with the messaging application. ### Claim 6 (depends on 4) The method of claim 4, wherein training of the machine learning model comprises: performing a first stage of training the machine learning model based on the first set of images; providing the second set of images as input to the machine learning model as trained in the first stage of training, in order to label the second set of images; generating a third set of images by decreasing image quality for the second set of images; and performing a second stage of training the machine learning model based on the labeled second set of images and the third set of images. ### Claim 7 (depends on 6) The method of claim 6, wherein decreasing image quality for the second set of images is based on at least one of down-sampling, adjusting exposure or Gaussian blurring of the second set of images. ### Claim 8 (depends on 6) The method of claim 6, wherein training of the machine learning model further comprises: performing a third stage of training the machine learning model based on a labeled subset of images included within other videos stored in association with the messaging application; providing an unlabeled subset of images included within the other videos to the machine learning model having been trained by the third stage of training, in order to pseudo-label the unlabeled subset of video frames; and performing a fourth stage of training the machine learning model based on the labeled subset of images and the pseudo-labeled subset of images. ### Claim 9 (depends on 8) The method of claim 8, the labeled subset of images having been labeled based on a preference for at least one of a subject of the image being centered, the subject being larger, the subject being foregrounded, or the subject having a predefined expression or characteristic. ### Claim 10 (depends on 1) The method of claim 1, wherein the selected video frame corresponds to a thumbnail image which represents the set of video frames. ### Claim 11 (depends on 1) The method of claim 1, wherein the selected video frame is used for at least one of visual tagging of the video, labeling a highlighted part of the video, ranking the video relative to other videos for a video compilation, content curation of videos within a map interface, or generating an animated thumbnail based on other video frames selected to present the set of video frames. ### Claim 12 A system comprising: a processor; and a memory storing instructions that, when executed by the processor, configure the processor to perform operations comprising: receiving a set of video frames corresponding to a video; determining a first subset of video frames by removing, from the set of video frames, those video frames which are outside of an image quality threshold; determining a second subset of video frames by removing, from the first subset of video frames, those video frames which are outside of an image stillness threshold; computing feature data for each video frame in the second subset of video frames; providing, for each video frame in the second subset of video frames, the feature data of the video frame as input to a machine learning model, wherein the machine learning model is configured to output a score for each video frame in the second subset of video frames based on the feature data of the video frame, the machine learning model having been trained with a first set of images labeled based on image aesthetics, and further having been trained with second set of images labeled based on image quality, the first and second set of images being associated with different domains; and selecting, from among the second subset of video frames, a video frame to represent the set of video frames based on the scores output by the machine learning model. ### Claim 13 (depends on 12) The system of claim 12, wherein determining the first subset of video frames comprises: applying, for each video frame in the set of video frames, respective image quantifiers that relate to luminance, sharpness, uniformity and blurriness of the video frame; and removing, from the set of video frames, those video frames which are above or below a preset threshold with respect to each of the respective image quantifiers. ### Claim 14 (depends on 12) The system of claim 12, wherein determining the second subset of video frames comprises: calculating a respective stillness value, corresponding to pixel-wise frame difference, between consecutive video frames in the first subset of video frames; and removing, from the first subset of video frames, a preset amount of video frames based on the respective stillness values. ### Claim 15 (depends on 12) The system of claim 12, wherein the domain associated with the first set of images corresponds to an aesthetic visual analysis (AVA) database, wherein the video is stored in association with a messaging application, and wherein the domain associated with the second set of images corresponds to the messaging application, the second set of images corresponding to additional videos stored in association with the messaging application. ### Claim 16 (depends on 15) The system of claim 15, wherein the second set of images corresponds to randomly-selected video frames in the additional videos stored in association with the messaging application. ### Claim 17 (depends on 15) The system of claim 15, wherein training of the machine learning model comprises: performing a first stage of training the machine learning model based on the first set of images; providing the second set of images as input to the machine learning model as trained in the first stage of training, in order to label the second set of images; generating a third set of images by decreasing image quality for the second set of images; and performing a second stage of training the machine learning model based on the labeled second set of images and the third set of images. ### Claim 18 (depends on 17) The system of claim 17, wherein decreasing image quality for the second set of images is based on at least one of down-sampling, adjusting exposure or Gaussian blurring of the second set of images. ### Claim 19 (depends on 17) The system of claim 17, wherein training of the machine learning model further comprises: performing a third stage of training the machine learning model based on a labeled subset of images included within other videos stored in association with the messaging application; providing an unlabeled subset of images included within the other videos to the machine learning model having been trained by the third stage of training, in order to pseudo-label the unlabeled subset of video frames; and performing a fourth stage of training the machine learning model based on the labeled subset of images and the pseudo-labeled subset of images. ### Claim 20 A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising: receiving a set of video frames corresponding to a video; determining a first subset of video frames by removing, from the set of video frames, those video frames which are outside of an image quality threshold; determining a second subset of video frames by removing, from the first subset of video frames, those video frames which are outside of an image stillness threshold; computing feature data for each video frame in the second subset of video frames; providing, for each video frame in the second subset of video frames, the feature data of the video frame as input to a machine learning model, wherein the machine learning model is configured to output a score for each video frame in the second subset of video frames based on the feature data of the video frame, the machine learning model having been trained with a first set of images labeled based on image aesthetics, and further having been trained with second set of images labeled based on image quality, the first and second set of images being associated with different domains; and selecting, from among the second subset of video frames, a video frame to represent the set of video frames based on the scores output by the machine learning model.