# US 2023/0054130 A1 · Optimal format selection for video players based on predicted visual quality using machine learning > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0054130 A1 - Document ID: US20230054130A1 - Status: Published application - Kind: A1 - Application number: 17/790,102 - Filed: 2019-12-31 - Published: 2023-02-23 - Earliest claimed: 2019-12-31 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Yilin Wang; Yue Guo; Balineedu Chowdary Adsumilli - Web page: https://patentgrep.com/patent/US20230054130A1 ## CPC - H04N 17/004: {for digital television systems} - H04N 21/234309: {by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 or from Quicktime to Realvideo (conversion of standards in analog television systems H04N7/01)} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - H04N 19/12: Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264 - H04N 19/154: Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion (use of rate-distortion criteria H04N19/147) - H04N 19/40: using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/234363: {by altering the spatial resolution, e.g. for clients with a lower screen resolution} - H04N 21/23439: {for generating different versions} - H04N 21/44029: {for generating different versions} - G06N 20/10: using kernel methods, e.g. support vector machines [SVM] - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A system and methods are disclosed for optimal format selection for video players based on visual quality. The method includes generating a plurality of reference transcoded versions of a reference video, obtaining quality scores for frames of the plurality of reference transcoded versions of the reference video, generating a first training input comprising a set of color attributes, spatial attributes, and temporal attributes of the frames of the reference video, and generating a first target output for the first training input, wherein the first target output comprises the quality scores for the frames of the plurality of reference transcoded versions of the reference video. The method further includes providing the training data to train a machine learning model on (i) a set of training inputs comprising the first training input and (ii) a set of target outputs comprising the first target output. ## Claims ### Claim 1 A method comprising: generating training data for a machine learning model to be trained to identify quality scores for a set of transcoded versions of a new video at a set of display resolutions, wherein generating the training data comprises: generating a plurality of reference transcoded versions of a reference video; obtaining quality scores for frames of the plurality of reference transcoded versions of the reference video; generating a first training input comprising a set of color attributes, spatial attributes, and temporal attributes of the frames of the reference video; and generating a first target output for the first training input, wherein the first target output comprises the quality scores for the frames of the plurality of reference transcoded versions of the reference video; and providing the training data to train the machine learning model on (i) a set of training inputs comprising the first training input and (ii) a set of target outputs comprising the first target output. ### Claim 2 (depends on 1) The method of claim 1 wherein the quality scores comprise peak signal-to-noise ratio (PSNR) of the frames. ### Claim 3 (depends on 1) The method of claim 1 wherein the quality scores comprise video multimethod assessment fusion (VMAF) of the frames. ### Claim 4 (depends on 1) The method of claim 1 wherein the color attributes comprises at least one of an RGB or Y value of the frames. ### Claim 5 (depends on 1) The method of claim 1 wherein the spatial attributes comprises a Gabor feature filter bank. ### Claim 6 (depends on 1) The method of claim 1 wherein the temporal attributes comprises an optical flow. ### Claim 7 (depends on 1) The method of claim 1 wherein the machine learning model is configured to process the new video and generate one or more outputs indicating a quality score for the set of transcoded versions of the new video at the set of display resolutions. ### Claim 8 (depends on 1) The method of claim 1 wherein the plurality of transcoded versions of the reference video comprise a transcoding of the reference video at each of a plurality of different video resolutions, transcoding configurations, and the different display resolutions. ### Claim 9 An apparatus comprising: a memory to store a video; and a processing device, operatively coupled to the memory, to: provide sampled frames of the video as input to a trained machine learning model; obtain one or more outputs from the trained machine learning model; and extract, from the one or more outputs, a quality score that indicates a perceptual quality of the video at a first video format, a first transcoding configuration, and a first display resolution. ### Claim 10 (depends on 9) The apparatus of claim 9, wherein the processing device is further to provide the quality score to a client device, the quality score to inform format selection of the video at a media viewer of the client device, wherein the first display resolution is a display resolution of the client device. ### Claim 11 (depends on 10) The apparatus of claim 10 wherein the format selection of the video at the media viewer is based on whether a difference between the quality score and another quality score of the one or more outputs exceeds a threshold value, the another quality score indicating the perceptual quality at a second video format, a second transcoding configuration, and the first display resolution. ### Claim 12 (depends on 9) The apparatus of claim 9 wherein the quality score comprises at least one of a peak signal-to-noise ratio (PSNR) measurement or video multimethod assessment fusion (VMAF) measurement. ### Claim 13 (depends on 9) The apparatus of claim 9 wherein the trained machine learning model is trained with an input-output mapping comprising an input and an output, the input based on a set of color attributes, spatial attributes, and temporal attributes of frames of a reference video, and the output based on quality scores for frames of a plurality of transcoded versions of the reference video. ### Claim 14 (depends on 13) The apparatus of claim 13 wherein the color attributes comprises at least one of RGB or Y value of the frames, the spatial attributes comprises a Gabor feature filter bank of the frames, and the temporal attributes comprise an optical flow of the frames. ### Claim 15 (depends on 9) The apparatus of claim 9 wherein the one or more outputs from the trained machine learning model comprise a plurality of quality scores corresponding to a plurality of possible index tuples of video format, transcoding configuration, and display resolution. ### Claim 16 A non-transitory machine-readable storage medium storing instructions which, when executed, cause a processing device to perform operations comprising: generating training data for a machine learning model to be trained to identify quality scores for a set of transcoded versions of a new video at a set of display resolutions, wherein generating the training data comprises: generating a plurality of reference transcoded versions of a reference video; obtaining quality scores for frames of the plurality of reference transcoded versions of the reference video; generating a first training input comprising a set of color attributes, spatial attributes, and temporal attributes of the frames of the reference video; and generating a first target output for the first training input, wherein the first target output comprises the quality scores for the frames of the plurality of reference transcoded versions of the reference video; and providing the training data to train the machine learning model on (i) a set of training inputs comprising the first training input and (ii) a set of target outputs comprising the first target output. ### Claim 17 (depends on 16) The non-transitory machine-readable storage medium of claim 16 wherein the quality scores comprise at least one of peak signal-to-noise ratio (PSNR) of the frames or video multimethod assessment fusion (VMAF) of the frames. ### Claim 18 (depends on 16) The non-transitory machine-readable storage medium of claim 16 wherein the color attributes comprises at least one of an RGB or Y value of the frames, and wherein the spatial attributes comprises a Gabor feature filter bank, and wherein the temporal attributes comprises an optical flow. ### Claim 19 (depends on 16) The non-transitory machine-readable storage medium of claim 16 wherein the machine learning model is a convolutional neural network (CNN) training model. ### Claim 20 (depends on 16) The non-transitory machine-readable storage medium of claim 16 wherein the machine learning model is configured to process the new video and to generate one or more outputs indicating a quality score for the set of transcoded versions of the new video at the set of display resolutions.