# US 10,454,987 B2 · Bitrate optimization for multi-representation encoding using playback statistics > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,454,987 B2 - Document ID: US10454987B2 - Status: Granted - Kind: B2 - Application number: 15/337,806 - Filed: 2016-10-28 - Published: 2018-05-03 - Granted: 2019-10-22 - Earliest claimed: 2016-10-28 - Assignees: Google Inc. - Applicants: Google Inc. - Inventors: Chao Chen; Yao-Chung Lin; Anil Kokaram; Steve Benting - Web page: https://patentgrep.com/patent/US10454987B2 ## CPC - H04L 65/70: Media network packetisation - H04L 47/38: by adapting coding or compression rate - H04L 65/61: for supporting one-way streaming services, e.g. Internet radio - H04N 21/23439: {for generating different versions} - H04N 21/258: Client or end-user data management, e.g. managing client capabilities, user preferences or demographics, processing of multiple end-users preferences to derive collaborative data {(arrangements for services using the result on the distributing side of broadcast systems H04H60/66; profiles in network data switching protocols H04L67/30)} - H04L 65/80: Responding to QoS - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Implementations disclose bitrate optimization for multi-representation encoding using playback statistics. A method includes generating multiple versions of a segment of a source video, the versions comprising encodings of the segment at different encoding bitrates for each resolution of the segment, measuring a quality metric for each version of the segment, generating rate-quality models for each resolution of the segment based on the measured quality metrics corresponding to the resolutions, generating a probability model to predict requesting probabilities that representations of the segment are requested, the probability model based on a joint probability distribution of network speed and viewport size that is generated from client-side feedback statistics associated with prior playbacks of other videos, determining an encoding bitrate for each of the representations of the segment based on the rate-quality models and the probability model, and assigning determined encoding bitrates to corresponding representations of the segment. ## Claims ### Claim 1 A method comprising: generating multiple versions of a segment of a source video, the versions comprising encodings of the segment at different encoding bitrates for each resolution of the segment; measuring a quality metric for each version of the segment; generating rate-quality models for each resolution of the segment based on the measured quality metrics corresponding to the resolutions; obtaining client-side feedback statistics associated with prior playbacks of other videos different from the source video, wherein the client-side feedback statistics comprise playback traces transmitted from media players at client devices in a same geographic region of the source video, the playback traces comprising a respective network speed and a respective viewport size utilized on a respective client device for each of the prior playbacks; generating, by a processing device, a probability model specific to the geographic region to predict requesting probabilities that representations of the segment are requested, the probability model based on a joint probability distribution of network speed and viewport size that is generated from the client-side feedback statistics associated with the prior playbacks of the other videos; determining, by the processing device, an encoding bitrate for each of the representations of the segment based on the rate-quality models and the probability model; and assigning determined encoding bitrates to corresponding representations of the segment. ### Claim 2 (depends on 1) The method of claim 1, wherein the segment comprises the entire source video. ### Claim 3 (depends on 1) The method of claim 1, wherein the requesting probability for one of the representations is further based on: the encoding bitrate of the representation and a relation of the encoding bitrate to network speed in the joint probability distribution; and the resolution of the representation and a relation of the resolution to viewport size in the joint probability distribution. ### Claim 4 (depends on 1) The method of claim 1, wherein the joint probability distribution is generated from cumulative measurements of the network speeds determined from the playback traces and from cumulative measurements of the viewport sizes determined from the playback traces. ### Claim 5 (depends on 4) The method of claim 4, wherein the playback traces are collected for a type of the source video, and wherein the joint probability distribution is specific to the type of the source video. ### Claim 6 (depends on 1) The method of claim 1, wherein determining the encoding bitrate for each of the representations further comprises minimizing an average egress traffic for the segment such that an average quality of the segment is maintained at or above a defined quality level, wherein the average egress traffic is a function of the different encoding bitrates and the requesting probabilities, and wherein the average quality is a function of the quality metrics and the requesting probabilities. ### Claim 7 (depends on 1) The method of claim 1, wherein determining the encoding bitrate for each of the representations further comprises maximizing an average quality for the segment such that an average egress traffic of the segment is maintained at or below a defined video egress traffic level, wherein the average quality is a function of the quality metrics and the requesting probabilities, and wherein the average egress traffic is a function of the multiple bitrates and the requesting probabilities. ### Claim 8 (depends on 1) The method of claim 1, wherein assigning the determined encoding bitrates to the corresponding representations further comprises providing the selected encoding bitrates to at least one transcoder for encoding of each of the representations of the segment at the corresponding bitrate. ### Claim 9 (depends on 1) The method of claim 1, wherein the representation comprises a bitrate/resolution combination of the segment, and wherein the segment comprises one or more representations for each of the resolutions of the segment. ### Claim 10 (depends on 1) The method of claim 1, wherein the quality metric comprises at least one of a Peak Signal-to-Noise Ratio (PSNR) measurement or a Structural Similarity (SSIM) measurement. ### Claim 11 A system comprising: a memory; and a processing device coupled to the memory, wherein the processing device is to: obtain client-side feedback statistics associated with prior playbacks of multiple videos different from a source video, wherein the client-side feedback statistics comprise playback traces transmitted from media players at client devices in a same geographic region of the source video, the playback traces comprising a respective network speed and a respective viewport size utilized on a respective client device for each of the prior playbacks; determine a joint probability distribution for network speed and viewport size based on the client-side feedback statistics received from client systems, the joint probability distribution specific to the geographic region; generate rate-quality models for resolutions of a segment of the source video based on quality metrics measured for the segment; estimate a delivered quality and egress for representations of the segment based on the generated rate-quality models and based on requesting probabilities that the representations are requested, wherein the requesting probabilities are based on the joint probability distribution; and determine a set of bitrates comprising a bitrate to correspond to each of the representations of the segment, the set of bitrates determined to minimize the egress while maintaining the delivered quality at or above a quality threshold value. ### Claim 12 (depends on 11) The system of claim 11, wherein the requesting probability that one of the representations is requested is further based on: the bitrate of the representation and a relation of the bitrate to network speed in the joint probability distribution; and the resolution of the representation and a relation of the resolution to viewport size in the joint probability distribution. ### Claim 13 (depends on 11) The system of claim 11, wherein the joint probability distribution is generated from cumulative measurements of the network speeds determined from the playback traces and from cumulative measurements of the viewport sizes determined from the playback traces. ### Claim 14 (depends on 13) The system of claim 13, wherein the playback traces are collected for a type of the source video, and wherein the joint probability distribution is specific to the type of the source video. ### Claim 15 (depends on 11) The system of claim 11, wherein the processing device is further to provide the determined set of bitrates to at least one transcoder for encoding of each of the representations of the segment at the corresponding bitrate. ### Claim 16 (depends on 11) The system of claim 11, wherein the delivered quality is based on at least one of Peak Signal-to-Noise Ratio (PSNR) measurements of the encodings or Structural Similarity (SSIM) measurements of the encodings. ### Claim 17 A non-transitory machine-readable storage medium storing instructions which, when executed, cause a processing device to perform operations comprising: obtaining client-side feedback statistics associated with prior playbacks of multiple videos different from a source video, wherein the client-side feedback statistics comprise playback traces transmitted from media players at client devices in a same geographic region of the source video, the playback traces comprising a respective network speed and a respective viewport size utilized on a respective client device for each of the prior playbacks; determining a joint probability distribution for network speed and viewport size based on the client-side feedback statistics received from client systems, the joint probability distribution specific to the geographic region; generating rate-quality models for resolutions of a segment of a video based on quality metrics measure for the segment; estimating, by the processing device, a delivered quality and egress for representations of the segment based on the generated rate-quality models and based on requesting probabilities that the representations is requested, wherein the requesting probabilities are based on the joint probability distribution; and determining, by the processing device, a set of bitrates comprising a bitrate to correspond to each of the representations of the segment, the set of bitrates determined to maximize the delivered quality while keeping the egress at or below an egress threshold value. ### Claim 18 (depends on 17) The non-transitory machine-readable storage medium of claim 17, wherein the requesting probability that one of the representations is requested is further based on: the bitrate of the representation and a relation of the bitrate to network speed in the joint probability distribution; and the resolution of the representation and a relation of the resolution to viewport size in the joint probability distribution. ### Claim 19 (depends on 17) The non-transitory machine-readable storage medium of claim 17, wherein the joint probability distribution is generated from cumulative measurements of the network speeds determined from the playback traces and from cumulative measurements of the viewport sizes determined from the playback traces. ### Claim 20 (depends on 19) The non-transitory machine-readable storage medium of claim 19, wherein the playback traces are collected for a type of the source video, and wherein the joint probability distribution is specific to the type of source video. ### Claim 21 (depends on 17) The non-transitory machine-readable storage medium of claim 17, wherein the processing device is further to provide the determined set of bitrates to at least one transcoder for encoding of each of the representations of the segment at the corresponding bitrate. ### Claim 22 (depends on 17) The non-transitory machine-readable storage medium of claim 17, wherein the delivered quality is based on at least one of Peak Signal-to-Noise Ratio (PSNR) measurements of the segment or Structural Similarity (SSIM) measurements of the segment.