# US 11,910,039 B2 · Encoding technique for optimizing distortion and bitrate > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,910,039 B2 - Document ID: US11910039B2 - Status: Granted - Kind: B2 - Application number: 16/882,386 - Filed: 2020-05-22 - Published: 2020-09-10 - Granted: 2024-02-20 - Earliest claimed: 2017-07-18 - Assignees: Netflix, Inc. - Applicants: NETFLIX Inc. - Inventors: Ioannis Katsavounidis - Web page: https://patentgrep.com/patent/US11910039B2 ## CPC - H04N 21/23439: {for generating different versions} - H04N 19/124: Quantisation - H04N 19/147: according to rate distortion criteria (rate-distortion as a criterion for motion estimation H04N19/567) - H04N 19/179: the unit being a scene or a shot - H04N 19/192: the adaptation method, adaptation tool or adaptation type being iterative or recursive - H04N 19/59: involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution - H04N 21/234345: {the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment} - H04N 21/23805: {Controlling the feeding rate to the network, e.g. by controlling the video pump} - H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An encoding engine encodes a video sequence to provide optimal quality for a given bitrate. The encoding engine cuts the video sequence into a collection of shot sequences. Each shot sequence includes video frames captured from a particular capture point. The encoding engine resamples each shot sequence across a range of different resolutions, encodes each resampled sequence with a range of quality parameters, and then upsamples each encoded sequence to the original resolution of the video sequence. For each upsampled sequence, the encoding engine computes a quality metric and generates a data point that includes the quality metric and the resample resolution. The encoding engine collects all such data points and then computes the convex hull of the resultant data set. Based on all convex hulls across all shot sequences, the encoding engine determines an optimal collection of shot sequences for a range of bitrates. ## Claims ### Claim 1 A computer-implemented method, comprising: generating multiple sets of encoded chunks for a source video sequence, wherein each set of encoded chunks corresponds to a different shot sequence included in the source video sequence; performing one or more convex hull operations across a first set of data points associated with a first set of encoded chunks to generate a first convex hull, wherein the first set of encoded chunks is included in the multiple sets of encoded chunks and corresponds to a first shot sequence; computing a first slope value between two data points included in the first convex hull; comparing the first slope value to a second slope value, wherein the second slope value is computed between two data points included in a second convex hull generated for a second shot sequence; and including a first encoded chunk from the first set of encoded chunks in a final encoded version of the source video sequence based on the first slope value having a greater magnitude than the second slope value. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein a subset of data points associated with the first convex hull is optimized across at least two metrics, and wherein a first metric included in the at least two metrics comprises distortion and a second metric included in the at least two metrics comprises bitrate. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein generating the multiple sets of encoded chunks comprises: identifying within the source video sequence a first sequence of frames that is associated with a first point of capture; resampling the first sequence of frames at a plurality of different resolutions to generate a resolution ladder of resampled versions of the first sequence of frames; and encoding each resampled version of the first sequence of frames with a different encoding parameter to generate the first set of encoded chunks. ### Claim 4 (depends on 3) The computer-implement method of claim 3, wherein generating the multiple sets of encoded chunks further comprises: identifying within the source video sequence a second sequence of frames that is associated with a second point of capture that is different than the first point of capture; resampling the second sequence of frames at a plurality of different resolutions to generate a resolution ladder of resampled versions of the second sequence of frames; and encoding each resampled version of the second sequence of frames with a different encoding parameter to generate the first set of encoded chunks. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, further comprising generating the first set of data points by: decoding each encoded chunk in the first set of encoded chunks to generate a first set of decoded chunks; upsampling each decoded chunk in the first set of decoded chunks to a source resolution associated with the source video sequence to generate a first set of upsampled chunks; and generating a different data point for each upsampled chunk in the first set of upsampled chunks. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein a specific data point in the first set of data points is generated by: generating a specific objective quality metric for a specific upsampled chunk in the first set of upsampled chunks; converting the specific objective quality metric to a specific distortion metric; computing a bitrate for the specific upsampled chunk; and combining the specific distortion metric and the bitrate to generate the specific data point. ### Claim 7 (depends on 2) The computer-implemented method of claim 2, further comprising: performing one or more convex hull operations across a second set of data points associated with a second set of encoded chunks associated with the second shot sequence to compute a second subset of data points that are optimized across the at least two metrics; and computing the second slope value between a third data point included in the second subset of data points and a fourth data point included in the second subset of data points. ### Claim 8 (depends on 7) The computer-implemented method of claim 7, further comprising generating the second set of data points by: decoding each encoded chunk in the second set of encoded chunks to generate a second set of decoded chunks; upsampling each decoded chunk in the second set of decoded chunks to a source resolution associated with the source video sequence to generate a second set of upsampled chunks; and generating a different data point for each upsampled chunk in the second set of upsampled chunks. ### Claim 9 (depends on 2) The computer-implemented method of claim 2, wherein performing the one or more convex hull operations across the first set of data points to compute the first subset of data points comprises: determining a first region that includes the first set of data points; identifying a first boundary of the first region, wherein no data points in the first set of points reside on a first side of the first boundary; and discarding any data points that do not reside along the first boundary, wherein each data point that resides along the first boundary optimizes a first metric included in the at least two metrics with respect to a second metric included in the at least two metrics. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the first set of encoded chunks is associated with a first sequence of video frames captured continuously from a first point of capture, and a second set of encoded chunks associated with the second shot sequence is associated with a second sequence of video frames captured continuously from a second point of capture. ### Claim 11 One or more non-transitory computer-readable media storing program instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of: generating multiple sets of encoded chunks for a source video sequence, wherein each set of encoded chunks corresponds to a different shot sequence included in the source video sequence; performing one or more convex hull operations across a first set of data points associated with a first set of encoded chunks to generate a first convex hull, wherein the first set of encoded chunks is included in the multiple sets of encoded chunks and corresponds to a first shot sequence; computing a first slope value between two data points included in the first convex hull; performing a first comparison between the first slope value and a second slope value, wherein the second slope value is computed between two data points included in a second convex hull generated for a second shot sequence; and including a first encoded chunk from the first set of encoded chunks in a final encoded version of the source video sequence based on the first comparison. ### Claim 12 (depends on 11) The one or more non-transitory computer-readable media of claim 11, wherein a subset of data points associated with the first convex hull is optimized across at least two metrics, and wherein a first metric included in the at least two metrics comprises distortion, and a second metric included in the at least two metrics comprises bitrate. ### Claim 13 (depends on 11) The one or more non-transitory computer-readable media of claim 11, wherein generating the multiple sets of encoded chunks comprises: identifying within the source video sequence a first sequence of frames that is associated with a first point of capture; resampling the first sequence of frames at a plurality of different resolutions to generate a resolution ladder of resampled versions of the first sequence of frames; and encoding each resampled version of the first sequence of frames with a different encoding parameter to generate the first set of encoded chunks. ### Claim 14 (depends on 13) The one or more non-transitory computer-readable media of claim 13, wherein generating the multiple sets of encoded chunks further comprises: identifying within the source video sequence a second sequence of frames that is associated with a second point of capture that is different than the first point of capture; resampling the second sequence of frames at a plurality of different resolutions to generate a resolution ladder of resampled versions of the second sequence of frames; and encoding each resampled version of the second sequence of frames with a different encoding parameter to generate the first set of encoded chunks. ### Claim 15 (depends on 11) The one or more non-transitory computer-readable media of claim 11, further comprising generating the first set of data points by: decoding each encoded chunk in the first set of encoded chunks to generate a first set of decoded chunks; upsampling each decoded chunk in the first set of decoded chunks to a source resolution associated with the source video sequence to generate a first set of upsampled chunks; and generating a different data point for each upsampled chunk in the first set of upsampled chunks. ### Claim 16 (depends on 15) The one or more non-transitory computer-readable media of claim 15, wherein a specific data point in the first set of data points is generated by: generating a specific objective quality metric for a specific upsampled chunk in the first set of upsampled chunks; converting the specific objective quality metric to a specific distortion metric; computing a bitrate for the specific upsampled chunk; and combining the specific distortion metric and the bitrate to generate the specific data point. ### Claim 17 (depends on 11) The one or more non-transitory computer-readable media of claim 11, further comprising: performing one or more convex hull operations across a second set of data points associated with a second set of encoded chunks associated with the second shot sequence to compute a subset of data points that are optimized across at least two metrics; and computing the second slope value between a third data point included in the subset of data points and a fourth data point included in the subset of data points. ### Claim 18 (depends on 17) The one or more non-transitory computer-readable media of claim 17, further comprising generating the second set of data points by: decoding each encoded chunk in the second set of encoded chunks to generate a second set of decoded chunks; upsampling each decoded chunk in the second set of decoded chunks to a source resolution associated with the source video sequence to generate a second set of upsampled chunks; and generating a different data point for each upsampled chunk in the second set of upsampled chunks. ### Claim 19 (depends on 11) The one or more non-transitory computer-readable media of claim 11, wherein performing the one or more convex hull operations across the first set of data points to generate the first convex hull comprises: determining a first region that includes the first set of data points; identifying a first boundary of the first region, wherein no data points in the first set of points reside on a first side of the first boundary; and discarding any data points that do not reside along the first boundary, wherein each data point that resides along the first boundary optimizes a first metric with respect to a second metric. ### Claim 20 (depends on 11) The one or more non-transitory computer-readable media of claim 11, wherein the first set of encoded chunks is associated with a first sequence of video frames captured continuously from a first point of capture, and a second set of encoded chunks associated with the second shot sequence is associated with a second sequence of video frames captured continuously from a second point of capture. ### Claim 21 A system, comprising: one or more memories storing instructions; and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform the steps of: generating multiple sets of encoded chunks for a source video sequence, wherein each set of encoded chunks corresponds to a different shot sequence included in the source video sequence; performing one or more convex hull operations across a first set of data points associated with a first set of encoded chunks to generate a first convex hull that includes a first subset of data points, wherein the first set of encoded chunks is included in the multiple sets of encoded chunks and corresponds to a first shot sequence; computing a first parameter for a first data point included in the first convex hull; performing a first comparison between the first parameter and a second parameter computed for a second data point included in a second convex hull generated for a second shot sequence; and including a first encoded chunk from the first set of encoded chunks in a final encoded version of the source video sequence based on the first comparison.