# US 11,910,042 B2 · Dynamic allocation of compute capacity in video stream processing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,910,042 B2 - Document ID: US11910042B2 - Status: Granted - Kind: B2 - Application number: 17/834,842 - Filed: 2022-06-07 - Published: 2022-09-22 - Granted: 2024-02-20 - Earliest claimed: 2014-12-08 - Assignees: Harmonic, Inc. - Applicants: Harmonic, Inc. - Inventors: Neven Haltmayer - Web page: https://patentgrep.com/patent/US11910042B2 ## CPC - H04N 21/2385: Channel allocation (H04N21/266 takes precedence); Bandwidth allocation (H04N21/24 takes precedence {; allocation of channels according to the instantaneous demands of the users in time-division multiplex systems H04J3/1682; admission control, resource allocation in open networks H04L12/5692; arrangements for maintenance or administration in data switching networks involving bandwidth and capacity management H04L41/0896; negotiating bandwidth in wireless networks H04W28/16}) - H04N 7/01: Conversion of standards {, e.g. involving analogue television standards or digital television standards processed at pixel level} - H04N 7/0806: {the signals being two or more video signals (H04N7/0803, H04N7/081 take precedence)} - H04N 19/127: Prioritisation of hardware or computational resources - H04N 19/154: Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion (use of rate-distortion criteria H04N19/147) - H04N 19/172: the region being a picture, frame or field - H04N 19/179: the unit being a scene or a shot - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/2405: {Monitoring of the internal components or processes of the server, e.g. server load (allocation of resources in multiprogramming arrangements G06F9/50; performance measurement of computer activity G06F11/34)} - H04N 21/241: Operating system [OS] processes, e.g. server setup - H04N 21/6143: {involving transmission via a satellite} - H04N 19/136: Incoming video signal characteristics or properties - H04N 19/436: using parallelised computational arrangements - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Approaches for dynamically allocating compute capacity for processing a video stream. Video complexity information for two or more digital video streams actively being processed by one or more video encoders is determined at periodic intervals. Video complexity information describes the complexity of digital video carried by the digital video streams across a bounded number of consecutive digital frames which includes digital frames not yet processed by the one or more video encoders. A determination is made as to whether the compute capacity allocated for processing a particular digital video stream should be adjusted in some manner based on the determined video complexity information. The amount of compute capacity allocated for processing the particular digital video stream may be dynamically adjusted in response to maximizing a measure of optimal video quality calculated for the two or more digital video streams using, at least in part, the determined video complexity information. ## Claims ### Claim 1 A non-transitory computer-readable storage medium storing one or more sequences of instructions for dynamically allocating compute capacity for use in encoding a video stream, which when executed by one or more processors, causes: upon the occurrence of each periodic interval in a series of periodic intervals, performing: (a) determining, for a particular future periodic interval of said series of periodic intervals, video complexity information for two or more digital video streams actively being encoded at the same time, (b) determining whether an amount of the compute capacity allocated for encoding a particular digital video stream of the two or more digital video streams should be adjusted during said particular future periodic interval of said series of periodic intervals based on (i) the determined video complexity information and (ii) an assessment of impact to quality of said particular digital video stream resulting from making said adjustment to said amount of compute capacity, and (c) upon determining that the amount of compute capacity currently allocated for encoding said particular digital video stream of the two or more digital video streams should be adjusted during said future periodic interval, adjusting the amount of compute capacity currently allocated during said future periodic interval to encoding the particular digital video stream of the two or more digital video streams. ### Claim 2 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein each periodic interval in the series of periodic intervals corresponds to a frame of digital video in the particular video stream. ### Claim 3 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein each periodic interval in the series of periodic intervals corresponds to a period of time or a certain number of CPU clock cycles. ### Claim 4 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein the determined video complexity information describes the complexity of digital video carried by the two or more digital video streams across a bounded number of consecutive digital frames, wherein said bounded number of consecutive digital frames includes digital frames not yet encoded. ### Claim 5 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein the execution of the one or more sequences of instructions further causes: dynamically adjusting a portion of the compute capacity allocated for encoding the particular digital video stream by maximizing a measure of optimal video quality calculated for the two or more digital video streams using, at least in part, the determined video complexity information. ### Claim 6 (depends on 5) The non-transitory computer-readable storage medium of claim 5, wherein maximizing the measure of optimal video quality for the two or more digital video streams comprises: identifying one or more features in at least one of the two or more digital video streams in which support for will be reduced or ceased during said future periodic interval. ### Claim 7 (depends on 5) The non-transitory computer-readable storage medium of claim 5, wherein the measure of optimal video quality for the two or more digital video streams is determined by assessing the video quality of at least one of the two or more digital video streams at a greater importance than another digital video stream in the two or more digital video streams. ### Claim 8 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein the execution of the one or more sequences of instructions further causes: dynamically increasing some portion of the compute capacity allocated to encode the particular digital video stream based on, at least in part, the determined video complexity information, wherein dynamically increasing comprising: assessing a first impact of decreasing a first portion of the compute capacity allocated to encode at least one digital video stream of the two or more digital video streams; assessing a second impact of increasing a second portion of the compute capacity to encode the particular digital video stream proportional to the first portion. ### Claim 9 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein at least two of the two or more video streams are not encoded on the same physical machine. ### Claim 10 An apparatus for dynamically allocating compute capacity for use in encoding a video stream, comprising: one or more processors; and one or more non-transitory computer-readable storage mediums storing one or more sequences of instructions, which when executed, cause: upon the occurrence of each periodic interval in a series of periodic intervals, performing: (a) determining, for a particular future periodic interval of said series of periodic intervals, video complexity information for two or more digital video streams actively being encoded at the same time, (b) determining whether an amount of the compute capacity allocated for encoding a particular digital video stream of the two or more digital video streams should be adjusted during said particular future periodic interval of said series of periodic intervals based on Ithe determined video complexity information and (ii) an assessment of impact to quality of said particular digital video stream resulting from making said adjustment to said amount of compute capacity, and (c) upon determining that the amount of compute capacity currently allocated for encoding said particular digital video stream of the two or more digital video streams should be adjusted during said future periodic interval, adjusting the amount of compute capacity currently allocated during said future periodic interval to encoding the particular digital video stream of the two or more digital video streams. ### Claim 11 (depends on 10) The apparatus of claim 10, wherein each periodic interval in the series of periodic intervals corresponds to a frame of digital video in the particular video stream. ### Claim 12 (depends on 10) The apparatus of claim 10, wherein each periodic interval in the series of periodic intervals corresponds to a period of time or a certain number of CPU clock cycles. ### Claim 13 (depends on 10) The apparatus of claim 10, wherein the determined video complexity information describes the complexity of digital video carried by the two or more digital video streams across a bounded number of consecutive digital frames, wherein said bounded number of consecutive digital frames includes digital frames not yet encoded by said one or more video encoders. ### Claim 14 (depends on 10) The apparatus of claim 10, wherein the execution of the one or more sequences of instructions further causes: dynamically adjusting a portion of the compute capacity allocated for encoding the particular digital video stream by maximizing a measure of optimal video quality calculated for the two or more digital video streams using, at least in part, the determined video complexity information. ### Claim 15 (depends on 14) The apparatus of claim 14, wherein maximizing the measure of optimal video quality for the two or more digital video streams comprises: identifying one or more features in at least one of the two or more digital video streams in which support for will be reduced or ceased during said periodic interval. ### Claim 16 (depends on 14) The apparatus of claim 14, wherein the measure of optimal video quality for the two or more digital video streams is determined by assessing the video quality of at least one of the two or more digital video streams at a greater importance than another digital video stream in the two or more digital video streams. ### Claim 17 (depends on 10) The apparatus of claim 10, wherein the execution of the one or more sequences of instructions further causes: dynamically increasing some portion of the compute capacity allocated to encode the particular digital video stream based on, at least in part, the determined video complexity information, wherein dynamically increasing comprising: assessing a first impact of decreasing a first portion of the compute capacity allocated to encode at least one digital video stream of the two or more digital video streams; assessing a second impact of increasing a second portion of the compute capacity to encode the particular digital video stream proportional to the first portion. ### Claim 18 (depends on 10) The apparatus of claim 10, wherein at least two of the two or more video streams are not encoded on the same physical machine. ### Claim 19 A method for dynamically allocating compute capacity for use in encoding a video stream, comprising: upon the occurrence of each periodic interval in a series of periodic intervals, performing: (a) determining, for a particular future periodic interval of said series of periodic intervals, video complexity information for two or more digital video streams actively being encoded at the same time, (b) determining whether an amount of the compute capacity allocated for encoding a particular digital video stream of the two or more digital video streams should be adjusted during said particular future periodic interval of said series of periodic intervals based on (i) the determined video complexity information and (ii) an assessment of impact to quality of said particular digital video stream resulting from making said adjustment to said amount of compute capacity, and (c) upon determining that the amount of compute capacity currently allocated for encoding said particular digital video stream of the two or more digital video streams should be adjusted during said future periodic interval, adjusting the amount of compute capacity currently allocated during said future periodic interval to encoding the particular digital video stream of the two or more digital video streams. ### Claim 20 (depends on 19) The method of claim 19, wherein the determined video complexity information describes the complexity of digital video carried by the two or more digital video streams across a bounded number of consecutive digital frames, wherein said bounded number of consecutive digital frames includes digital frames not yet encoded. ### Claim 21 (depends on 19) The method of claim 19, further comprising: dynamically adjusting a portion of the compute capacity allocated for encoding the particular digital video stream by maximizing a measure of optimal video quality calculated for the two or more digital video streams using, at least in part, the determined video complexity information. ### Claim 22 (depends on 21) The method of claim 21, wherein maximizing the measure of optimal video quality for the two or more digital video streams comprises: identifying one or more features in at least one of the two or more digital video streams in which support for will be reduced or ceased during said future periodic interval. ### Claim 23 (depends on 21) The method of claim 21, wherein the measure of optimal video quality for the two or more digital video streams is determined by assessing the video quality of at least one of the two or more digital video streams at a greater importance than another digital video stream in the two or more digital video streams. ### Claim 24 (depends on 19) The method of claim 19, further comprising: dynamically increasing some portion of the compute capacity allocated to encode the particular digital video stream based on, at least in part, the determined video complexity information, wherein dynamically increasing comprising: assessing a first impact of decreasing a first portion of the compute capacity allocated to encode at least one digital video stream of the two or more digital video streams; assessing a second impact of increasing a second portion of the compute capacity to encode the particular digital video stream proportional to the first portion.