# US 11,445,243 B2 · Adaptive rate control adjustment for hardware encoder > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,445,243 B2 - Document ID: US11445243B2 - Status: Granted - Kind: B2 - Application number: 16/869,835 - Filed: 2020-05-08 - Published: 2020-11-12 - Granted: 2022-09-13 - Earliest claimed: 2019-05-10 - Assignees: Plantronics, Inc. - Applicants: Plantronics, Inc. - Inventors: Jian Wang - Web page: https://patentgrep.com/patent/US11445243B2 ## CPC - H04N 21/2401: {Monitoring of the client buffer} - H04N 21/44004: {involving video buffer management, e.g. video decoder buffer or video display buffer} - H04N 7/147: {Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q)} - H04N 21/23406: {involving management of server-side video buffer} - H04N 21/23805: {Controlling the feeding rate to the network, e.g. by controlling the video pump} - H04N 21/4305: {Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets (arrangements for synchronising receiver with transmitter by comparing receiver clock with transmitter clock H04L7/0012; arrangements for synchronising receiver with transmitter wherein the receiver takes measures against momentary loss of synchronisation H04L7/0083)} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Correcting encode bit rate errors developed by a hardware encoder with an outer software rate adjustment loop. The rate adjustment loop maintains a VBV (video buffering verifier) buffer model. When VBV buffer fullness is over a threshold, the rate adjustment loop drops frames. If below the threshold, periodically a rate adjustment analysis performed. If a number of frames above a threshold have been dropped, the hardware encoder bit rate is decreased. A leaky bucket is checked to determine in bits and out bits. If the number of in bits exceeds the number of out bits times a multiplier, the hardware encoder bit rate is decreased. If the number of in bits times a multiplier is less than the number of out bits, the hardware encoder bit rate is increased. Dropping frames on buffer condition provides shorter term correction and adjustment of the hardware encoder bit rate provides longer term correction. ## Claims ### Claim 1 An encoded video transmitter comprising: a video input; an encoded video output; a hardware video encoder coupled to the video input and to the encoded video output; an outgoing video buffer coupled to the hardware video encoder and to the encoded video output and receiving frames from the hardware video encoder and providing frames to the encoded video output; a processor coupled to the hardware video encoder, the video input and the outgoing video buffer; and a non-transitory memory coupled to the processor and storing instructions to cause the processor to perform a method comprising the steps of: receiving a video frame to be encoded; determining a fullness level value of the outgoing video buffer; causing encoding of the received video frame to be skipped when the buffer fullness level value is above a threshold; and periodically performing video encoding rate operations and adjusting of an encoding bit rate of the hardware video encoder if appropriate. ### Claim 2 (depends on 1) The encoded video transmitter of claim 1, wherein causing encoding of the received video frame to be skipped includes incrementing a counter of skipped frames, and wherein performing video encoding rate operations and adjusting of the encoding bit rate of the hardware video encoder includes: determining if the value of the skipped frame counter is above a second threshold; and decreasing the encoding bit rate of the hardware video encoder when the skipped frame counter value is above the second threshold. ### Claim 3 (depends on 2) The encoded video transmitter of claim 2, wherein performing video encoding rate operations and adjusting of the encoding bit rate of the hardware video encoder includes: determining if the bits flowing into the outgoing video buffer are greater than the bits flowing out of the outgoing video buffer multiplied by a high rate factor; and decreasing the encoding bit rate of the hardware video encoder when the bits flowing into the outgoing video buffer are greater than the bits flowing out of the outgoing video buffer multiplied by a high rate factor. ### Claim 4 (depends on 3) The encoded video transmitter of claim 3, wherein performing video encoding rate operations and adjusting of the encoding bit rate of the hardware video encoder includes: determining if the bits flowing into the outgoing video buffer multiplied by a low rate factor are less than the bits flowing out of the outgoing video buffer; and increasing the encoding bit rate of the hardware video encoder when the bits flowing into the outgoing video buffer multiplied by a low rate factor are less than the bits flowing out of the outgoing video buffer. ### Claim 5 (depends on 2) The encoded video transmitter of claim 2, wherein performing video encoding rate operations and adjusting of the encoding bit rate of the hardware video encoder includes: determining if the bits flowing into the outgoing video buffer multiplied by a low rate factor are less than the bits flowing out of the outgoing video buffer; and increasing the encoding bit rate of the hardware video encoder when the bits flowing into the outgoing video buffer multiplied by a low rate factor are less than the bits flowing out of the outgoing video buffer. ### Claim 6 (depends on 1) The encoded video transmitter of claim 1, wherein performing video encoding rate operations and adjusting of the encoding bit rate of the hardware video encoder includes: determining if the bits flowing into the outgoing video buffer multiplied by a low rate factor are less than the bits flowing out of the outgoing video buffer; and increasing the encoding bit rate of the hardware video encoder when the bits flowing into the outgoing video buffer multiplied by a low rate factor are less than the bits flowing out of the outgoing video buffer. ### Claim 7 (depends on 6) The encoded video transmitter of claim 6, wherein performing video encoding rate operations and adjusting of the encoding bit rate of the hardware video encoder includes: determining if the bits flowing into the outgoing video buffer are greater than the bits flowing out of the outgoing video buffer multiplied by a high rate factor; and decreasing the encoding bit rate of the hardware video encoder when the bits flowing into the outgoing video buffer are greater than the bits flowing out of the outgoing video buffer multiplied by a high rate factor. ### Claim 8 (depends on 1) The encoded video transmitter of claim 1, wherein performing video encoding rate operations and adjusting of the encoding bit rate of the hardware video encoder includes: determining if the bits flowing into the outgoing video buffer are greater than the bits flowing out of the outgoing video buffer multiplied by a high rate factor; and decreasing the encoding bit rate of the hardware video encoder when the bits flowing into the outgoing video buffer are greater than the bits flowing out of the outgoing video buffer multiplied by a high rate factor. ### Claim 9 A method of transmitting encoded video from an encoded video transmitter using a hardware video encoder, the method comprising: receiving a video frame to be encoded; determining a fullness level value of an outgoing video buffer; causing encoding of the received video frame to be skipped when the buffer fullness level value is above a threshold; and periodically performing video encoding rate operations and adjusting of an encoding bit rate of the hardware video encoder if appropriate. ### Claim 10 (depends on 9) The method of claim 9, wherein causing encoding of the received video frame to be skipped includes incrementing a counter of skipped frames, and wherein performing video encoding rate operations and adjusting of the encoding bit rate of the hardware video encoder includes: determining if the value of the skipped frame counter is above a second threshold; and decreasing the encoding bit rate of the hardware video encoder when the skipped frame counter value is above the second threshold. ### Claim 11 (depends on 10) The method of claim 10, wherein performing video encoding rate operations and adjusting of the encoding bit rate of the hardware video encoder includes: determining if the bits flowing into the outgoing video buffer are greater than the bits flowing out of the outgoing video buffer multiplied by a high rate factor; decreasing the encoding bit rate of the hardware video encoder when the bits flowing into the outgoing video buffer are greater than the bits flowing out of the outgoing video buffer multiplied by a high rate factor; determining if the bits flowing into the outgoing video buffer multiplied by a low rate factor are less than the bits flowing out of the outgoing video buffer; and increasing the encoding bit rate of the hardware video encoder when the bits flowing into the outgoing video buffer multiplied by a low rate factor are less than the bits flowing out of the outgoing video buffer. ### Claim 12 (depends on 9) The method of claim 9, wherein performing video encoding rate operations and adjusting of the encoding bit rate of the hardware video encoder includes: determining if the bits flowing into the outgoing video buffer multiplied by a low rate factor are less than the bits flowing out of the outgoing video buffer; and increasing the encoding bit rate of the hardware video encoder when the bits flowing into the outgoing video buffer multiplied by a low rate factor are less than the bits flowing out of the outgoing video buffer. ### Claim 13 (depends on 12) The method of claim 12, wherein performing video encoding rate operations and adjusting of the encoding bit rate of the hardware video encoder includes: determining if the bits flowing into the outgoing video buffer are greater than the bits flowing out of the outgoing video buffer multiplied by a high rate factor; and decreasing the encoding bit rate of the hardware video encoder when the bits flowing into the outgoing video buffer are greater than the bits flowing out of the outgoing video buffer multiplied by a high rate factor. ### Claim 14 (depends on 9) The method of claim 9, wherein performing video encoding rate operations and adjusting of the encoding bit rate of the hardware video encoder includes: determining if the bits flowing into the outgoing video buffer are greater than the bits flowing out of the outgoing video buffer multiplied by a high rate factor; and decreasing the encoding bit rate of the hardware video encoder when the bits flowing into the outgoing video buffer are greater than the bits flowing out of the outgoing video buffer multiplied by a high rate factor. ### Claim 15 A non-transitory memory coupled to a processor and storing instructions to cause the processor to perform a method comprising the steps of: receiving a video frame to be encoded; determining a fullness level value of an outgoing video buffer; causing encoding of the received video frame to be skipped when the buffer fullness level value is above a threshold; and periodically performing video encoding rate operations and adjusting of an encoding bit rate of a hardware video encoder if appropriate. ### Claim 16 (depends on 15) The non-transitory memory of claim 15, wherein causing encoding of the received video frame to be skipped includes incrementing a counter of skipped frames, and wherein performing video encoding rate operations and adjusting of the encoding bit rate of the hardware video encoder includes: determining if the value of the skipped frame counter is above a second threshold; and decreasing the encoding bit rate of the hardware video encoder when the skipped frame counter value is above the second threshold. ### Claim 17 (depends on 16) The non-transitory memory of claim 16, wherein performing video encoding rate operations and adjusting of the encoding bit rate of the hardware video encoder includes: determining if the bits flowing into the outgoing video buffer are greater than the bits flowing out of the outgoing video buffer multiplied by a high rate factor; decreasing the encoding bit rate of the hardware video encoder when the bits flowing into the outgoing video buffer are greater than the bits flowing out of the outgoing video buffer multiplied by a high rate factor; determining if the bits flowing into the outgoing video buffer multiplied by a low rate factor are less than the bits flowing out of the outgoing video buffer; and increasing the encoding bit rate of the hardware video encoder when the bits flowing into the outgoing video buffer multiplied by a low rate factor are less than the bits flowing out of the outgoing video buffer. ### Claim 18 (depends on 15) The non-transitory memory of claim 15, wherein performing video encoding rate operations and adjusting of the encoding bit rate of the hardware video encoder includes: determining if the bits flowing into the outgoing video buffer multiplied by a low rate factor are less than the bits flowing out of the outgoing video buffer; and increasing the encoding bit rate of the hardware video encoder when the bits flowing into the outgoing video buffer multiplied by a low rate factor are less than the bits flowing out of the outgoing video buffer. ### Claim 19 (depends on 18) The non-transitory memory of claim 18, wherein performing video encoding rate operations and adjusting of the encoding bit rate of the hardware video encoder includes: determining if the bits flowing into the outgoing video buffer are greater than the bits flowing out of the outgoing video buffer multiplied by a high rate factor; and decreasing the encoding bit rate of the hardware video encoder when the bits flowing into the outgoing video buffer are greater than the bits flowing out of the outgoing video buffer multiplied by a high rate factor. ### Claim 20 (depends on 15) The non-transitory memory of claim 15, wherein performing video encoding rate operations and adjusting of the encoding bit rate of the hardware video encoder includes: determining if the bits flowing into the outgoing video buffer are greater than the bits flowing out of the outgoing video buffer multiplied by a high rate factor; and decreasing the encoding bit rate of the hardware video encoder when the bits flowing into the outgoing video buffer are greater than the bits flowing out of the outgoing video buffer multiplied by a high rate factor.