# US 12,120,312 B2 · Video encoding rate control for intra and scene change frames using machine learning > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,120,312 B2 - Document ID: US12120312B2 - Status: Granted - Kind: B2 - Application number: 16/950,367 - Filed: 2020-11-17 - Published: 2021-03-04 - Granted: 2024-10-15 - Earliest claimed: 2020-11-17 - Assignees: Intel Corporation - Applicants: Intel Corporation - Inventors: Ximin Zhang; Sang-hee Lee; Keith W. Rowe - Web page: https://patentgrep.com/patent/US12120312B2 ## CPC - H04N 19/124: Quantisation - H04N 19/159: Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction - G06N 3/0499: Feedforward networks - G06N 3/08: Learning methods - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - H04N 19/146: Data rate or code amount at the encoder output - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/179: the unit being a scene or a shot - H04N 19/70: characterised by syntax aspects related to video coding, e.g. related to compression standards - H04N 19/30: using hierarchical techniques, e.g. scalability (H04N19/63 takes precedence) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Techniques related to quantization parameter estimation for coding intra and scene change frames are discussed. Such techniques include generating features based on an intra or scene change frame including a proportion of smooth blocks and one or both of a measure of block variance and a prediction distortion, and applying a machine learning model to generate an estimated quantization parameter for encoding the intra or scene change frame. ## Claims ### Claim 1 A system for video coding comprising: a memory to store at least a portion of an intra or scene change frame of input video; and one or more processors coupled to the memory, the one or more processors to: generate a plurality of features based on the intra or scene change frame, the features comprising a proportion of smooth blocks corresponding to the intra or scene change frame and at least one of a measure of block variance of blocks corresponding to the intra or scene change frame or a prediction distortion corresponding to the intra or scene change frame; apply a machine learning model to a first input comprising the features and a target bitrate for the intra or scene change frame to generate a first estimated quantization parameter for the intra or scene change frame; encode the intra or scene change frame the first estimated quantization parameter to generate a first bitstream having a number of bits; determine whether a difference between the target bitrate and the number of bits of the first bitstream exceeds a threshold; in response to determining the difference exceeds the threshold, apply the machine learning model to a second input comprising the features and the number of bits of the first bitstream to generate a second estimated quantization parameter for the intra or scene change frame; and encode the intra or scene change frame using an encode quantization parameter to generate a second bitstream, wherein the encode quantization parameter is a function of the first estimated quantization parameter and the second estimated quantization parameter. ### Claim 2 (depends on 1) The system of claim 1, wherein the features comprise the proportion of smooth blocks, the measure of block variance, and the prediction distortion. ### Claim 3 (depends on 2) The system of claim 2, wherein the proportion of smooth blocks comprises a percentage of the blocks having a block variance that does not exceed a threshold, the measure of block variance comprises an average of the block variances of the blocks, and the predicted distortion is based on a difference between pixels corresponding to the intra or scene change frame and reference pixels corresponding to the intra or scene change frame. ### Claim 4 (depends on 2) The system of claim 2, wherein the features comprise a number of bits generated by a look ahead encode of the intra or scene change frame and a proportion of syntax bits of the number of bits from the look ahead encode. ### Claim 5 (depends on 1) The system of claim 1, wherein the threshold has a value between 3% to 5% of the target bitrate. ### Claim 6 (depends on 5) The system of claim 5, wherein the encode quantization parameter comprises a linear combination of the first estimated quantization parameter and a difference between the first estimated quantization parameter and the second estimated quantization parameter. ### Claim 7 (depends on 6) The system of claim 6, wherein the linear combination is a difference between the first estimated quantization parameter and a factor multiplied by the difference between the first estimated quantization parameter and the second estimated quantization parameter. ### Claim 8 (depends on 5) The system of claim 5, the one or more processors further to: update parameters of the machine learning model via on the fly training using the features and the encode quantization parameter as a training input-output pair. ### Claim 9 (depends on 1) The system of claim 1, wherein the machine learning model is pretrained using a training set comprising a plurality of feature vectors generated using corresponding training intra or scene change frames and a plurality of input frame sizes and corresponding ground truth quantization parameters for each of the training intra or scene change frames generated by encoding each training intra or scene change frame using a plurality of available quantization parameters. ### Claim 10 (depends on 1) The system of claim 1, wherein the machine learning model comprises a neural network comprising an input layer, a hidden layer, and an output layer. ### Claim 11 A method for video encoding comprising: generating a plurality of features based on an intra or scene change frame of input video, the features comprising a proportion of smooth blocks corresponding to the intra or scene change frame and at least one of a measure of block variance of blocks corresponding to the intra or scene change frame or a prediction distortion corresponding to the intra or scene change frame; applying a machine learning model to a first input comprising the features and a target bitrate for the intra or scene change frame to generate a first estimated quantization parameter for the intra or scene change frame; encoding the intra or scene change frame using the first estimated quantization parameter to generate a first bitstream having a number of bits; determining whether a difference between the target bitrate and the number of bits of the first bitstream exceeds a threshold; in response to determining the difference exceeds the threshold, applying the machine learning model to a second input comprising the features and the number of bits of the first bitstream to generate a second estimated quantization parameter for the intra or scene change frame; and encoding the intra or scene change frame using an encode quantization parameter to generate a second bitstream, wherein the encode quantization parameter is a function of the first estimated quantization parameter and the second estimated quantization parameter. ### Claim 12 (depends on 11) The method of claim 11, wherein the features comprise the proportion of smooth blocks, the measure of block variance, and the prediction distortion. ### Claim 13 (depends on 12) The method of claim 12, wherein the features comprise a number of bits generated by a look ahead encode of the intra or scene change frame and a proportion of syntax bits of the number of bits from the look ahead encode. ### Claim 14 (depends on 11) The method of claim 11, wherein the function computes a difference between the first estimated quantization parameter and a factor multiplied by the difference between the first estimated quantization parameter and the second estimated quantization parameter. ### Claim 15 (depends on 14) The method of claim 14, further comprising: updating parameters of the machine learning model via on the fly training using the features and the encode quantization parameter as a training input-output pair. ### Claim 16 At least one non-transitory machine readable medium comprising a plurality of instructions that, in response to being executed on a computing device, cause the computing device to perform video coding by: generating a plurality of features based on an intra or scene change frame of input video, the features comprising a proportion of smooth blocks corresponding to the intra or scene change frame and at least one of a measure of block variance of blocks corresponding to the intra or scene change frame or a prediction distortion corresponding to the intra or scene change frame; applying a machine learning model to a first input comprising the features and a target bitrate for the intra or scene change frame to generate a first estimated quantization parameter for the intra or scene change frame; encoding the intra or scene change frame using the first estimated quantization parameter to generate a first bitstream having a number of bits; determining whether a difference between the target bitrate and the number of bits of the first bitstream exceeds a threshold; in response to determining the difference exceeds the threshold, applying the machine learning model to a second input comprising the features and the number of bits of the first bitstream to generate a second estimated quantization parameter for the intra or scene change frame; and encoding the intra or scene change frame using an encode quantization parameter to generate a second bitstream, wherein the encode quantization parameter is a function of the first estimated quantization parameter and the second estimated quantization parameter. ### Claim 17 (depends on 16) The at least one non-transitory machine readable medium of claim 16, wherein the features comprise the proportion of smooth blocks, the measure of block variance, and the prediction distortion. ### Claim 18 (depends on 17) The at least one non-transitory machine readable medium of claim 17, wherein the features comprise a number of bits generated by a look ahead encode of the intra or scene change frame and a proportion of syntax bits of the number of bits from the look ahead encode. ### Claim 19 (depends on 16) The at least one non-transitory machine readable medium of claim 16, wherein the encode quantization parameter is a difference between the first estimated quantization parameter and a factor multiplied by a difference between the first estimated quantization parameter and the second estimated quantization parameter, and the factor is between 0.6 and 0.8. ### Claim 20 (depends on 19) The at least one non-transitory machine readable medium of claim 19, further comprising instructions that, in response to being executed on the computing device, cause the computing device to perform video coding by: updating parameters of the machine learning model via on the fly training using the features and the encode quantization parameter as a training input-output pair.