# US 2013/0156098 A1 · Method and apparatus for dual pass rate control video encoding > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2013/0156098 A1 - Document ID: US20130156098A1 - Status: Published application - Kind: A1 - Application number: 13/330,137 - Filed: 2011-12-19 - Published: 2013-06-20 - Earliest claimed: 2011-12-19 - Assignees: General Instrument Corporation - Inventors: Mayer D. Schwartz; Peter A. Borgwardt; Shiv Saxena; Brenda L. Van Veldhuisen - Web page: https://patentgrep.com/patent/US20130156098A1 ## CPC - H04N 19/124: Quantisation - H04N 19/114: Adapting the group of pictures [GOP] structure, e.g. number of B-frames between two anchor frames (H04N19/107 takes precedence) - H04N 19/115: Selection of the code volume for a coding unit prior to coding - H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635) - H04N 19/142: Detection of scene cut or scene change - H04N 19/146: Data rate or code amount at the encoder output - H04N 19/15: by monitoring actual compressed data size at the memory before deciding storage at the transmission buffer - H04N 19/152: by measuring the fullness of the transmission buffer - H04N 19/172: the region being a picture, frame or field - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/186: the unit being a colour or a chrominance component - H04N 19/194: involving only two passes - H04N 19/103: Selection of coding mode or of prediction mode - H04N 19/107: between spatial and temporal predictive coding, e.g. picture refresh - H04N 19/179: the unit being a scene or a shot - H04N 19/184: the unit being bits, e.g. of the compressed video stream - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method of dual-pass rate control video encoding and an encoder are disclosed. Uncompressed input video is received with a video encoder. First pass encoding is performed, and a second pass encoding is performed with the video encoder at a delay relative to the first pass. The first pass encoding detects scene changes within the input video, assigns macroblocks to a pre-determined number of activity classes, and determines a complexity of each picture within the sequence of pictures. Information from the first pass encoding concerning the scene changes, the activity classes, and the complexity are used for rate control determined in second pass encoding for producing and outputting a bit stream of compressed video. During at least one of the first pass and second pass encoding, boundaries of at least some of the GOPs are adjusted to be aligned with the scene changes. ## Claims ### Claim 1 A method of dual-pass rate control video encoding, comprising the steps of: receiving uncompressed input video including a sequence of pictures with a video encoder, the input video including a plurality of contiguous sequences of pictures each referred to as a Group of Pictures (GOP) that, when coded, begin with an I-picture in display order and end with a last picture before a next I-picture; performing a first encoding pass of the uncompressed video with the video encoder; and performing a second encoding pass for rate control of the uncompressed video with the video encoder at a delay relative to the first encoding pass; the first encoding pass detecting scene changes within the input video, assigning macroblocks to a pre-determined number of activity classes, and determining a complexity of each picture within the sequence of pictures, wherein macroblocks refer to rectangular arrays of luma samples and two corresponding blocks of chroma samples in the pictures, activity class refers to different defined classes of macroblocks based on luma values of the pictures of the input video, and complexity is a number of bits generated per picture and activity class; information from the first encoding pass concerning the scene changes, the activity classes, and the complexity being used for rate control determined in the second encoding pass for producing and outputting a bit stream of compressed video; and during at least one of the first encoding pass and the second encoding pass, the boundaries of at least some of the GOPs are adjusted to be aligned with the scene changes. ### Claim 2 (depends on 1) A method according to claim 1, wherein, during the first encoding pass, macroblock activity values are calculated for each picture and activity class and are equal to a sum of luma values of macroblocks within a picture and a given activity class divided by the total number of macroblocks within the picture and the given activity class, and wherein said detecting scene changes step is based on a comparison of an average of the macroblock activity values of a set of prior pictures relative to an average of the macroblock activity values of a next set of pictures in display order. ### Claim 3 (depends on 2) A method according to claim 2, wherein the first encoding pass generates the following data for use during the second encoding pass: picture type, scene change flag, header bits, generated bits per activity class, number of macroblocks per activity class, and average macroblock activity per activity class. ### Claim 4 (depends on 1) A method according to claim 1, wherein quantizers for rate control are determined by the second encoding pass for each macroblock on a per picture type and activity class basis, wherein an average complexity for a given picture type and activity class is determined for each of the GOPs, and wherein ratios between the average complexities of GOPs are used to determine bit allocation and quantizer selection for rate control during the second encoding pass. ### Claim 5 (depends on 1) A method according to claim 1, wherein, during the first encoding pass, every picture of a given picture type is encoded with a constant quantizer (QP) for all macroblocks of the given picture type and wherein the complexity for each picture is stored in a sliding window look-ahead buffer used during said step of performing the second encoding pass. ### Claim 6 (depends on 1) A method according to claim 1, wherein each GOP includes a plurality of sub-GOPs, each sub-GOP being a contiguous sequence of coded pictures that begin, in display order, with an I-picture or a P-picture and ends with a last picture before a next I-picture or P-picture; and wherein, during the first encoding pass, a first picture at which a new scene starts and each sub-GOP containing a first picture of a new scene are identified. ### Claim 7 (depends on 6) A method according to claim 6, wherein said step of performing the first encoding pass includes promoting a P-picture to an I-picture for every sub-GOP containing the first picture of a new scene. ### Claim 8 (depends on 1) A method according to claim 1, wherein the video encoder is selected from a group selected from a constant bit rate (CBR) encoder and a statistical multiplexer (stat-mux) encoder. ### Claim 9 (depends on 1) A method according to claim 1, wherein a target encoder buffer fullness is calculated during the second encoding pass at the beginning of each GOP and at every intermediate picture within a GOP. ### Claim 10 (depends on 1) A method according to claim 1, wherein during the second encoding pass, a size of encoded bits generated for a picture at a given quantizer (QP) and of a given activity class is predicted by applying a content-dependent multiplication factor derived from a set of estimation filters. ### Claim 11 (depends on 1) A method according to claim 1, wherein if, after encoding by the second encoding pass, the bits for a given picture would cause an encoder buffer to overflow, encoded pictures of minimal known size are substituted in the encoder buffer before outputting of the bit stream of compressed video from the video encoder. ### Claim 12 A method of dual-pass rate control video encoding, comprising the steps of: receiving uncompressed input video including a sequence of pictures with a video encoder, the input video including a plurality of contiguous sequences of pictures each referred to as a Group of Pictures (GOP) that, when coded, begin with an I-picture in display order and end with a last picture before a next I-picture, the GOP also including picture types encoded as P pictures, B pictures and B-reference pictures; performing a first encoding pass of the uncompressed video with the video encoder to detect scene changes, to assign macroblocks to a pre-determined number of activity classes, and to determine a complexity of each picture within the sequence of pictures, wherein macroblocks refer to rectangular arrays of luma samples and two corresponding blocks of chroma samples in the pictures, activity class refers to different defined classes of macroblocks based on luma values of the pictures of the input video, and complexity is a number of bits generated per picture and activity class; performing a second encoding pass for rate control of the uncompressed video with the video encoder at a delay relative to the first encoding pass wherein information from the first encoding pass concerning the scene changes, the activity classes, and the complexity are used for rate control determined in the second encoding pass for producing and outputting a bit stream of compressed video and wherein quantizer values for rate control are determined by the second encoding pass for each macroblock on a per picture type and activity class basis; and during the second encoding pass, encoding each macroblock determined to require a maximum quantizer value by the video encoder as a skip picture if encoding the macroblock with a maximum quantizer value would cause an overflow of the coded picture buffer (CPB) of the video encoder. ### Claim 13 (depends on 12) A method according to claim 12, furthering comprising the steps of: calculating estimated bits for each macroblock determined to require a maximum quantizer value by the video encoder; and estimating buffer fullness of the CPB as a result of encoding the macroblock with the maximum quantizer value. ### Claim 14 (depends on 12) A method according to claim 12, wherein said step of encoding each macroblock determined to require a maximum quantizer value by the video encoder as a skip picture is performed if an estimated butter fullness after encoding the macroblock is greater than 90% of a size of the CPB. ### Claim 15 (depends on 12) A method according to claim 12, wherein, if the macroblock encoded as a skip picture is a P picture, all subsequent B-reference pictures and B pictures that reference the P picture are also encoded as skip pictures and wherein, if the macroblock encoded as a skip picture is a B-reference picture, all subsequent B pictures that reference the B-reference picture are also encoded as skip pictures. ### Claim 16 A dual-pass rate control video encoder, comprising: a first pass encoder module for receiving uncompressed input video including a sequence of pictures, the input video including a plurality of contiguous sequences of pictures each referred to as a Group of Pictures (GOP) that, when coded, begin with an I-picture in display order and end with a last picture before a next I-picture, the first pass encoder module detecting scene changes within the input video, assigning macroblocks to a pre-determined number of activity classes, and determining a complexity of each picture within the sequence of pictures, wherein macroblocks refer to rectangular arrays of luma samples and two corresponding blocks of chroma samples in the pictures, activity class refers to different defined classes of macroblocks based on luma values of the pictures of the input video, and complexity is a number of bits generated per picture and activity class; a delay unit for receiving the uncompressed input video; and a second pass encoder module for receiving the uncompressed input video via said delay unit, for obtaining information from the first pass encoder module concerning the scene changes, the activity classes, and the complexity used to determine rate control of the uncompressed video, and for producing and outputting a bit stream of compressed video; at least one of said first pass encoder module and said second pass encoder module adjusting the boundaries of at least some of the GOPs to be aligned with the scene changes. ### Claim 17 (depends on 16) A dual-pass rate control video encoder according to claim 16, wherein the video encoder is selected from a group selected from a constant bit rate (CBR) encoder and a statistical multiplexer (stat-mux) encoder, wherein said first pass encoder module determines macroblock activity values for each picture and activity class, where macroblock activity values are equal to a sum of luma values of macroblocks within a picture and a given activity class divided by the total number of macroblocks within the picture and the given activity class, and wherein scene changes are identified by the first pass encoder module based on a comparison of an average of the macroblock activity values of a set of prior pictures relative to an average of the macroblock activity values of a next set of pictures in display order. ### Claim 18 (depends on 16) A dual-pass rate control video encoder according to claim 16, wherein the first pass encoder module generates the following data for use by the second pass encoder module: picture type, scene change flag, header bits, generated bits per activity class, number of macroblocks per activity class, and average macroblock activity per activity class, wherein quantizers for rate control are determined by the second pass encoder module for each macroblock on a per picture type and activity class basis, wherein an average complexity for a given picture type and activity class is determined for each of the GOPs by the second pass encoder module, and wherein ratios between the average complexities of GOPs are used to determine bit allocation and quantizer selection for rate control by the second pass encoder module. ### Claim 19 (depends on 16) A dual-pass rate control video encoder according to claim 16, wherein said first pass encoder module codes all macroblocks of a given picture type with a constant quantizer (QP), and wherein the complexity for each picture determined by the first pass encoder module is stored in a sliding window look-ahead buffer of the second pass encoder module. ### Claim 20 (depends on 16) A dual-pass rate control video encoder according to claim 16, wherein each GOP includes a plurality of sub-GOPs, each sub-GOP being a contiguous sequence of coded pictures that begin, in display order, with an I-picture or a P-picture and ends with a last picture before a next I-picture or P-picture; wherein the first pass encoder module identifies a first picture at which a new scene starts and each sub-GOP containing a first picture of a new scene; and wherein said first pass encoder module promotes a P-picture to an I-picture for every sub-GOP containing the first picture of a new scene.