# US 9,521,411 B2 · Method and apparatus for encoder assisted-frame rate up conversion (EA-FRUC) for video compression > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,521,411 B2 - Document ID: US9521411B2 - Status: Granted - Kind: B2 - Application number: 13/760,991 - Filed: 2013-02-06 - Published: 2013-07-25 - Granted: 2016-12-13 - Earliest claimed: 2004-07-20 - Assignees: QUALCOMM Incorporated - Applicants: QUALCOMM Incorporated - Inventors: Vijayalakshmi R. Raveendran; Fang Shi; Gordon Kent Walker - Web page: https://patentgrep.com/patent/US9521411B2 ## CPC - H04N 19/85: using pre-processing or post-processing specially adapted for video compression - H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking - H04N 19/137: Motion inside a coding unit, e.g. average field, frame or block difference - H04N 19/139: Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability - H04N 19/14: Coding unit complexity, e.g. amount of activity or edge presence estimation (H04N19/146 takes precedence) - H04N 19/142: Detection of scene cut or scene change - H04N 19/15: by monitoring actual compressed data size at the memory before deciding storage at the transmission buffer - H04N 19/172: the region being a picture, frame or field - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/194: involving only two passes - H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) - H04N 19/583: Motion compensation with overlapping blocks - H04N 19/587: involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence - H04N 19/61: in combination with predictive coding - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An Encoder Assisted Frame Rate Up Conversion (EA-FRUC) system that utilizes video coding and pre-processing operations at the video encoder to exploit the FRUC processing that will occur in the decoder in order to improve compression efficiency and reconstructed video quality is disclosed. One operation of the EA-FRUC system involves determining whether to encode a frame in a sequence of frames of a video content by determining a spatial activity in a frame of the sequence of frames; determining a temporal activity in the frame; determining a spatio-temporal activity in the frame based on the determined spatial activity and the determined temporal activity; determining a level of a redundancy in the source frame based on at least one of the determined spatial activity, the determined temporal activity, and the determined spatio-temporal activity; and, encoding the non-redundant information in the frame if the determined redundancy is within predetermined thresholds. ## Claims ### Claim 1 A method for processing a video bitstream, the method comprising: using an entropy coding technique to code differential information in the video bitstream selected from a group consisting of a variable length coding technique, a Huffman coding technique, and an arithmetic coding technique, wherein the differential information comprises differential information with respect to a predicted frame coding and an interpolated frame generated by a frame rate up conversion process; and conveying the coded information in a user data syntax specified in a video coding standard. ### Claim 2 (depends on 1) The method of claim 1, further comprising: generating a standards compliant video bitstream which includes the coded information. ### Claim 3 (depends on 1) The method of claim 1, further comprising: using the differential information of the coded information to assist a decoder with frame rate up conversion process. ### Claim 4 (depends on 3) The method of claim 3, wherein the decoder assistance provided by the differential information of the coded information is to enhance a quality of reconstructed video. ### Claim 5 (depends on 3) The method of claim 3, wherein the decoder assistance provided by the differential information of the coded information is to reduce a computational load on the decoder. ### Claim 6 (depends on 5) The method of claim 5, wherein the computational load on the decoder is reduced by the coded information aiding a frame rate up conversion engine of the decoder in making decisions during a mode decision process. ### Claim 7 (depends on 1) The method of claim 1, further comprising: determining the differential information between two frames by determining a difference measure selected from a group consisting of a pixel differences measure, a motion information differences measure, a mode decision threshold measure and an interpolated frame refinement measure, wherein the determination of the difference measure is performed using a frame rate up conversion process. ### Claim 8 (depends on 1) The method of claim 1, wherein the entropy coding process is specified in a video coding standard, and wherein a standards-compliant processor is able to process the differential information in conjunction with a frame rate up conversion process to generate a video frame. ### Claim 9 (depends on 1) The method of claim 1, further comprising: processing the video bitstream with differential information encoded therein by extracting the coded differential information from the user data syntax, decoding the differential information, and generating a video frame using the decoded differential information in a frame rate up conversion process. ### Claim 10 A non-transitory tangible computer readable medium having instructions stored thereon for causing a computer to execute a method for processing a video bitstream, the video bitstream having differential information contained therein, the method comprising: using an entropy coding technique to code differential information in the video bitstream selected from a group consisting of a variable length coding technique, a Huffman coding technique, and an arithmetic coding technique, wherein the differential information comprises differential information with respect to a predicted frame coding and an interpolated frame generated by a frame rate up conversion process; and conveying the coded information in a user data syntax specified in a video coding standard. ### Claim 11 (depends on 10) The computer readable medium of claim 10, further comprising: generating a standards compliant video bitstream. ### Claim 12 (depends on 10) The computer readable medium of claim 10, further comprising: using the differential information of the coded information to assist a decoder with the frame rate up conversion process. ### Claim 13 (depends on 12) The computer readable medium of claim 12, wherein the decoder assistance provided by the differential information of the coded information is to enhance a quality of reconstructed video. ### Claim 14 (depends on 12) The computer readable medium of claim 12, wherein the decoder assistance provided by the differential information of the coded information is to reduce a computational load on the decoder. ### Claim 15 (depends on 14) The computer readable medium of claim 14, wherein the computational load on the decoder is reduced by the coded information aiding a frame rate up conversion engine of the decoder in making decisions during a mode decision process. ### Claim 16 (depends on 10) The computer readable medium of claim 10, further comprising: determining the differential information between two frames by determining a difference measure selected from a group consisting of a pixel differences measure, a motion information differences measure, a mode decision threshold measure and an interpolated frame refinement measure, wherein the determination of the difference measure is performed using a frame rate up conversion process. ### Claim 17 (depends on 10) The computer readable medium of claim 10, wherein the entropy coding process is specified in a video coding standard, and wherein a standards-compliant processor is able to process the differential information in conjunction with a frame rate up conversion process to generate a video frame. ### Claim 18 (depends on 10) The computer readable medium of claim 10, further comprising: processing the video bitstream with differential information encoded therein by extracting the coded differential information from the user data syntax, decoding the differential information, and generating a video frame using the decoded differential information in a frame rate up conversion process. ### Claim 19 An apparatus for processing a video bitstream, the video bitstream having differential information contained therein, the apparatus comprising: means for entropy coding differential information in the video bitstream with a coding technique selected from a group consisting of a variable length coding technique, a Huffman coding technique, and an arithmetic coding technique, wherein the differential information comprises differential information with respect to a predicted frame coding and an interpolated frame generated by a frame rate up conversion process; and means for conveying the coded information in a user data syntax specified in a video coding standard. ### Claim 20 (depends on 19) The apparatus of claim 19, further comprising: means for generating a standards compliant video bitstream. ### Claim 21 (depends on 19) The apparatus of claim 19, further comprising: means for using the differential information of the coded information to assist a decoder with frame rate up conversion process. ### Claim 22 (depends on 21) The apparatus of claim 21, wherein the decoder assistance provided by the differential information of the coded information is to enhance a quality of reconstructed video. ### Claim 23 (depends on 21) The apparatus of claim 21, wherein the decoder assistance provided by the differential information of the coded information is to reduce a computational load on the decoder. ### Claim 24 (depends on 23) The apparatus of claim 23, wherein the computational load on the decoder is reduced by the coded information aiding a frame rate up conversion engine of the decoder in making decisions during a mode decision process. ### Claim 25 (depends on 19) The apparatus of claim 19, further comprising: means for determining the differential information between two frames by determining a difference measure selected from a group consisting of a pixel differences measure, a motion information differences measure, a mode decision threshold measure and an interpolated frame refinement measure, wherein the determination of the difference measure is performed using a frame rate up conversion process. ### Claim 26 (depends on 19) The apparatus of claim 19, wherein the entropy coding process is specified in a video coding standard, and wherein a standards-compliant processor is able to process the differential information in conjunction with a frame rate up conversion process to generate a video frame. ### Claim 27 (depends on 19) The apparatus of claim 19, further comprising: means for processing the video bitstream with differential information encoded therein by extracting the coded differential information from the user data syntax, decoding the differential information, and generating a video frame using the decoded differential information in a frame rate up conversion process.