# US 9,521,413 B2 · Optimized filter selection for reference picture processing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,521,413 B2 - Document ID: US9521413B2 - Status: Granted - Kind: B2 - Application number: 15/004,582 - Filed: 2016-01-22 - Published: 2016-05-19 - Granted: 2016-12-13 - Earliest claimed: 2010-10-01 - Assignees: Dolby Laboratories Licensing Corporation - Applicants: Dolby Laboratories Licensing Corporation - Inventors: Peshala V. Pahalawatta; Yuwen He; Alexandros Tourapis - Web page: https://patentgrep.com/patent/US9521413B2 ## CPC - H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635) - H04N 5/21: Circuitry for suppressing or minimising disturbance, e.g. moiré or halo - H04N 19/147: according to rate distortion criteria (rate-distortion as a criterion for motion estimation H04N19/567) - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/39: involving multiple description coding [MDC], i.e. with separate layers being structured as independently decodable descriptions of input picture data - H04N 19/52: by predictive encoding - H04N 19/597: specially adapted for multi-view video sequence encoding - H04N 19/82: involving filtering within a prediction loop - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Reference processing may be used in a video encoder or decoder to derive reference pictures that are better correlated with a source image to be encoded or decoded, which generally yields better coding efficiency. Methods for filter selection for a reference processing unit adapted for use in a video codec system are discussed. Specifically, methods for filter selection based on performing motion estimation and obtaining distortion/cost information by comparing reference pictures, either processed or non-processed, with the source image to be encoded are discussed. ## Claims ### Claim 1 A method for selecting a particular filter from among a plurality of filters, the particular filter adapted for use in a reference picture processing unit in a multi-layered video coding system, the multi-layered video coding system comprising a base layer and one or more enhancement layers, the method comprising: a) providing (i) an enhancement layer source picture, and (ii) a reference picture from the base layer or an inter-layer, the inter-layer being a layer from among the one or more enhancement layers that is processed prior to the enhancement layer of the enhancement layer source picture; b) dividing the reference picture into regions, wherein the region sizes and the region shapes are determined based on performing edge detection on the reference picture, and wherein each of the following steps is performed on the regions of the reference picture: c) filtering copies of the reference picture using at least one filter from the plurality of filters to obtain at least one filtered reference picture, wherein each filter is applied to a corresponding copy of the reference picture; d) performing disparity estimation based on the enhancement layer source picture, a full set or subset of the at least one filtered reference picture, temporal reference pictures from an enhancement layer reference picture buffer, and motion information from the base layer or an inter-layer, the inter-layer being a layer from among the one or more enhancement layers that is processed prior to the enhancement layer of the enhancement layer source picture, and wherein the disparity estimation is adapted to generate disparity information; e) selecting a set of temporal reference pictures from the enhancement layer reference picture buffer; f) obtaining distortion information based on the disparity information and the selected set of temporal reference pictures; and g) selecting the particular filter based on comparing the distortion information generated in step f); wherein: the disparity information comprises a function of at least one of motion vectors, prediction distortion, intra prediction information, illumination parameters, luma components, chroma components, and tone mapping parameters, motion information of the enhancement layer is based on the motion information from the base layer or the inter-layer, differences between the motion information of the enhancement layer and the motion information from the base layer or the inter-layer are used to compute a confidence level, the distortion information is based on the differences between the motion information of the enhancement layer and the motion information from the base layer or the inter-layer, and the distortion information is weighted by the confidence level. ### Claim 2 (depends on 1) The method as recited in claim 1, wherein the step of providing further comprises processing the enhancement layer source picture, wherein the processing involves applying at least one of filtering, motion transformation, motion compensation, illumination compensation, scaling, inverse and forward tone mapping, color format conversion, and gamma correction. ### Claim 3 (depends on 1) The method as recited in claim 1, wherein the step of performing disparity estimation further comprises obtaining cost information for the at least one filter from among the plurality of filters used in the step of filtering, and wherein the step of selecting the particular filter is further based on the cost information; and wherein the cost information is a function of the disparity information, distortion between the enhancement layer source picture and each filtered reference picture, number of bits to be used in signaling filter parameters of each filter, number of bits to be used in signaling the motion vectors corresponding to each filtered reference picture, number of bits to be used in signaling the prediction distortion corresponding to each filtered reference picture, computational complexity in applying each filter, and power consumption of the filter. ### Claim 4 (depends on 1) The method as recited in claim 1, further comprising, between the step of disparity estimation and the step of obtaining distortion information, a step of performing disparity compensation on the enhancement layer source picture to obtain a final reconstructed picture, wherein the disparity compensation is based on the step of disparity estimation. ### Claim 5 (depends on 4) The method as recited in claim 4, wherein the obtaining distortion information is further based on the final reconstructed picture. ### Claim 6 (depends on 1) The method as recited in claim 1, wherein the disparity estimation comprises block-based motion estimation. ### Claim 7 (depends on 1) The method as recited in claim 1, wherein the step of selecting temporal reference pictures is based on a respective time difference between the temporal reference pictures and the enhancement layer source picture. ### Claim 8 (depends on 7) The method as recited in claim 7, wherein the step of selecting temporal reference pictures is based on a respective correlation between each of the temporal reference pictures and the enhancement layer source picture. ### Claim 9 (depends on 1) The method as recited in claim 1, further comprising a step of prioritizing importance of reference pictures, from high importance to low importance, based on comparing the cost information and the distortion information between the reference pictures to obtain a prioritized set of reference pictures. ### Claim 10 (depends on 9) The method as recited in claim 9, wherein fewer bits are used for signaling pictures in the prioritized set of reference pictures of higher importance than for pictures in the prioritized set of reference pictures of lower importance. ### Claim 11 (depends on 1) The method as recited in claim 1, wherein sub-pixel accurate motion estimation is performed on a subset of the at least one filtered reference picture and a subset of the temporal reference pictures while integer pixel motion estimation is performed on remaining filtered reference pictures and remaining temporal reference pictures. ### Claim 12 (depends on 9) The method as recited in claim 9, wherein reference pictures of importance below a pre-defined threshold are not used in the step of obtaining distortion information. ### Claim 13 (depends on 9) The method as recited in claim 9, wherein the disparity estimation is performed at higher precision for reference pictures of higher importance and the disparity estimation is performed at lower precision for reference pictures of lower importance. ### Claim 14 A system for selecting a particular filter from among a plurality of filters, the particular filter adapted for use in a reference picture processing unit in a multi-layered video coding system, the multi-layered video coding system comprising a base layer and one or more enhancement layers, the system comprising: a) means for providing (i) an enhancement layer source picture, and (ii) a reference picture from the base layer or an inter-layer, the inter-layer being a layer from among the one or more enhancement layers that is processed prior to the enhancement layer of the enhancement layer source picture; b) means for dividing the reference picture into regions, wherein the region sizes and the region shapes are determined based on performing edge detection on the reference picture, and wherein each of the following means is configured to perform the respective action on the regions of the reference picture: c) means for filtering copies of the reference picture using at least one filter from the plurality of filters to obtain at least one filtered reference picture, wherein each filter is applied to a corresponding copy of the reference picture; d) means for performing disparity estimation based on the enhancement layer source picture, a full set or subset of the at least one filtered reference picture, temporal reference pictures from an enhancement layer reference picture buffer, and motion information from the base layer or an inter-layer, the inter-layer being a layer from among the one or more enhancement layers that is processed prior to the enhancement layer of the enhancement layer source picture, and wherein the disparity estimation is adapted to generate disparity information; e) means for selecting a set of temporal reference pictures from the enhancement layer reference picture buffer; f) means for obtaining distortion information based on the disparity information and the selected set of temporal reference pictures; and g) means for selecting the particular filter based on comparing the distortion information generated with the distortion information obtaining means f) and based on the divided regions of each filtered reference picture; wherein: the disparity information comprises a function of at least one of motion vectors, prediction distortion, intra prediction information, illumination parameters, luma components, chroma components, and tone mapping parameters, motion information of the enhancement layer is based on the motion information from the base layer or the inter-layer, differences between the motion information of the enhancement layer and the motion information from the base layer or the inter-layer are used to compute a confidence level, the distortion information is based on the differences between the motion information of the enhancement layer and the motion information from the base layer or the inter-layer, and the distortion information is weighted by the confidence level. ### Claim 15 (depends on 1) A non-transitory computer readable storage medium, comprising instructions that are tangibly encoded therewith, which when executed with a processor, cause the processor to cause, control, program or configure, at least in part, a process for selecting a particular filter from among a plurality of filters as recited in claim 1.