# US 12,603,998 B2 · Configurable neural network model depth in neural network-based video coding > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,603,998 B2 - Document ID: US12603998B2 - Status: Granted - Kind: B2 - Application number: 17/848,068 - Filed: 2022-06-23 - Published: 2023-01-26 - Granted: 2026-04-14 - Earliest claimed: 2021-07-07 - Assignees: Lemon Inc. - Applicants: Lemon, Inc. - Inventors: Yue Li; Kai Zhang; Li Zhang - Web page: https://patentgrep.com/patent/US12603998B2 ## CPC - H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635) - H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction - H04N 19/174: the region being a slice, e.g. a line of blocks or a group of blocks - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/18: the unit being a set of transform coefficients - H04N 19/182: the unit being a pixel - H04N 19/61: in combination with predictive coding - H04N 19/80: Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation (H04N19/635, H04N19/86 take precedence) - H04N 19/82: involving filtering within a prediction loop - H04N 19/86: involving reduction of coding artifacts, e.g. of blockiness - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method implemented by a video coding apparatus includes applying a neural network (NN) filter to an unfiltered sample of a video unit to generate a filtered sample, where the NN filter is based on a first NN filter model having a first depth, or a second NN filter model having a second depth, where the depth comprises a number of residual blocks of the respective NN filter model, and where the second depth is different than the first depth. The method also includes performing a conversion between a video media file and a bitstream based on the filtered sample. ## Claims ### Claim 1 A method implemented by a video coding apparatus, comprising: applying a neural network (NN) filter to an unfiltered sample of a video unit to generate a filtered sample, wherein the NN filter is based on a first NN filter model having a first depth, or a second NN filter model having a second depth, wherein the depth comprises a number of residual blocks of the respective NN filter model, and wherein the second depth is different than the first depth; and performing a conversion between a video media file and a bitstream based on the filtered sample, wherein the NN filter uses the first NN filter model for a first video unit in a first temporal layer or a first type of slice, and wherein the NN filter uses the second NN filter model for a second video unit in a second temporal layer or a second type of slice, and wherein the NN filter uses the first NN filter model for a third video unit in a third temporal layer, wherein the first temporal layer and the third temporal layer are in a same subgroup of temporal layers, and wherein the second temporal layer is in a different subgroup of temporal layers. ### Claim 2 (depends on 1) The method of claim 1, wherein the video unit is a first video unit, wherein when the NN filter is applied to the first video unit, the NN filter uses the first NN filter model, and wherein when the NN filter is applied to a second video unit, the NN filter uses the second NN filter model. ### Claim 3 (depends on 1) The method of claim 1, further comprising determining whether to use the first or second NN filter model based on a temporal layer of the video unit, a type of a slice or a picture that contains the video unit, a list of reference pictures that are matched to the video unit, a color component of the video unit, a color format of the video unit, or combinations thereof. ### Claim 4 (depends on 1) The method of claim 1, wherein the NN filter uses the first NN filter model for a first video unit in a picture or a slice, and wherein NN filter uses the second NN filter model for a second video unit in the picture or the slice. ### Claim 5 (depends on 1) The method of claim 1, wherein the NN filter uses the first NN filter model for a first video unit in a first picture or a first slice, and wherein the NN filter uses the second NN filter model for a second video unit in a second picture or a second slice. ### Claim 6 (depends on 1) The method of claim 1, wherein determining whether to use the first or second NN filter model is based on whether the video unit is a coding tree unit (CTU), a coding tree block (CTB), a row of a CTU, a row of a CTB, a slice, a tile, a picture, a sequence, or a subpicture. ### Claim 7 (depends on 1) The method of claim 1, wherein the NN filter uses the first NN filter model for a first video unit in a first temporal layer, wherein the NN filter uses the second NN filter model for a second video unit in a second temporal layer, and wherein the first temporal layer is a higher temporal layer than the second temporal layer. ### Claim 8 (depends on 7) The method of claim 7, wherein the first depth is 8. ### Claim 9 (depends on 1) The method of claim 1, wherein the NN filter uses the first NN filter model for a first video unit in an inter slice, and wherein the NN filter uses the second NN filter model for a second video unit in an intra slice. ### Claim 10 (depends on 9) The method of claim 9, wherein the first depth is 8 and wherein the second depth is 32. ### Claim 11 (depends on 1) The method of claim 1, wherein the depth of a NN filter model to be used is sent to the decoder using one selected from at least one of a supplemental enhancement information (SEI) message, a sequence parameter set (SPS), a picture parameter set (PPS), an adaptation parameter set (APS), a video parameter set (VPS), a picture header, or a slice header. ### Claim 12 (depends on 1) The method of claim 1, wherein the depth of the first NN filter model is sent to the decoder for a first video unit, and the depth of the second NN filter model is sent to the decoder for a second video unit. ### Claim 13 (depends on 1) The method of claim 1, wherein the unfiltered sample comprises a luma component and a chroma component, and wherein a different depth is signaled to a decoder for the luma component than for the chroma component. ### Claim 14 (depends on 1) The method of claim 1, wherein the NN filter is one or more selected from at least one of an adaptive loop filter, a deblocking filter, or a sample adaptive offset filter. ### Claim 15 (depends on 1) The method of claim 1, wherein the conversion comprises generating the bitstream according to the video media file. ### Claim 16 (depends on 1) The method of claim 1, wherein the conversion comprises parsing the bitstream to obtain the video media file. ### Claim 17 (depends on 1) The method of claim 1, wherein the depth of a NN filter model to be used is signaled to a decoder. ### Claim 18 An apparatus for coding video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor cause the processor to: apply a neural network (NN) filter to an unfiltered sample of a video unit to generate a filtered sample, wherein the NN filter is based on a first NN filter model having a first depth, or a second NN filter model having a second depth, wherein the depth comprises a number of residual blocks of the respective NN filter model, and wherein the second depth is different than the first depth; and convert between a video media file and a bitstream based on the filtered sample, wherein the NN filter uses the first NN filter model for a first video unit in a first temporal layer or a first type of slice, and wherein the NN filter uses the second NN filter model for a second video unit in a second temporal layer or a second type of slice, and wherein the NN filter uses the first NN filter model for a third video unit in a third temporal layer, wherein the first temporal layer and the third temporal layer are in a same subgroup of temporal layers, and wherein the second temporal layer is in a different subgroup of temporal layers. ### Claim 19 Method for storing a bitstream of a video, comprising: applying a neural network (NN) filter to an unfiltered sample of a video unit to generate a filtered sample, wherein the NN filter is based on a first NN filter model having a first depth, or a second NN filter model having a second depth, wherein the depth comprises a number of residual blocks of the respective NN filter model, and wherein the second depth is different than the first depth; generating the bitstream based on the filtered sample; and storing the bitstream in a non-transitory computer readable medium, wherein the NN filter uses the first NN filter model for a first video unit in a first temporal layer or a first type of slice, and wherein the NN filter uses the second NN filter model for a second video unit in a second temporal layer or a second type of slice, and wherein the NN filter uses the first NN filter model for a third video unit in a third temporal layer, wherein the first temporal layer and the third temporal layer are in a same subgroup of temporal layers, and wherein the second temporal layer is in a different subgroup of temporal layers.