# US 11,451,818 B2 · Bidirectional optical flow based video coding and decoding > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,451,818 B2 - Document ID: US11451818B2 - Status: Granted - Kind: B2 - Application number: 17/405,374 - Filed: 2021-08-18 - Published: 2021-12-09 - Granted: 2022-09-20 - Earliest claimed: 2019-04-02 - Assignees: Beijing Bytedance Network Technology Co., Ltd.; Bytedance Inc. - Applicants: Beijing Bytedance Network Technology Co., Ltd.; Bytedance Inc. - Inventors: Hongbin Liu; Li Zhang; Kai Zhang; Jizheng Xu; Yue Wang - Web page: https://patentgrep.com/patent/US11451818B2 ## CPC - H04N 19/513: Processing of motion vectors - H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction - H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking - H04N 19/159: Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction - H04N 19/176: the region being a block, e.g. a macroblock - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Devices, systems and methods for sample refinement and filtering method for video coding are described. In an exemplary aspect, a method for video processing includes modifying, for a conversion between a block of a video and a bitstream representation of the video, a refinement value for a prediction sample in the block by applying a clipping operation to refinement value. The refinement value is derived based on a gradient value of an optical flow coding process. An output of the clipping operation is within a range. The method also includes refining the prediction sample based on the refinement value and performing the conversion based on the refined prediction sample. ## Claims ### Claim 1 A method of processing video data, comprising: determining, for a first block of a video, a first prediction sample; refining the first prediction sample, based on a first optical flow refinement procedure, with a first prediction sample offset by adding the first prediction sample offset to the first prediction sample to acquire a first final prediction sample, wherein the first prediction sample offset is determined based on at least one first gradient of the first prediction sample, wherein the first gradient is calculated based on at least a difference between two prediction samples acquired based on a location of the first prediction sample, and wherein before being added to the first prediction sample to acquire the first final prediction sample, the first prediction sample offset is clipped to a first range; and performing a conversion between the first block and a bitstream of the video based on the first final prediction sample, wherein the first range is determined based on an input bit depth of the first block; wherein the method further comprises: determining, for a second block of the video, a second prediction sample; refining the second prediction sample, based on a second optical flow refinement procedure, with a second prediction sample offset to acquire a second final prediction sample; wherein the second prediction sample offset is determined based on at least one second gradient of the second prediction sample, wherein the second gradient is calculated based on at least a difference between two prediction samples acquired based on a location of the second prediction sample; clipping the second final prediction sample to a second range; and performing a conversion between the second block and the bitstream based on the second final prediction sample. ### Claim 2 (depends on 1) The method of claim 1, wherein the first video block is coded with an affine mode. ### Claim 3 (depends on 1) The method of claim 1, wherein the first range is [min, max], min being smaller than 0 and max being larger than 0. ### Claim 4 (depends on 1) The method of claim 1, wherein the first range is predefined. ### Claim 5 (depends on 1) The method of claim 1, wherein the second block is coded with a non-affine mode and is bi-predicated. ### Claim 6 (depends on 1) The method of claim 1, wherein the second range is determined based on an input bit depth of the second block. ### Claim 7 (depends on 1) The method of claim 1, wherein the second range is [min, max], min being smaller than 0 and max being larger than 0. ### Claim 8 (depends on 1) The method of claim 1, wherein the second range is predefined. ### Claim 9 (depends on 1) The method of claim 1, wherein the first video block is coded with an affine mode and the second video block is coded with a bi-predictive mode and a non-affine mode. ### Claim 10 (depends on 1) The method of claim 1, wherein the two prediction samples acquired based on the location of the first prediction sample are derived from a same reference picture list, and wherein the two prediction samples acquired based on the location of the second prediction sample are derived from a different reference picture list. ### Claim 11 (depends on 1) The method of claim 1, wherein performing the conversion includes decoding the first blocks from the bitstream. ### Claim 12 (depends on 1) The method of claim 1, wherein performing the conversion includes encoding the first blocks into the bitstream. ### Claim 13 An apparatus for processing video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to: determine, for a first block of a video, a first prediction sample; refine the first prediction sample, based on a first optical flow refinement procedure, with a first prediction sample offset by adding the first prediction sample offset to the first prediction sample to acquire a first final prediction sample, wherein the first prediction sample offset is determined based on at least one first gradient of the first prediction sample, wherein the first gradient is calculated based on at least a difference between two prediction samples acquired based on a location of the first prediction sample, and wherein before being added to the first prediction sample to acquire the first final prediction sample, the first prediction sample offset is clipped to a first range; and perform a conversion between the first block and a bitstream of the video based on the first final prediction sample, wherein the first range is determined based on an input bit depth of the first block; wherein the instructions upon execution by the processor, further cause the processor to: determine, for a second block of the video, a second prediction sample; refine the second prediction sample, based on a second optical flow refinement procedure, with a second prediction sample offset to acquire a second final prediction sample; wherein the second prediction sample offset is determined based on at least one second gradient of the second prediction sample, wherein the second gradient is calculated based on at least a difference between two prediction samples acquired based on a location of the second prediction sample; clip the second final prediction sample to a second range; and perform a conversion between the second block and the bitstream based on the second final prediction sample. ### Claim 14 (depends on 13) The apparatus of claim 13, wherein the first video block is coded with an affine mode. ### Claim 15 (depends on 13) The apparatus of claim 13, wherein the second block is coded with a non-affine mode and is bi-predicated. ### Claim 16 A non-transitory computer-readable storage medium storing instructions that cause a processor to: determine, for a first block of a video, a first prediction sample; refine the first prediction sample, based on a first optical flow refinement procedure, with a first prediction sample offset by adding the first prediction sample offset to the first prediction sample to acquire a first final prediction sample, wherein the first prediction sample offset is determined based on at least one first gradient of the first prediction sample, wherein the first gradient is calculated based on at least a difference between two prediction samples acquired based on a location of the first prediction sample, and wherein before being added to the first prediction sample to acquire the first final prediction sample, the first prediction sample offset is clipped to a first range; and perform a conversion between the first block and a bitstream of the video based on the first final prediction sample, wherein the first range is determined based on an input bit depth of the first block; wherein the instructions further cause the processor to: determine, for a second block of the video, a second prediction sample; refine the second prediction sample, based on a second optical flow refinement procedure, with a second prediction sample offset to acquire a second final prediction sample; wherein the second prediction sample offset is determined based on at least one second gradient of the second prediction sample, wherein the second gradient is calculated based on at least a difference between two prediction samples acquired based on a location of the second prediction sample; clip the second final prediction sample to a second range; and perform a conversion between the second block and the bitstream based on the second final prediction sample. ### Claim 17 A method performed by a video processing apparatus, wherein the method comprises: determining, for a first block of a video, a first prediction sample; refining the first prediction sample, based on a first optical flow refinement procedure, with a first prediction sample offset by adding the first prediction sample offset to the first prediction sample to acquire a first final prediction sample, wherein the first prediction sample offset is determined based on at least one first gradient of the first prediction sample, wherein the first gradient is calculated based on at least a difference between two prediction samples acquired based on a location of the first prediction sample, and wherein before being added to the first prediction sample to acquire the first final prediction sample, the first prediction sample offset is clipped to a first range; generating the bitstream based on the first final prediction sample, and storing the bitstream to a non-transitory computer-readable medium; wherein the first range is determined based on an input bit depth of the first block; wherein the method further comprises: determining, for a second block of the video, a second prediction sample; refining the second prediction sample, based on a second optical flow refinement procedure, with a second prediction sample offset to acquire a second final prediction sample; wherein the second prediction sample offset is determined based on at least one second gradient of the second prediction sample, wherein the second gradient is calculated based on at least a difference between two prediction samples acquired based on a location of the second prediction sample; clipping the second final prediction sample to a second range; and performing a conversion between the second block and the bitstream based on the second final prediction sample. ### Claim 18 (depends on 13) The apparatus of claim 13, wherein the first range is [min, max], min being smaller than 0 and max being larger than 0. ### Claim 19 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein the first video block is coded with an affine mode, and the second block is coded with a non-affine mode and is bi-predicated. ### Claim 20 (depends on 17) The method of claim 17, wherein the first video block is coded with an affine mode, and the second block is coded with a non-affine mode and is bi-predicated.