# US 2019/0246138 A1 · Method of pre-processing of video information for optimized video encoding > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0246138 A1 - Document ID: US20190246138A1 - Status: Published application - Kind: A1 - Application number: 16/333,279 - Filed: 2017-09-05 - Published: 2019-08-08 - Earliest claimed: 2016-09-14 - Assignees: Beamr Imaging Ltd - Applicants: Beamr Imaging Ltd. - Inventors: Mikhail TERTEROV; Alexey MARTEMYANOV; Boris FILIPPOV; Tamar SHOHAM - Web page: https://patentgrep.com/patent/US20190246138A1 ## CPC - H04N 19/521: {for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors} - H04N 5/21: Circuitry for suppressing or minimising disturbance, e.g. moiré or halo - H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635) - H04N 19/137: Motion inside a coding unit, e.g. average field, frame or block difference - H04N 19/14: Coding unit complexity, e.g. amount of activity or edge presence estimation (H04N19/146 takes precedence) - H04N 19/172: the region being a picture, frame or field - H04N 19/174: the region being a slice, e.g. a line of blocks or a group of blocks - H04N 19/52: by predictive encoding - H04N 19/523: with sub-pixel accuracy - H04N 19/86: involving reduction of coding artifacts, e.g. of blockiness - H04N 23/683: {performed by a processor, e.g. controlling the readout of an image memory} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract There are provided computerized systems and methods of pre-processing video information for optimized video encoding. In various aspects, the pre-processing can be performed in different ways. By way of example, this can be performed by aligning a plurality of consecutive frames in the sequence of video frames. By way of another example, this can be performed by conducting adaptive pre-filtering of the video information. By way of yet another example, this can be performed by filtering the video information in order to remove grain content thereof. ## Claims ### Claim 1 A computerized method of pre-processing video information for optimized video encoding, the method comprising: receiving, by an I/O interface, video information to be encoded, the video information comprising a sequence of video frames; aligning, by a control circuitry, a plurality of consecutive frames in the sequence of video frames by spatially shifting at least one frame of the plurality of consecutive frames, giving rise to a pre-processed sequence of video frames including the aligned plurality of consecutive frames; and encoding, by the control circuitry, the pre-processed sequence of video frames using block-based motion estimation, wherein the block-based motion estimation is optimized by using the pre-processed video information. ### Claim 2 (depends on 1) The computerized method according to claim 1, wherein the aligning comprises detecting non-uniform camera movements of the plurality of consecutive frames, determining a camera movement value representative of the non-uniform camera movements, and spatially shifting the at least one frame using the camera movement value. ### Claim 3 (depends on 2) The computerized method according to claim 2, wherein the non-uniform camera movements are along a direction of camera movement. ### Claim 4 (depends on 2) The computerized method according to claim 2, wherein the non-uniform camera movements are with respect to a center position of at least a portion of each of the plurality of consecutive frames. ### Claim 5 (depends on 2) The computerized method according to claim 2, wherein the determining comprises determining the camera movement value based on an average of the nonuniform camera movements. ### Claim 6 (depends on 2) The computerized method according to claim 2, wherein the shifting is performed in a pixel or sub-pixel granularity. ### Claim 7 (depends on 1) The computerized method according to claim 1, wherein the aligning comprises aligning the plurality of consecutive frames along a sub-pixel grid that corresponds to an encoding scheme used in the encoding. ### Claim 8 (depends on 7) The computerized method according to claim 7, wherein the aligning comprises: estimating high resolution sub-pixel motion in the at least one frame; and shifting the at least one frame in accordance with the estimated high resolution sub-pixel motion such that the at least one frame aligns with the sub-pixel grid. ### Claim 9 A computerized method of pre-processing video information for optimized video encoding, the method comprising: receiving, by an I/O interface, video information to be encoded, the video information comprising a sequence of video frames; conducting, by a control circuitry, adaptive pre-filtering of the video information, giving rise to pre-processed video information, wherein the conducting comprises, for each given frame of the sequence of video frames: estimating encoding complexity for one or more blocks in the given frame, the encoding complexity indicative of difference between pixels in the one or more blocks and corresponding prediction of the pixels; determining a filter strength parameter for the given frame at least based on the encoding complexity; and filtering the given frame using the filter strength parameter, giving rise to a filtered frame; thereby obtaining a sequence of filtered frames constituting the pre-processed video information; and encoding, by the control circuitry, the pre-processed video information. ### Claim 10 (depends on 9) The computerized method according to claim 9, wherein the conducting further comprises, for each given frame: generating a saliency map for at least a portion of the given frame, the saliency map indicative of one or more areas in the at least a portion each associated with a degree of visual importance; and wherein the filtering comprises: adaptively filtering the given frame according to the saliency map using the filter strength parameter. ### Claim 11 (depends on 9) The computerized method according to claim 9, wherein the determining comprises determining the filter strength parameter using a linear model based on one or more video characteristics of the video information selected from a group comprising: the encoding complexity, bitrate, frame rate, and frame resolution. ### Claim 12 (depends on 9) The computerized method according to claim 9, wherein the determining comprises determining the filter strength parameter using a machine learning model based on one or more video characteristics of the video information selected from a group comprising: the encoding complexity, bitrate, frame rate, and frame resolution. ### Claim 13 (depends on 10) The computerized method according to claim 10, wherein the generating comprises extracting one or more features characterizing the given frame, the one or more features selected from a group comprising: intricateness, gradient, smoothness, grain, face, skin, chroma, level of photo-realism, luminance, and motion of the given frame, and generating the saliency map using the one or more features. ### Claim 14 A computerized method of pre-processing video information for optimized video encoding, the method comprising: receiving, by an I/O interlace, video information to be encoded, the video information comprising a sequence of video frames; filtering, by a control circuitry, the video information in order to remove grain content thereof, giving rise to pre-processed video information, wherein the filtering comprises, for each given frame of the sequence of video frames; obtaining temporal prediction information for the given frame; detecting areas in the given frame with bad temporal prediction using the temporal prediction information and excluding the detected areas from the given frame, giving rise to a remaining frame area; performing a temporal filtration on the remaining frame area, giving rise to a temporal filtered frame; performing a spatial filtration on the temporal filtered frame, giving rise to a grain filtered frame; thereby obtaining a sequence of grain filtered frames constituting the pre-processed video information; and encoding, by the control circuitry, the pre-processed video information. ### Claim 15 (depends on 14) The computerized method according to claim 14, wherein the temporal filtration is performed by: calculating a filtered pixel value for each given pixel in the remaining frame area based on a weighted summation of pixel values of the given pixel in the remaining frame area and at least one corresponding pixel in at least one motion estimated frame of the given frame, giving rise to the temporal filtered frame. ### Claim 16 (depends on 14) The computerized method according to claim 14, wherein the detecting areas in the given frame with bad temporal prediction is configured using one or more grain strength values associated with the given frame. ### Claim 17 (depends on 14) The computerized method according to claim 14, wherein the performing a spatial filtration is configured based on one or more grain strength values associated with the given frame. ### Claim 18 (depends on 16) The computerized method according to claim 16, wherein the filtering further comprises calculating the one or more grain strength values associated with the given frame, comprising; splitting the given frame into one or more luminance intervals based on pixel values thereof; and calculating a grain strength value for each given luminance interval based on variance of blocks within the given interval, giving rise to one or more grain strength values corresponding to the one or more luminance intervals. ### Claim 19 (depends on 16) The computerized method according to claim 16, wherein the detecting area is configured by setting at least one detection threshold to be used to detect areas with bad temporal prediction based on the one or more grain strength values. ### Claim 20 (depends on 14) The computerized method according to claim 14, wherein the detecting comprises: calculating absolute summation between the given frame and a motion-compensated frame thereof on a per block basis, comparing the absolute summation with at least one detection threshold; and identifying, based on the comparison, one or more poorly predicted blocks to be the areas with bad temporal prediction. ### Claim 21 (depends on 20) The computerized method according to claim 20, wherein the calculating an absolute summation comprises: calculating difference values between pixels of the given frame and a motion-compensated frame thereof, giving rise to a residual frame; splitting the residual frame into a plurality of blocks; and calculating an absolute summation for each given block in the residual frame. ### Claim 22 (depends on 21) The computerized method according to claim 21, wherein the absolute summation is a sum of absolute difference values within the given block. ### Claim 23 (depends on 21) The computerized method according to claim 21, wherein the absolute summation is absolute of sum of difference values within the given block. ### Claim 24 (depends on 17) The computerized method according to claim 17, wherein the performing a spatial filtration is further configured by setting one or more following parameters used in the spatial filtration: filter size, filter strength, and edge related parameters. ### Claim 25 A computerized system of pre-processing video information for optimized video encoding, the system comprising; an I/O interface configured to receive video information to be encoded, the video information comprising a sequence of video frames; a control circuitry operatively connected to the I/O interface, the control circuitry comprising a processor and a memory coupled thereto and configured to: align a plurality of consecutive frames in the sequence of video frames by spatially shifting at least one frame of the plurality of consecutive frames, giving rise to a pre-processed sequence of video frames including the aligned plurality of consecutive frames; and encode the pre-processed sequence of video frames using block-based motion estimation, wherein the block-based motion estimation is optimized by using the pre-processed video information. ### Claim 26 (depends on 25) The computerized system according to claim 25, wherein the control circuitry is configured to align the plurality of consecutive frames by detecting non-uniform camera movements of the plurality of consecutive frames, determining a camera movement value representative of the non-uniform camera movements and spatially shifting the at least one frame using the camera movement value. ### Claim 27 (depends on 26) The computerized system according to claim 26, wherein the non-uniform camera movements are along a direction of camera movement. ### Claim 28 (depends on 26) The computerized system according to claim 26, wherein the non-uniform camera movements are with respect to a center position of at least a portion of each of the plurality of consecutive frames. ### Claim 29 (depends on 26) The computerized system according to claim 26, wherein the determining comprises determining the camera movement value based on an average of the non-uniform camera movements. ### Claim 30 (depends on 26) The computerized system according to claim 26, wherein the shifting is performed in a pixel or sub-pixel granularity. ### Claim 31 (depends on 25) The computerized system according to claim 25, wherein the control circuitry is configured to align the plurality of consecutive frames along a sub-pixel grid that corresponds to an encoding scheme used in the encoding. ### Claim 32 (depends on 31) The computerized system according to claim 31, wherein the control circuitry is configured to align the plurality of consecutive frames along the sub-pixel grid by; estimating high resolution sub-pixel motion in the at least one frame; and shifting the at least one frame in accordance with the estimated high resolution sub-pixel motion such that the at least one frame aligns with the sub-pixel grid. ### Claim 33 A computerized system of pre-processing video information for optimized video encoding, the system comprising: an I/O interface configured to receive video information to be encoded, the video information comprising a sequence of video frames; a control circuitry operatively connected to the I/O interface, the control circuitry comprising a processor and a memory coupled thereto and configured to: conduct adaptive pre-filtering of the video information, giving rise to pre-processed video information, wherein the conducting comprises, for each given frame of the sequence of video frames: estimating encoding complexity for one or more blocks in the given frame, the encoding complexity indicative of difference between pixels in the one or more blocks and corresponding prediction of the pixels; determining a filter strength parameter for the given frame at least based on the encoding complexity; and filtering the given frame using the filter strength parameter, giving rise to a filtered frame; thereby obtaining a sequence of filtered frames constituting the pre-processed video information; and encode the pre-processed video information. ### Claim 34 (depends on 33) The computerized system according to claim 33, wherein the conducting further comprises, for each given frame: generating a saliency map for at least a portion of the given frame, the saliency map indicative of one or more areas in the at least a portion each associated with a degree of visual importance; and wherein the filtering comprises: adaptively filtering the given frame according to the saliency map using the filter strength parameter. ### Claim 35 (depends on 33) The computerized system according to claim 33, wherein the determining comprises determining the filter strength parameter using a linear model based on one or more video characteristics of the video information selected from a group comprising: the encoding complexity, bitrate, frame rate, and frame resolution. ### Claim 36 (depends on 33) The computerized system according to claim 33, wherein the determining comprises determining the filter strength parameter using a machine learning model based on one or more video characteristics of the video information selected from a group comprising: the encoding complexity, bitrate, frame rate, and frame resolution. ### Claim 37 (depends on 34) The computerized system according to claim 34, wherein the generating comprises extracting one or more features characterizing the given frame, the one or more features selected from a group comprising: intricateness, gradient, smoothness, grain, face, skin, chroma, level of photo-realism, luminance, and motion of the given frame, and generating the saliency map using the one or more features. ### Claim 38 A computerized system of pre-processing video information for optimized video encoding, the system comprising: an I/O interface configured to receive video information to be encoded, the video information comprising a sequence of video frames; a control circuitry operatively connected to the I/O interface, the control circuitry comprising a processor and a memory coupled thereto and configured to: filter the video information in order to remove grain content thereof, giving rise to pre-processed video information, wherein the filtering comprises, for each given frame of the sequence of video frames: obtaining temporal prediction information for the given frame; detecting areas in the given frame with bad temporal prediction using the temporal prediction information and excluding the detected areas from the given frame, giving rise to a remaining frame area; performing a temporal filtration on the remaining frame area, giving rise to a temporal filtered frame; and performing a spatial filtration on the temporal filtered frame, giving rise to a grain filtered frame; thereby obtaining a sequence of grain filtered frames constituting the pre-processed video information; and encode the pre-processed video information. ### Claim 39 (depends on 38) The computerized system according to claim 38, wherein the control circuitry is configured to perform the temporal filtration by: calculating a filtered pixel value for each given pixel in the remaining frame area based on a weighted summation of pixel values of the given pixel in the remaining frame area and at least one corresponding pixel in at least one motion estimated frame of the given frame, giving rise to the temporal filtered frame. ### Claim 40 (depends on 38) The computerized system according to claim 38, wherein the detecting areas in the given frame with bad temporal prediction is configured using one or more grain strength values associated with the given frame. ### Claim 41 (depends on 38) The computerized system according to claim 38, wherein the performing a spatial filtration is configured based on one or more grain strength values associated with the given frame. ### Claim 42 (depends on 40) The computerized system according to claim 40, wherein the filtering further comprises calculating the one or more grain strength values associated with the given frame, comprising: splitting the given frame into one or more luminance intervals based on pixel values thereof: and calculating a grain strength value for each given luminance interval based on variance of blocks within the given interval, giving rise to one or more grain strength values corresponding to the one or more luminance intervals. ### Claim 43 (depends on 40) The computerized system according to claim 40, wherein the detecting areas are configured by setting at least one detection threshold to be used to detect areas with bad temporal prediction based on the one or more grain strength values. ### Claim 44 (depends on 38) The computerized system according to claim 38, wherein the detecting comprises: calculating absolute summation between the given frame and a motion-compensated frame thereof on a per block basis, comparing the absolute summation with at least one detection threshold; and identifying, based on the comparison, one or more poorly predicted blocks to be the areas with bad temporal prediction. ### Claim 45 (depends on 44) The computerized system according to claim 44, wherein the calculating an absolute summation comprises: calculating difference values between pixels of the given frame and a motion-compensated frame thereof, giving rise to a residual frame: splitting the residual frame into a plurality of blocks; and calculating an absolute summation for each given block in the residual frame. ### Claim 46 (depends on 45) The computerized system according to claim 45, wherein the absolute summation is a sum of absolute difference values within the given block. ### Claim 47 (depends on 45) The computerized system according to claim 45, wherein the absolute summation is the absolute of sum of difference values within the given block. ### Claim 48 (depends on 41) The computerized system according to claim 41, wherein the performing a spatial filtration is further configured by setting one or more of the following parameters used in the spatial filtration: filter size, filter strength, and edge related parameters. ### Claim 49 (depends on 1) A non-transitory computer readable storage medium tangibly embodying a program of instructions that, when executed by a computer, causing the computer to perform the method steps of claim 1. ### Claim 50 (depends on 9) A non-transitory computer readable storage medium tangibly embodying a program of instructions that, when executed by a computer, causing the computer to perform the method steps of claim 9. ### Claim 51 (depends on 14) A non-transitory computer readable storage medium tangibly embodying a program of instructions that, when executed by a computer, causing the computer to perform the method steps of claim 14.