# US 11,115,673 B2 · Image encoder using machine learning and data processing method of the image encoder > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,115,673 B2 - Document ID: US11115673B2 - Status: Granted - Kind: B2 - Application number: 15/980,808 - Filed: 2018-05-16 - Published: 2019-04-25 - Granted: 2021-09-07 - Earliest claimed: 2017-10-19 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Jungyeop Yang - Web page: https://patentgrep.com/patent/US11115673B2 ## CPC - G06N 20/20: Ensemble learning - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06N 3/092: Reinforcement learning - G06N 20/00: Machine learning - G06T 9/002: {using neural networks} - G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) - H04N 19/103: Selection of coding mode or of prediction mode - H04N 19/136: Incoming video signal characteristics or properties - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/51: Motion estimation or motion compensation - G06N 3/045: Combinations of networks - G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound - G06N 20/10: using kernel methods, e.g. support vector machines [SVM] - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An image encoder for outputting a bitstream by encoding an input image includes a predictive block, a machine learning based prediction enhancement (MLBE) block, and a subtractor. The predictive block is configured to generate a prediction block using data of a previous input block. The MLBE block is configured to transform the prediction block into an enhanced prediction block by applying a machine learning technique to the prediction block. The subtractor is configured to generate a residual block by subtracting pixel data of the enhanced prediction block from pixel data of a current input block. ## Claims ### Claim 1 An image encoder for outputting a bitstream by encoding an input image, the image encoder comprising: a predictive block configured to generate a prediction block using data of a previous input block; a machine learning based prediction enhancement (MLBE) block configured to transform the prediction block into an enhanced prediction block by applying a machine learning technique to the prediction block, wherein the MLBE block: selects one of the prediction block and the enhanced prediction block as a selected block in response to a rate-distortion optimization (RDO) value, wherein the enhanced prediction block is selected as the selected block in response to a rate-distortion optimization (RDO) value corresponding to the enhanced prediction block indicating a performance gain with respect to an RDO value corresponding to the prediction block, and the prediction block is selected as the selected block in response to the RDO value corresponding to the enhanced prediction block indicating no performance gain with respect to the RDO value corresponding to the prediction block; and a subtractor configured to generate a residual block of residual data by subtracting pixel data of the selected block from pixel data of a current input block. ### Claim 2 (depends on 1) The image encoder of claim 1, wherein the MLBE block is configured to execute a plurality of machine learning algorithms for processing the prediction block. ### Claim 3 (depends on 2) The image encoder of claim 2, wherein the MLBE block is configured to: select at least one of the plurality of machine learning algorithms as a selected machine learning algorithm with reference to coding information of the input image; and process the prediction block using the selected machine learning algorithm. ### Claim 4 (depends on 3) The image encoder of claim 3, wherein the coding information comprises at least one of a prediction mode corresponding to the prediction block, a magnitude and direction of a motion vector, an intra direction, a size of a coding unit, a partition form of an image, and a size of a transformation unit. ### Claim 5 (depends on 2) The image encoder of claim 2, wherein the plurality of machine learning algorithms comprise at least one of a decision tree, a neural network (NN), a convolution neural network (CNN), a support vector machine (SVM), reinforcement learning, and a K-nearest neighbor (K-NN) algorithm. ### Claim 6 (depends on 1) The image encoder of claim 1, wherein the MLBE block is configured to select one of a plurality of machine learning algorithms as a selected machine learning algorithm, process the prediction block using the selected machine learning algorithm to obtain a processed result, and generate the enhanced prediction block depending on the processed result. ### Claim 7 (depends on 6) The image encoder of claim 6, wherein a flag indicating whether to apply the machine learning technique is written in a video stream syntax of the bitstream according to the selected block. ### Claim 8 (depends on 2) The image encoder of claim 2, wherein the plurality of machine learning algorithms have decision parameters learned using a plurality of predetermined training data sets. ### Claim 9 (depends on 2) The image encoder of claim 2, wherein if the input image is provided, the plurality of machine learning algorithms are trained using frames of the input image during a training interval. ### Claim 10 (depends on 1) The image encoder of claim 1, further comprising: a transformer configured to transform the residual data which is time-domain data into frequency-domain data; a quantizer configured to output a quantized coefficient by quantizing the frequency-domain data; and an entropy coder configured to transform the quantized coefficient and coding information into the bitstream. ### Claim 11 (depends on 1) The image encoder of claim 1, wherein the MLBE block is configured to execute a machine learning algorithm of selecting one of a plurality of parameter sets as a selected parameter set depending on coding information and processing the prediction block using the selected parameter set. ### Claim 12 A method for processing image data, the method comprising: generating a prediction block from time-domain data of a previous input block; transforming the prediction block into an enhanced prediction block by applying at least one of a plurality of machine learning techniques to the prediction block; selecting one of the prediction block and the enhanced prediction block as a selected block in response to a rate-distortion optimization (RDO) value, wherein the enhanced prediction block is selected as the selected block in response to a rate-distortion optimization (RDO) value corresponding to the enhanced prediction block indicating a performance gain with respect to an RDO value corresponding to the prediction block; and the prediction block is selected as the selected block in response to the RDO value corresponding to the enhanced prediction block indicating no performance gain with respect to the RDO value corresponding to the prediction block; and generating a residual block by subtracting the selected block from a current input block. ### Claim 13 (depends on 12) The method of claim 12, further comprising obtaining coding information extracted when generating the prediction block. ### Claim 14 (depends on 13) The method of claim 13, wherein the transforming into the enhanced prediction block comprises selecting the at least one of the plurality of machine learning techniques depending on the coding information. ### Claim 15 (depends on 13) The method of claim 13, wherein the coding information comprises at least one of a prediction mode corresponding to the prediction block, a magnitude and direction of a motion vector, an intra direction, a size of a coding unit, a partition form of an image, and a size of a transformation unit. ### Claim 16 (depends on 13) The method of claim 13, wherein the plurality of machine learning techniques comprise at least one of a decision tree, a neural network (NN), a convolution neural network (CNN), a support vector machine (SVM), reinforcement learning, and a K-nearest neighbor (K-NN) algorithm. ### Claim 17 A method for processing image data, the method comprising: generating a prediction block from time-domain data of a previous input block; transforming the prediction block into an enhanced prediction block by applying at least one of a plurality of machine learning techniques to the prediction block; selecting one of the prediction block and the enhanced prediction block as a selected block in response to a rate-distortion optimization (RDO) value corresponding to the enhanced prediction block; and generating a residual block by subtracting the selected block from a current input block. ### Claim 18 (depends on 17) The method of claim 17, wherein the selecting comprises selecting one block having an RDO value with better compression efficiency as a selected result between the prediction block and the enhanced prediction block. ### Claim 19 (depends on 17) The method of claim 17, further comprising writing a flag indicating whether to apply the at least one of the plurality of machine learning techniques in a video stream syntax of a bitstream depending on the selecting of the prediction block or the enhanced prediction block.