Systems and methods for rate control in video coding using joint machine learning and game theory

A method for video coding Rate Control (RC) for High Efficiency Video Coding (HEVC), the method comprising:

• utilizing machine learning to provide selection of different Rate-Distortion (R-D) models of a plurality of inter frame Coding Tree Unit (CTU) level R-D models for video coding of frames of a video stream, wherein the machine learning is based upon R-D model classification;
• and
• utilizing mixed R-D model based cooperative bargaining game theory modeling in RC bit allocation using the R-D models selected for video coding different portions of the video stream, wherein a minimum utility of the mixed R-D model based cooperative bargaining game theory modeling is adjusted by a reference coding distortion and frame level Quantization parameter (QP) change, wherein the minimum utility is defined by {Ui,d(ri,d), i=1, 2, . . . , N} where ri,d is a minimum bit consumption to achieve the minimum utility Ui,d for frame i of the mixed R-D model based cooperative bargaining game theory modeling, and wherein an intra frame QP and inter frame bit ratio are adjusted to make inter frames have more bit resources to enhance video visual quality smoothness.