# US 11,451,790 B2 · Method and apparatus in video coding for machines > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,451,790 B2 - Document ID: US11451790B2 - Status: Granted - Kind: B2 - Application number: 17/360,838 - Filed: 2021-06-28 - Published: 2022-04-14 - Granted: 2022-09-20 - Earliest claimed: 2020-10-09 - Assignees: Tencent America LLC - Applicants: Tencent America LLC - Inventors: Shan Liu; Wen Gao; Xiaozhong Xu - Web page: https://patentgrep.com/patent/US11451790B2 ## CPC - H04N 19/154: Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion (use of rate-distortion criteria H04N19/147) - H04N 19/115: Selection of the code volume for a coding unit prior to coding - H04N 19/124: Quantisation - H04N 19/147: according to rate distortion criteria (rate-distortion as a criterion for motion estimation H04N19/567) - H04N 19/172: the region being a picture, frame or field - H04N 19/182: the unit being a pixel - H04N 19/184: the unit being bits, e.g. of the compressed video stream - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Aspects of the disclosure provide methods and apparatuses for use in video coding for machines. In some examples, an apparatus of video coding for machines includes processing circuitry. The processing circuitry determines a first picture quality vs. coding efficiency characteristic for a first coding scheme of video coding for machines (VCM), and a second picture quality vs. coding efficiency characteristic for a second coding scheme of VCM. Then, the processing circuitry determines a Bjøntegaard delta (BD) metric for a comparison of the first coding scheme and the second coding scheme, based on the first picture quality vs. coding efficiency characteristic and the second picture quality vs. coding efficiency characteristic. ## Claims ### Claim 1 A method for video coding, comprising: determining, by a processing circuit, a first picture quality vs. coding efficiency characteristic for a first coding scheme of video coding for machines (VCM), wherein the first picture quality vs. coding efficiency characteristic includes plural first picture quality vs. coding efficiency curves for the first coding scheme; determining, by the processing circuit, a second picture quality vs. coding efficiency characteristic for a second coding scheme of VCM, wherein the second picture quality vs. coding efficiency characteristic includes plural second picture quality vs. coding efficiency curves for the first coding scheme; computing a first Pareto front curve of the plural first picture quality vs. coding efficiency curves for the first coding scheme; computing a second Pareto front curve of the plural second picture quality vs. coding efficiency curves for the second coding scheme; determining, by the processing circuit, a Bjøntegaard delta (BD) metric for a comparison of the first coding scheme and the second coding scheme, based on the computed first Pareto front curve for the first coding scheme and the computed second Pareto front curve for the second coding scheme; and selecting, for use in a video encoder, one of the first coding scheme of VCM or the second coding scheme of VCM based on the BD metric indicating the comparison of the first coding scheme and the second coding scheme. ### Claim 2 (depends on 1) The method of claim 1, further comprising: calculating the BD metric in at least one of mean average precision (mAP), bits per pixel (BPP), multiple object tracking accuracy (MOTA), average precision at intersection over union threshold of 50% (AP50), average precision at intersection over union threshold of 75% (AP75), or mean accuracy. ### Claim 3 (depends on 1) The method of claim 1, wherein the plural first picture quality vs. coding efficiency curves are in a two dimensional plane for picture quality and coding efficiency, and the plural second picture quality vs. coding efficiency curves are in the two dimensional plane for the picture quality and coding efficiency. ### Claim 4 (depends on 3) The method of claim 3, further comprising: calculating the BD metric as an average gap between the first Pareto front curve and the second Pareto front curve. ### Claim 5 (depends on 1) The method of claim 1, wherein the first coding scheme and the second coding scheme are used for video coding for both machine vision and human vision, and the method comprises: calculating a first BD-rate of bits per pixel (BPP) for machine vision; calculating a second BD-rate for human vision; and calculating a weighted sum of the first BD-rate and the second BD-rate as an overall BD metric for the comparison of the first coding scheme and the second coding scheme. ### Claim 6 (depends on 1) The method of claim 1, wherein the first coding scheme and the second coding scheme are used for video coding for both machine vision and human vision, and the method comprises: determining the first picture quality based on a weighted sum of distortions for the machine vision and the human vision by the first coding scheme; and determining the second picture quality based on a weighted sum of distortions for the machine vision and the human vision by the second coding scheme. ### Claim 7 (depends on 1) The method of claim 1, wherein the first coding scheme and the second coding scheme are used for video coding for multiple vision tasks, and the method comprises: determining the first picture quality based on a weighted sum of distortions for the multiple vision tasks by the first coding scheme; and determining the second picture quality based on a weighted sum of distortions for the multiple vision tasks by the second coding scheme. ### Claim 8 An apparatus for video coding comprising: processing circuitry configured to: determine a first picture quality vs. coding efficiency characteristic for a first coding scheme of video coding for machines (VCM), wherein the first picture quality vs. coding efficiency characteristic includes plural first picture quality vs. coding efficiency curves for the first coding scheme; determine a second picture quality vs. coding efficiency characteristic for a second coding scheme of VCM, wherein the second picture quality vs. coding efficiency characteristic includes plural second picture quality vs. coding efficiency curves for the first coding scheme; compute a first Pareto front curve of the plural first picture quality vs. coding efficiency curves for the first coding scheme; compute a second Pareto front curve of the plural second picture quality vs. coding efficiency curves for the second coding scheme; determine a Bjøntegaard delta (BD) metric for a comparison of the first coding scheme and the second coding scheme, based on the computed first Pareto front curve for the first coding scheme and the computed second Pareto front curve for the second coding scheme; and select, for use in a video encoder, one of the first coding scheme of VCM or the second coding scheme of VCM based on the BD metric indicating the comparison of the first coding scheme and the second coding scheme. ### Claim 9 (depends on 8) The apparatus of claim 8, wherein the processing circuitry is configured to: calculate the BD metric in at least one of mean average precision (mAP), bits per pixel (BPP), multiple object tracking accuracy (MOTA), average precision at intersection over union threshold of 50% (AP50), average precision at intersection over union threshold of 75% (AP75), or mean accuracy. ### Claim 10 (depends on 8) The apparatus of claim 8, wherein the plural first picture quality vs. coding efficiency curves are in a two dimensional plane for picture quality and coding efficiency, and the plural second picture quality vs. coding efficiency curves are in the two dimensional plane for the picture quality and coding efficiency. ### Claim 11 (depends on 10) The apparatus of claim 10, wherein the processing circuitry is configured to: calculate the BD metric as an average gap between the first Pareto front curve and the second Pareto front curve. ### Claim 12 (depends on 8) The apparatus of claim 8, wherein the first coding scheme and the second coding scheme are used for video coding for both machine vision and human vision, and the processing circuitry is configured to: calculate a first BD-rate of bits per pixel (BPP) for machine vision; calculate a second BD-rate for human vision; and calculate a weighted sum of the first BD-rate and the second BD-rate as an overall BD metric for the comparison of the first coding scheme and the second coding scheme. ### Claim 13 (depends on 8) The apparatus of claim 8, wherein the first coding scheme and the second coding scheme are used for video coding for both machine vision and human vision, and the processing circuitry is configured to: determine the first picture quality based on a weighted sum of distortions for the machine vision and the human vision by the first coding scheme; and determine the second picture quality based on a weighted sum of distortions for the machine vision and the human vision by the second coding scheme. ### Claim 14 (depends on 8) The apparatus of claim 8, wherein the first coding scheme and the second coding scheme are used for video coding for multiple vision tasks, and the processing circuitry is configured to: determine the first picture quality based on a weighted sum of distortions for the multiple vision tasks by the first coding scheme; and determine the second picture quality based on a weighted sum of distortions for the multiple vision tasks by the second coding scheme.