# US 12,112,460 B2 · Image processing method and apparatus > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,460 B2 - Document ID: US12112460B2 - Status: Granted - Kind: B2 - Application number: 17/564,653 - Filed: 2021-12-29 - Published: 2022-04-21 - Granted: 2024-10-08 - Earliest claimed: 2019-07-11 - Assignees: Huawei Technologies Co., Ltd. - Applicants: Huawei Technologies Co., Ltd. - Inventors: Quanhe Yu - Web page: https://patentgrep.com/patent/US12112460B2 ## CPC - G06T 5/92: based on global image properties - G09G 5/10: Intensity circuits - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T 2207/20208: High dynamic range [HDR] image processing - G09G 2320/0673: for control of gamma adjustment, e.g. selecting another gamma curve - G09G 2360/16: Calculation or use of calculated indices related to luminance levels in display data - G09G 2370/04: Exchange of auxiliary data, i.e. other than image data, between monitor and graphics controller - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) ## Abstract An image processing method includes obtaining a target picture and an initial tone mapping curve of the target picture, determining a plurality of interpolation points based on luminance information and/or dynamic metadata of the target picture, and determining a spline function that passes through the plurality of interpolation points, where the spline function is smoothly connected to the initial tone mapping curve at at least one of the plurality of interpolation points the initial tone mapping curve is modified based on the spline function to obtain a modified tone mapping curve of the target picture; and tone mapping is performed on the target picture based on the modified tone mapping curve. This image processing apparatus enables a display device to vividly and realistically reproduce a scene in a real world. ## Claims ### Claim 1 An image processing method, comprising: obtaining a target picture and an initial tone mapping curve of the target picture; determining a plurality of interpolation points based on luminance information and/or dynamic metadata of the target picture; determining a spline function that passes through the plurality of interpolation points, wherein the spline function is smoothly connected to the initial tone mapping curve at at least one of the plurality of interpolation points; obtaining a modified tone mapping curve of the target picture based on the initial tone mapping curve and the spline function; and performing tone mapping on the target picture based on the modified tone mapping curve, wherein the initial tone mapping curve comprises a first endpoint and a second endpoint, the first endpoint and the second endpoint are respectively located at two ends of the initial tone mapping curve, a luminance value corresponding to the first endpoint is less than a luminance value corresponding to the second endpoint, a quantity of dark region pixels comprised in the target picture is greater than or equal to a second preset threshold, luminance values of the dark region pixels are less than a first preset threshold, the plurality of interpolation points comprise a first interpolation point, and the determining the plurality of interpolation points based on luminance information of the target picture further comprises: determining that a first coordinate value of the first interpolation point is the same as a first coordinate value of the first endpoint. ### Claim 2 (depends on 1) The image processing method according to claim 1, wherein the plurality of interpolation points further comprise a second interpolation point, and the determining the plurality of interpolation points based on luminance information of the target picture further comprises: determining a first coordinate value of the second interpolation point based on the quantity of the dark region pixels. ### Claim 3 (depends on 1) The image processing method according to claim 1, wherein the initial tone mapping curve comprises the first endpoint and the second endpoint, the first endpoint and the second endpoint are respectively located at the two ends of the initial tone mapping curve, the luminance value corresponding to the first endpoint is less than the luminance value corresponding to the second endpoint, a quantity of bright region pixels comprised in the target picture is greater than or equal to a fourth preset threshold, luminance values of the bright region pixels are greater than a third preset threshold, the plurality of interpolation points comprise a fourth interpolation point, and the determining the plurality of interpolation points based on luminance information of the target picture further comprises: determining that a first coordinate value of the fourth interpolation point is the same as a first coordinate value of the second endpoint. ### Claim 4 (depends on 1) The image processing method according to claim 1, wherein the plurality of interpolation points comprise a seventh interpolation point, an eighth interpolation point, and a ninth interpolation point, the spline function comprises a third spline function that passes through the seventh interpolation point, the eighth interpolation point, and the ninth interpolation point, and a first coordinate value of the eighth interpolation point is between a first coordinate value of the seventh interpolation point and a first coordinate value of the ninth interpolation point; before determining the plurality of interpolation points, the method further comprises: obtaining a first offset, wherein the first offset is used to represent a modification range of the initial tone mapping curve at a first modification position, and the first modification position is the first coordinate value of the eighth interpolation point, determining the plurality of interpolation points further comprises: determining a second coordinate value of the eighth interpolation point based on a function value of the initial tone mapping curve at the first modification position and the first offset, and determining the spline function that passes through the plurality of interpolation points further comprises: determining the third spline function based on the first coordinate value of the eighth interpolation point and the second coordinate value of the eighth interpolation point, wherein the third spline function is smoothly connected to the initial tone mapping curve at the seventh interpolation point and/or the ninth interpolation point. ### Claim 5 (depends on 4) The image processing method according to claim 4, wherein the third spline function comprises a fourth spline function that passes through the seventh interpolation point and the eighth interpolation point, and a fifth spline function that passes through the eighth interpolation point and the ninth interpolation point, and a second-order derivative value of the fourth spline function at the eighth interpolation point is the same as a second-order derivative value of the fifth spline function at the eighth interpolation point. ### Claim 6 (depends on 4) The image processing method according to claim 4, wherein the plurality of interpolation points further comprise a tenth interpolation point, the third spline function passes through the tenth interpolation point, and a first coordinate value of the tenth interpolation point is between the first coordinate value of the seventh interpolation point and the first coordinate value of the ninth interpolation point, before determining the plurality of interpolation points, the method further comprises: obtaining a second offset, wherein the second offset is used to represent a modification range of the initial tone mapping curve at a second modification position, and the second modification position is the first coordinate value of the tenth interpolation point, determining the plurality of interpolation points further comprises: determining a second coordinate value of the tenth interpolation point based on a function value of the initial tone mapping curve at the second modification position and the second offset; and determining the third spline function based on the first coordinate value of the eighth interpolation point and the second coordinate value of the eighth interpolation point further comprises: determining the third spline function based on the first coordinate value and the second coordinate value of the eighth interpolation point, and the first coordinate value and the second coordinate value of the tenth interpolation point, wherein the third spline function is smoothly connected to the initial tone mapping curve at both the seventh interpolation point and the ninth interpolation point. ### Claim 7 (depends on 1) The image processing method according to claim 1, wherein the initial tone mapping curve comprises one of the following formulas: L ′ = a × ( p × L ( p - 1 ) × L + 1 ) m + b ; L ′ = a × L + b ; L ′ = c 1 + c 2 × L n 1 + c 3 × L n ; and L ′ ⁢ ∑ ρ = 0 N ⁢ ( N ρ ) × L ρ × ( 1 - L ) N - ρ × P ρ , wherein L represents a linear signal value, L′ represents a nonlinear signal value, p, a, m, b, c1, c2, c3, n, N, and ρ are all dynamic range mapping coefficients, and 0≤L≤1. ### Claim 8 An image processing apparatus, comprising a processor and a transmission interface, wherein the transmission interface is configured to receive or send image data; and the processor is configured to invoke program instructions stored in a memory, and execute the program instructions to cause the image processing apparatus to: obtain a target picture and an initial tone mapping curve of the target picture; determine a plurality of interpolation points based on luminance information and/or dynamic metadata of the target picture; determine a spline function that passes through the plurality of interpolation points, wherein the spline function is smoothly connected to the initial tone mapping curve at at least one of the plurality of interpolation points; obtain a modified tone mapping curve of the target picture based on the initial tone mapping curve and the spline function; and perform tone mapping on the target picture based on the modified tone mapping curve wherein the initial tone mapping curve comprises a first endpoint and a second endpoint, the first endpoint and the second endpoint are respectively located at two ends of the initial tone mapping curve, a luminance value corresponding to the first endpoint is less than a luminance value corresponding to the second endpoint, a quantity of dark region pixels comprised in the target picture is greater than or equal to a second preset threshold, luminance values of the dark region pixels are less than a first preset threshold, the plurality of interpolation points comprise a first interpolation point, wherein the processor is further configured to execute the program instructions to cause the image processing apparatus to: determine that a first coordinate value of the first interpolation point is the same as a first coordinate value of the first endpoint. ### Claim 9 (depends on 8) The image processing apparatus according to claim 8, wherein the plurality of interpolation points further comprise a second interpolation point, and wherein the processor is further configured to execute the program instructions to cause the image processing apparatus to: determine a first coordinate value of the second interpolation point based on the quantity of the dark region pixels. ### Claim 10 (depends on 8) The image processing apparatus according to claim 8, wherein the initial tone mapping curve comprises the first endpoint and the second endpoint, the first endpoint and the second endpoint are respectively located at the two ends of the initial tone mapping curve, the luminance value corresponding to the first endpoint is less than the luminance value corresponding to the second endpoint, a quantity of bright region pixels comprised in the target picture is greater than or equal to a fourth preset threshold, luminance values of the bright region pixels are greater than a third preset threshold, the plurality of interpolation points comprise a fourth interpolation point, and determining the plurality of interpolation points based on luminance information of the target picture further comprises: determining that a first coordinate value of the fourth interpolation point is the same as a first coordinate value of the second endpoint. ### Claim 11 (depends on 8) The image processing apparatus according to claim 8, wherein the plurality of interpolation points comprise a seventh interpolation point, an eighth interpolation point, and a ninth interpolation point, the spline function comprises a third spline function that passes through the seventh interpolation point, the eighth interpolation point, and the ninth interpolation point, and a first coordinate value of the eighth interpolation point is between a first coordinate value of the seventh interpolation point and a first coordinate value of the ninth interpolation point; before determining the plurality of interpolation points, wherein the processor is further configured to execute the program instructions to cause the image processing apparatus to: obtain a first offset, wherein the first offset is used to represent a modification range of the initial tone mapping curve at a first modification position, and the first modification position is the first coordinate value of the eighth interpolation point; determine a second coordinate value of the eighth interpolation point based on a function value of the initial tone mapping curve at the first modification position and the first offset; and determine the third spline function based on the first coordinate value of the eighth interpolation point and the second coordinate value of the eighth interpolation point, wherein the third spline function is smoothly connected to the initial tone mapping curve at the seventh interpolation point and/or the ninth interpolation point. ### Claim 12 (depends on 11) The image processing apparatus according to claim 11, wherein the plurality of interpolation points further comprise a tenth interpolation point, the third spline function passes through the tenth interpolation point, and a first coordinate value of the tenth interpolation point is between the first coordinate value of the seventh interpolation point and the first coordinate value of the ninth interpolation point; before determining the plurality of interpolation points, wherein the processor is further configured to execute the program instructions to cause the image processing apparatus to: obtain a second offset, wherein the second offset is used to represent a modification range of the initial tone mapping curve at a second modification position, and the second modification position is the first coordinate value of the tenth interpolation point; determine a second coordinate value of the tenth interpolation point based on a function value of the initial tone mapping curve at the second modification position and the second offset; determine the third spline function based on the first coordinate value and the second coordinate value of the eighth interpolation point, and the first coordinate value and the second coordinate value of the tenth interpolation point, wherein the third spline function is smoothly connected to the initial tone mapping curve at both the seventh interpolation point and the ninth interpolation point. ### Claim 13 (depends on 8) The image processing apparatus according to claim 8, wherein the initial tone mapping curve comprises one of the following formulas: L ′ = a × ( p × L ( p - 1 ) × L + 1 ) m + b ; L ′ = a × L + b ; L ′ = c 1 + c 2 × L n 1 + c 3 × L n ; and L ′ ⁢ ∑ ρ = 0 N ⁢ ( N ρ ) × L ρ × ( 1 - L ) N - ρ × P ρ , wherein L represents a linear signal value, L′ represents a nonlinear signal value, p, a, m, b, c1, c2, c3, n, N, and ρ are all dynamic range mapping coefficients, and 0≤L≤1. ### Claim 14 A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores instructions; and when the instructions are run on a computer or a processor, the computer or the processor is enabled to: obtain a target picture and an initial tone mapping curve of the target picture; determine a plurality of interpolation points based on luminance information and/or dynamic metadata of the target picture; determine a spline function that passes through the plurality of interpolation points, wherein the spline function is smoothly connected to the initial tone mapping curve at at least one of the plurality of interpolation points; obtain a modified tone mapping curve of the target picture based on the initial tone mapping curve and the spline function; and perform tone mapping on the target picture based on the modified tone mapping curve, wherein the initial tone mapping curve comprises a first endpoint and a second endpoint, the first endpoint and the second endpoint are respectively located at two ends of the initial tone mapping curve, a luminance value corresponding to the first endpoint is less than a luminance value corresponding to the second endpoint, a quantity of dark region pixels comprised in the target picture is greater than or equal to a second preset threshold, luminance values of the dark region pixels are less than a first preset threshold, the plurality of interpolation points comprise a first interpolation point, wherein the computer or the processor is further enabled to: determine that a first coordinate value of the first interpolation point is the same as a first coordinate value of the first endpoint. ### Claim 15 (depends on 14) The non-transitory computer-readable storage medium according to claim 14, wherein the plurality of interpolation points comprise a seventh interpolation point, an eighth interpolation point, and a ninth interpolation point, the spline function comprises a third spline function that passes through the seventh interpolation point, the eighth interpolation point, and the ninth interpolation point, and a first coordinate value of the eighth interpolation point is between a first coordinate value of the seventh interpolation point and a first coordinate value of the ninth interpolation point, before determining the plurality of interpolation points, wherein the computer or the processor is further enabled to: obtain a first offset, wherein the first offset is used to represent a modification range of the initial tone mapping curve at a first modification position, and the first modification position is the first coordinate value of the eighth interpolation point; determine a second coordinate value of the eighth interpolation point based on a function value of the initial tone mapping curve at the first modification position and the first offset; and determine the third spline function based on the first coordinate value of the eighth interpolation point and the second coordinate value of the eighth interpolation point, wherein the third spline function is smoothly connected to the initial tone mapping curve at the seventh interpolation point and/or the ninth interpolation point. ### Claim 16 (depends on 14) The non-transitory computer-readable storage medium according to claim 14, wherein the plurality of interpolation points further comprise a tenth interpolation point, the third spline function passes through the tenth interpolation point, and a first coordinate value of the tenth interpolation point is between the first coordinate value of the seventh interpolation point and the first coordinate value of the ninth interpolation point; before determining the plurality of interpolation points, wherein the computer or the processor is further enabled to: obtain a second offset, wherein the second offset is used to represent a modification range of the initial tone mapping curve at a second modification position, and the second modification position is the first coordinate value of the tenth interpolation point; determine a second coordinate value of the tenth interpolation point based on a function value of the initial tone mapping curve at the second modification position and the second offset; determine the third spline function based on the first coordinate value and the second coordinate value of the eighth interpolation point, and the first coordinate value and the second coordinate value of the tenth interpolation point, wherein the third spline function is smoothly connected to the initial tone mapping curve at both the seventh interpolation point and the ninth interpolation point. ### Claim 17 (depends on 14) The non-transitory computer-readable storage medium according to claim 14, wherein the initial tone mapping curve comprises one of the following formulas: L ′ = a × ( p × L ( p - 1 ) × L + 1 ) m + b ; L ′ = a × L + b ; L ′ = c 1 + c 2 × L n 1 + c 3 × L n ; and L ′ ⁢ ∑ ρ = 0 N ⁢ ( N ρ ) × L ρ × ( 1 - L ) N - ρ × P ρ , wherein L represents a linear signal value, L′ represents a nonlinear signal value, p, a, m, b, c1, c2, c3, n, N, and ρ are all dynamic range mapping coefficients, and 0≤L≤1.