# US 12,003,746 B2 · Joint forward and backward neural network optimization in image processing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,003,746 B2 - Document ID: US12003746B2 - Status: Granted - Kind: B2 - Application number: 17/800,886 - Filed: 2021-02-17 - Published: 2023-03-16 - Granted: 2024-06-04 - Earliest claimed: 2020-02-19 - Assignees: Dolby Laboratories Licensing Corporation - Applicants: Dolby Laboratories Licensing Corporation - Inventors: Guan-Ming Su - Web page: https://patentgrep.com/patent/US12003746B2 ## CPC - H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635) - H04N 19/436: using parallelised computational arrangements - H04N 19/119: Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks - H04N 19/136: Incoming video signal characteristics or properties - H04N 19/186: the unit being a colour or a chrominance component - H04N 19/30: using hierarchical techniques, e.g. scalability (H04N19/63 takes precedence) - H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A source color grade and a destination color grade may be received by an end-to-end image mapping pipeline comprising forward and backward paths. Forward neural networks in the forward path may be applied to generate, from the source color grade, a forward reshaped color grade corresponding to the destination color grade. Backward neural networks in the backward path may be applied to generate, from the forward reshaped color grade, a backward reshaped color grade corresponding to the source color grade. An overall neural network cost function may be specified for the forward path and the backward path to generate a cost that may be used as a basis for determining operational parameters for the forward and backward neural networks. ## Claims ### Claim 1 A computer-implemented method comprising: receiving a source image of a source color grade and a destination image of a destination color grade as input to an end-to-end image mapping pipeline comprising a forward path and a backward path, partitioning the forward path into multiple sub-nets represented by a plurality of sets of forward neural networks, with each of the multiple sub-nets in the forward path represented by a corresponding set of forward neural networks in the plurality of sets of forward neural networks, partitioning the backward path into multiple sub-nets represented by a plurality of sets of backward neural networks, with each of the multiple sub-nets in the backward path represented by a corresponding set of backward neural networks in the plurality of sets of backward neural networks, wherein the forward path and the backward path are concatenated together; applying the plurality of sets of forward neural networks in the forward path to forward reshape the source image of the source color grade to generate a forward reshaped image of the destination color grade; applying the plurality of sets of backward neural networks in the backward path to backward reshape the forward reshaped image of the destination color grade to generate a backward reshaped image of the source color grade; computing a joint neural network cost function specified for both the forward path and the backward path, wherein the joint neural network cost function comprises a forward cost portion that computes a first difference between the forward reshaped image and the destination image, wherein the joint neural network cost function further comprises a backward cost portion that computes a second difference between the backward reshaped image and the source image; and determining operational parameters for the plurality of sets of forward neural networks and for the plurality of sets of backward neural networks by back propagation using the joint neural network cost function; wherein the plurality of sets of forward neural networks convert the source image of the source color grade to a sequence of successive forward reshaped images for a plurality of successively lower quality destination color grades, wherein the plurality of sets of backward neural networks convert a forward reshaped image last generated in the sequence of successive forward reshaped images in the forward path to a plurality of backward reshaped images for the plurality of destination color grades, and wherein the forward reshaped image last generated in the sequence of successive forward reshaped images in the forward path and the plurality of backward reshaped images are used in the joint neural cost function. ### Claim 2 (depends on 1) The method of claim 1, further comprising: encoding image data of a selected color grade, generated from the forward path, in a video signal along with image metadata, wherein the image metadata comprises at least a subset of operational parameters for a subset of the plurality of sets of forward neural networks and/or the plurality of sets of backward neural networks for causing a recipient device of the video signal to use the image metadata and the image data of the selected color grade to generate reconstructed image data of a color grade different from the selected color grade. ### Claim 3 (depends on 2) The method of claim 2, wherein the selected color grade is a selected destination color grade of a dynamic range higher than dynamic ranges of all other destination color grades generated from the forward path and/or of a color space larger than color spaces of all other destination color grades generated from the forward path. ### Claim 4 (depends on 2) The method of claim 2, wherein the selected color grade is a selected destination color grade of a dynamic range lower than dynamic ranges of all other destination color grades generated from the forward path and/or of a color space smaller than color spaces of all other destination color grades generated from the forward path. ### Claim 5 (depends on 2) The method of claim 2, wherein the image metadata comprises operational parameters for the plurality of sets of backward neural networks only. ### Claim 6 (depends on 2) The method of claim 2, wherein the image metadata comprises operational parameters for a mixture of the plurality of sets of forward neural networks and the plurality of sets of backward neural networks. ### Claim 7 (depends on 2) The method of claim 2, wherein the video signal represents a single-layer backward compatible signal. ### Claim 8 (depends on 1) The method of claim 1, wherein the forward reshaped image of the destination color grade is the forward reshaped image last generated in the forward path. ### Claim 9 (depends on 1) The method of claim 1, wherein the forward reshaped image of the destination color grade is not the forward reshaped image last generated in the forward path. ### Claim 10 (depends on 1) The method of claim 1, wherein the plurality of sets of forward neural networks reshape the source image of the source color grade into a plurality of corresponding images in a plurality of destination color grades; wherein the plurality of destination color grades forms a two-dimensional grid with a first dimension indicating different color spaces and a second dimension indicating different dynamic ranges. ### Claim 11 (depends on 1) The method of claim 1, wherein the joint neural network cost function is formed as weighted combination of the forward cost portion and the backward cost portion by weighting factors. ### Claim 12 (depends on 11) The method of claim 11, wherein the weighting factors comprise a source color grade weighting factor corresponding to the source color grade and a destination color grade weighting factor corresponding to the destination color grade. ### Claim 13 (depends on 1) The method of claim 1, further comprising: decoding image data of a destination color grade, generated from a plurality of sets of forward neural networks in a forward path implemented in an upstream video encoder, from a video signal along with image metadata, wherein the image metadata comprises at least a subset of operational parameters for a subset of the plurality of sets of forward neural networks in the forward path and a plurality of sets of backward neural networks in a corresponding backward path implemented in the upstream video encoder; applying the subset of the plurality of sets of forward neural networks and the plurality of sets of backward neural networks with the subset of operational parameters to the image data of the destination color grade to generate reconstructed image data of a color grade different from the destination color grade; causing a display image derived from the reconstructed image data of the different color grade to be rendered with a display device, wherein the image data comprising a forward reshaped image of the destination color grade is selected from among the sequence of successive forward reshaped images generated by the plurality of sets of forward neural networks for the plurality of successively lower quality destination color grades. ### Claim 14 (depends on 1) A system comprising a processor and configured to perform the method recited in claim 1. ### Claim 15 (depends on 1) A non-transitory computer readable storage medium having stored thereon computer executable instruction for executing by one or more processors the method recited in claim 1.