# US 12,002,189 B2 · Remastering lower dynamic range content for higher dynamic range displays > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,189 B2 - Document ID: US12002189B2 - Status: Granted - Kind: B2 - Application number: 17/684,779 - Filed: 2022-03-02 - Published: 2023-09-07 - Granted: 2024-06-04 - Earliest claimed: 2022-03-02 - Assignees: NVIDIA Corporation - Applicants: NVIDIA Corporation - Inventors: Shaveen Kumar; Anjul Patney; Eric Xu; Anton Moor - Web page: https://patentgrep.com/patent/US12002189B2 ## CPC - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06T 5/92: based on global image properties - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06T 5/20: using local operators - G06T 5/60: using machine learning, e.g. neural networks - G06T 5/70: Denoising; Smoothing - G06T 5/90: Dynamic range modification of images or parts thereof - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/084: Backpropagation, e.g. using gradient descent - G06T 2200/24: involving graphical user interfaces [GUIs] - G06T 2207/10016: Video; Image sequence - G06T 2207/20028: Bilateral filtering - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20208: High dynamic range [HDR] image processing - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract The technology disclosed herein involves using a machine learning model (e.g., CNN) to expand lower dynamic-range image content (e.g., SDR images) into higher dynamic-range image content (e.g., HDR images). The machine learning model can take as input the lower dynamic-range image and can output multiple expansion maps that are used to make the expanded image appear more natural. The expansion maps may be used by image operators to smooth color banding and to dim overexposed regions or user interface elements in the expanded image. The expanded content (e.g., HDR image content) may then be provided to one or more devices for display or storage. ## Claims ### Claim 1 A system comprising: a memory; a processing device, operatively coupled with the memory, the processing device to: receive an image corresponding to a first pixel value range; process an input corresponding to the image using a trained machine learning model to generate a first output and a second output, wherein the second output comprises an expansion map that indicates pixel classifications for pixels in the image, wherein one or more pixels having a first classification are included in a region of the image for which luminance is to be modified; expand the image to generate image data corresponding to a second pixel value range, wherein the second pixel value range is larger than the first pixel value range; update the image data using a first operator, wherein the first operator applies a filter to reduce one or more banding artifacts in the image data based on the first output; update the image data using a second operator, wherein the second operator modifies the luminance of the region of the expanded image based on the second output; and output the image data to one or more devices. ### Claim 2 (depends on 1) The system of claim 1, wherein the image comprises a Standard Dynamic Range (SDR) image and the image data that is output comprises a High Dynamic Range (HDR) image. ### Claim 3 (depends on 1) The system of claim 1, wherein the first operator is a local operator comprising a clamped bilateral filter of variable width and the second operator is a global operator comprising a luminance multiplier. ### Claim 4 (depends on 1) The system of claim 1, wherein: the first output comprises a first expansion map that indicates, for each pixel in the image, a distance from the pixel to be considered by the first operator. ### Claim 5 (depends on 4) The system of claim 4, wherein the first classification comprises a user interface classification, and wherein the second operator reduces at least one luminance value corresponding to the region. ### Claim 6 (depends on 4) The system of claim 4, wherein the first classification comprises a high reflectance classification, and wherein the second operator increases at least one luminance value corresponding to the region. ### Claim 7 (depends on 1) The system of claim 1, wherein the trained machine learning model comprises a Deep Neural Network (DNN) that uses the image as input and outputs the first output and the second output in two or more different channels, wherein the two or more different channels comprise at least a first channel that comprises a first expansion map and a second channel that comprises a second expansion map. ### Claim 8 (depends on 1) The system of claim 1, wherein the input further comprises one or more previous outputs of the trained machine learning model. ### Claim 9 (depends on 1) The system of claim 1, wherein to expand the image, the processing device is to dequantize an 8 bit input image to generate at least one of: 10 bit image data, 12 bit image data, 16 bit image data, or 32 bit image data. ### Claim 10 (depends on 1) The system of claim 1, wherein the image comprises a frame of a video that comprises a sequence of frames, and the processing device is to output, to a display device, expanded image data for at least one frame of the sequence. ### Claim 11 A method comprising: receiving, by a processing device, an image corresponding to a first pixel value range; processing, by the processing device, an input corresponding to the image using a trained machine learning model to generate a first output and a second output, wherein the second output comprises an expansion map that indicates pixel classifications for pixels in the image, wherein one or more pixels having a first classification are included in a region of the image for which luminance is to be modified; expanding the image to generate image data corresponding to a second pixel value range, wherein the second pixel value range is larger than the first pixel value range; updating the image data using a first operator, wherein the first operator applies a filter to reduce one or more banding artifacts in the image data based on the first output; updating the image data using a second operator, wherein the second operator modifies the luminance of the region of the expanded image based on the second output; and outputting, by the processing device, the image data. ### Claim 12 (depends on 11) The method of claim 11, wherein the image comprises a Standard Dynamic Range (SDR) image and the image data that is output comprises a High Dynamic Range (HDR) image. ### Claim 13 (depends on 11) The method of claim 11, wherein the first operator is a local operator comprising a clamped bilateral filter of variable width and the second operator is a global operator comprising a luminance multiplier. ### Claim 14 (depends on 11) The method of claim 11, wherein: the first output comprises a first expansion map that indicates, for at least one pixel in the image, a distance from the pixel to be considered by the first operator. ### Claim 15 (depends on 14) The method of claim 14, wherein the first classification comprises a user interface classification, and wherein the second operator reduces at least one luminance value corresponding to the region. ### Claim 16 (depends on 14) The method of claim 14, wherein the first classification comprises a high reflectance classification, and wherein the second operator increases at least one luminance value corresponding to the region. ### Claim 17 A non-transitory machine-readable storage medium storing instructions which, when executed, cause a processing device to perform operations comprising: receiving an image corresponding to a first pixel value range; processing an input corresponding to the image using a trained machine learning model to generate a first output and a second output, wherein the second output comprises an expansion map that indicates pixel classifications for pixels in the image, wherein one or more pixels having a first classification are included in a region of the image for which luminance is to be modified; expanding the image to generate image data corresponding to a second pixel value range, wherein the second pixel value range is larger than the first pixel value range; updating the image data using a first operator, wherein the first operator applies a filter to reduce one or more banding artifacts in the image data based on the first output; updating the image data using a second operator, wherein the second operator modifies the luminance of the region of the expanded image based on the second output; and outputting the image data to one or more devices. ### Claim 18 (depends on 17) The non-transitory machine-readable storage medium of claim 17, wherein the image comprises a Standard Dynamic Range (SDR) image and the image data that is output comprises a High Dynamic Range (HDR) image. ### Claim 19 (depends on 17) The non-transitory machine-readable storage medium of claim 17, wherein the first operator is a local operator comprising a clamped bilateral filter of variable width and the second operator is a global operator comprising a luminance multiplier. ### Claim 20 (depends on 17) The non-transitory machine-readable storage medium of claim 17, wherein the trained machine learning model comprises a Convolutional Neural Network (CNN) that uses the image as input and outputs the first output and the second output in two or more different channels, wherein the two or more different channels comprise at least a first channel that comprises a first expansion map and a second channel that comprises a second expansion map.