# US 11,908,109 B1 · Enhanced video processor > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,109 B1 - Document ID: US11908109B1 - Status: Granted - Kind: B1 - Application number: 17/089,350 - Filed: 2020-11-04 - Granted: 2024-02-20 - Earliest claimed: 2019-11-04 - Assignees: madVR Holdings LLC - Applicants: madVR Holdings, LLC - Inventors: Mathias Rauen - Web page: https://patentgrep.com/patent/US11908109B1 ## CPC - G06T 5/92: based on global image properties - G06T 3/18: Image warping, e.g. rearranging pixels individually - G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 5/70: Denoising; Smoothing - G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00) - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T 2207/20024: Filtering details - G06T 2207/20192: Edge enhancement; Edge preservation - G06T 2207/20208: High dynamic range [HDR] image processing - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Various improvements to video processing are described, including highlight recovery by dividing an image into different frequency ranges and compressing the lower range before recombining, tone mapping to avoid hue shifts, dynamic tone mapping based on detected scene changes in the video, dynamic tone mapping using the shape of a tone mapping curve that is changed based on the histogram of each video frame, HDR Luminance Channel Repair of artifacts, changing the shape of the tone mapping curve based on a histogram of each video frame, upscaling Chroma by using Luma channel information motion interpolation using Neural Networks, motion compensated noise reduction using Neural Networks to filter random noise and film grain, grain/noise agnostic upscaling using Neural Networks, and hiding video frames by strategically dropping or repeating them in unnoticeable places to prevent a visible video stutter based on scene characteristics to accommodate clock variations. ## Claims ### Claim 1 A computer implemented method comprising: receiving a video sequence having image frames containing pixels with a high dynamic range; separating each image frame into low and high frequency components; compressing the low frequency components using a non-linear brightness compression curve including tone mapping the low frequency components; reducing brightness and saturation for pixels of the low frequency components which are outside a target gamut; reconstructing the image frames of the video sequence by combining the high frequency components with the compressed and reduced brightness and saturation for pixels of the low frequency components; and rendering the reconstructed image frames of the video sequence during real time playback for display on a device having a lower dynamic range, while preserving highlight detail, wherein tone mapping the low frequency components comprises: determining a tone mapping for each successive image frame; smoothing the tone mapping to reduce visible flickering artifacts; detecting a scene change; and discontinuing smoothing the tone mapping for a first image frame of the detected scene change. ### Claim 2 (depends on 1) The method of claim 1 wherein the image frame pixels have a luminance range of 0 to up to 10,000 nits, and further comprising: repairing damage to a luminance channel of each image frame caused by reducing brightness and saturation for pixels of the components; and reducing saturation following repairing damage such that the pixels fit withing the target gamut. ### Claim 3 (depends on 1) The method of claim 1 wherein the lower dynamic range comprises 0 to 200 nits. ### Claim 4 (depends on 1) The method of claim 1 wherein reconstructing the image frames of the video sequence comprises adding the high frequency components to the compressed low frequency components. ### Claim 5 (depends on 1) The method of claim 1 wherein separating each image into low and high frequency components comprises blurring each video frame image and subtracting the blurred video frame images from the received video frame images to obtain the high frequency components. ### Claim 6 (depends on 5) The method of claim 5 wherein the blurring is performed using a Gaussian blur. ### Claim 7 (depends on 5) The method of claim 5 wherein the blurring is performed using an edge preserving blur that preserves a geodesic distance between points on curves, adaptively warping an input signal to perform 1D edge-preserving filtering in linear time. ### Claim 8 A machine-readable storage device having instructions for execution by a processor of a machine to cause the processor to perform operations to perform a method, the operations comprising: receiving a video sequence having image frames with a high dynamic range; separating each image frame into low and high frequency components; compressing the low frequency components using a non-linear brightness compression curve including tone mapping the low frequency components; reducing brightness and saturation for pixels of the low frequency components which are outside a target gamut; reconstructing the image frames of the video sequence by combining the high frequency components with the compressed and reduced brightness and saturation for pixels of the low frequency components; and rendering the reconstructed image frames of the video sequence during real time playback for display on a device having a lower dynamic range, while preserving highlight detail, wherein tone mapping the low frequency components comprises: determining a tone mapping for each successive image frame; smoothing the tone mapping to reduce visible flickering artifacts; detecting a scene change; and discontinuing smoothing the tone mapping for a first image frame of the detected scene change. ### Claim 9 (depends on 8) The device of claim 8 wherein the image frame pixels have a luminance range of 0 to up to 10,000 nits. ### Claim 10 (depends on 8) The device of claim 8 wherein the lower dynamic range comprises 0 to 200 nits. ### Claim 11 (depends on 8) The device of claim 8 wherein reconstructing the image frames of the video sequence comprises adding the high frequency components to the compressed low frequency components. ### Claim 12 (depends on 8) The device of claim 8 wherein separating each image into low and high frequency components comprises blurring each video frame image and subtracting the blurred video frame images from the received video frame images to obtain the high frequency components. ### Claim 13 (depends on 12) The device of claim 12 wherein the blurring is performed using a Gaussian blur. ### Claim 14 (depends on 12) The device of claim 12 wherein the blurring is performed using an edge preserving blur that preserves a geodesic distance between points on curves, adaptively warping an input signal to perform 1D edge-preserving filtering in linear time. ### Claim 15 A device comprising: a processor; and a memory device coupled to the processor and having a program stored thereon for execution by the processor to perform operations comprising: receiving a video sequence having image frames with a high dynamic range; separating each image frame into low and high frequency components; compressing the low frequency components using a non-linear brightness compression curve including tone mapping the low frequency components; reducing brightness and saturation for pixels of the low frequency components which are outside a target gamut; reconstructing the image frames of the video sequence by combining the high frequency components with the compressed and reduced brightness and saturation for pixels of the low frequency components; and rendering the reconstructed image frames of the video sequence during real time playback for display on a device having a lower dynamic range, while preserving highlight detail, wherein tone mapping the low frequency components comprises: determining a tone mapping for each successive image frame; smoothing the tone mapping to reduce visible flickering artifacts; detecting a scene change; and discontinuing smoothing the tone mapping for a first image frame of the detected scene change. ### Claim 16 (depends on 15) The device of claim 15 wherein the image frame pixels have a luminance range of 0 to up to 10,000 its. ### Claim 17 (depends on 15) The device of claim 15 wherein reconstructing the image frames of the video sequence comprises adding the high frequency components to the compressed low frequency components and wherein separating each image into low and high frequency components comprises blurring each video frame image and subtracting the blurred video frame images from the received video frame images to obtain the high frequency components. ### Claim 18 (depends on 17) The device of claim 17 wherein the blurring is performed using an edge preserving blur that preserves a geodesic distance between points on curves, adaptively warping an input signal to perform 1D edge-preserving filtering in linear time.