# US 11,451,709 B1 · Increasing dynamic range of a virtual production display

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,451,709 B1
- Document ID: US11451709B1
- Status: Granted
- Kind: B1
- Application number: 17/538,921
- Filed: 2021-11-30
- Granted: 2022-09-20
- Earliest claimed: 2021-11-29
- Assignees: Unity Technologies SF
- Applicants: Unity Technologies SF
- Inventors: Joseph W. Marks; Luca Fascione; Kimball D. Thurston, III; Millie Maier; Kenneth Gimpelson; Dejan Momcilovic; Keith F. Miller; Peter M. Hillman; Jonathan S. Swartz
- Web page: https://patentgrep.com/patent/US11451709B1

## CPC

- H04N 23/741: by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 1/60: Memory management
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06T 5/90: Dynamic range modification of images or parts thereof
- G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215)
- G06T 7/174: involving the use of two or more images
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00)
- G06V 10/22: by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
- G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model
- G06V 10/70: using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88)
- G09G 5/10: Intensity circuits
- H04N 5/04: Synchronising (for television systems using pulse code modulation H04N7/56)
- H04N 5/2224: {related to virtual studio applications}
- H04N 5/265: Mixing
- H04N 5/91: Television signal processing therefor
- H04N 5/92: Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback
- H04N 7/181: {for receiving images from a plurality of remote sources}
- H04N 7/183: {for receiving images from a single remote source}
- H04N 23/60: Control of cameras or camera modules
- H04N 23/71: Circuitry for evaluating the brightness variation
- H04N 23/72: Combination of two or more compensation controls
- H04N 23/73: by influencing the exposure time
- H04N 23/74: by influencing the scene brightness using illuminating means
- H04N 23/76: by influencing the image signals
- H04N 23/815: {for controlling the resolution by using a single image}
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20056: Discrete and fast Fourier transform, [DFT, FFT]
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20208: High dynamic range [HDR] image processing
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30244: Camera pose
- G09G 2310/0224: Details of interlacing
- G09G 2320/0626: for control of overall brightness
- G09G 2320/0686: with two or more screen areas displaying information with different brightness or colours
- G09G 2340/0435: Change or adaptation of the frame rate of the video stream
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)

## Abstract

A processor performing postprocessing obtains an input image containing both bright and dark regions. The processor obtains a threshold between a first pixel value of the virtual production display and a second pixel value of the virtual production display. The processor modifies the region according to predetermined steps producing a pattern unlikely to occur within the input image, where the pattern corresponds to a difference between the original pixel value and the threshold. The processor can replace the region of the input image with the pattern to obtain a modified image. The virtual production display can present the modified image. A processor performing postprocessing detects the pattern within the modified image displayed on the virtual production display. The processor calculates the original pixel value of the region by reversing the predetermined steps. The processor replaces the pattern in the modified image with the original pixel value.

## Claims

### Claim 1

A system comprising:
at least one hardware processor; and
at least one non-transitory memory storing instructions, which, when executed by the at least one hardware processor, cause the system to:
obtain a presented image shown on a display;
obtain a pattern to detect in the presented image;
detect the pattern within the presented image,
wherein the pattern indicates a modified region,
wherein the pattern indicates a method used to modify the modified region, and
wherein the pattern indicates an expected pixel value of the modified region,
wherein the expected pixel value is a recording of a pixel value presented on the display;
based on the pattern, calculate the expected pixel value of the modified region;
replace the pattern in the presented image with the expected pixel value; and
store the expected pixel value.

### Claim 2 (depends on 1)

The system claim 1, the instructions to calculate the expected pixel value comprising the instructions to:
obtain the pattern to detect;
based on the pattern, obtain a function to apply to the modified region; and
calculate the expected pixel value by applying the function to the modified region.

### Claim 3 (depends on 1)

The system of claim 1, the instructions to calculate the expected pixel value comprising the instructions to:
obtain a signature frequency to detect;
detect the signature frequency indicating the modified region;
determine a magnitude of the signature frequency,
wherein the magnitude of the signature frequency indicates the expected pixel value; and
based on the magnitude of the signature frequency, modify a pixel value of the modified region to obtain the expected pixel value.

### Claim 4 (depends on 1)

The system of claim 1, comprising instructions to:
record the pattern from the display;
determine a signature pixel value of the recorded pattern; and
store the signature pixel value of the recorded pattern to use in detecting the pattern within the presented image presented on the display.

### Claim 5 (depends on 1)

The system of claim 1, comprising instructions to:
detect a second pattern within the presented image,
wherein the second pattern indicates a second region surrounding the modified region,
wherein a pixel value of the second region has been increased in correspondence to a decrease in a pixel value of the modified region,
wherein the second pattern indicates a second expected pixel value associated with the second region;
based on the second pattern, calculate the second expected pixel value of the second region,
wherein the second expected pixel value is lower than the pixel value of the second region; and
replace the second pattern in the presented image with the second expected pixel value.

### Claim 6 (depends on 1)

The system of claim 1, comprising the instructions to:
train a machine learning model to detect a discrepancy between a first image and a second image,
wherein the first image excludes the pattern, and
wherein the second image includes the first image modified with the pattern; and
use the machine learning model to detect the pattern within the second image.

### Claim 7 (depends on 1)

The system of claim 1, the instructions to detect the pattern comprising the instructions to use a Fast Fourier Transform to detect the pattern within the presented image.

### Claim 8 (depends on 1)

The system of claim 1, the instructions to detect the pattern comprising the instructions to use a machine learning model to detect the pattern within the presented image.

### Claim 9 (depends on 1)

The system of claim 1, comprising the instructions to:
obtain an input indicating a technique among multiple techniques,
wherein the technique is configured to increase dynamic range of an image recorded from the display; and
apply at least a portion of the technique to the image recorded from the display.

### Claim 10

A method comprising:
obtaining a presented image shown on a display;
obtaining a pattern to detect in the presented image;
detecting the pattern within the presented image,
wherein the pattern indicates a modified region,
wherein the pattern indicates a process used to modify the modified region, and
wherein the pattern indicates an expected pixel value of the modified region,
wherein the expected pixel value is a recording of a pixel value presented on the display;
based on the pattern, calculating the expected pixel value of the modified region;
replacing the pattern in the presented image with the expected pixel value; and
storing the expected pixel value.

### Claim 11 (depends on 10)

The method of claim 10, wherein calculating the expected pixel value comprises:
obtaining the pattern to detect;
based on the pattern, obtaining a function to apply to the modified region; and
calculating the expected pixel value by applying the function to the modified region.

### Claim 12 (depends on 10)

The method of claim 10, wherein calculating the expected pixel value comprises:
obtaining a signature frequency to detect;
detecting the signature frequency indicating the modified region;
determining a magnitude of the signature frequency,
wherein the magnitude of the signature frequency indicates the expected pixel value; and
based on the magnitude of the signature frequency, modifying a pixel value of the modified region to obtain the expected pixel value.

### Claim 13 (depends on 10)

The method of claim 10, comprising:
recording the pattern from the display;
determining a signature pixel value of the recorded pattern; and
storing the signature pixel value of the recorded pattern to use in detecting the pattern within the presented image presented on the display.

### Claim 14 (depends on 10)

The method of claim 10, comprising:
detecting a second pattern within the presented image,
wherein the second pattern indicates a second region surrounding the modified region,
wherein a pixel value of the second region has been increased in correspondence to a decrease in a pixel value of the modified region,
wherein the second pattern indicates a second expected pixel value associated with the second region;
based on the second pattern, calculating the second expected pixel value of the second region,
wherein the second expected pixel value is lower than the pixel value of the second region; and
replacing the second pattern in the presented image with the second expected pixel value.

### Claim 15 (depends on 10)

The method of claim 10, comprising:
training a machine learning model to detect a discrepancy between a first image and a second image,
wherein the first image excludes the pattern, and
wherein the second image includes the first image modified with the pattern; and
using the machine learning model to detect the pattern within the second image.

### Claim 16 (depends on 10)

The method of claim 10, wherein detecting the pattern comprises the instructions to use a Fast Fourier Transform to detect the pattern within the presented image.

### Claim 17 (depends on 10)

The method of claim 10, wherein detecting the pattern comprises the instructions to use a machine learning model to detect the pattern within the presented image.

### Claim 18 (depends on 10)

The method of claim 10, comprising:
obtaining an input indicating a technique among multiple techniques,
wherein the technique is configured to increase dynamic range of an image recorded from the display; and
applying at least a portion of the technique to the image recorded from the display.

### Claim 19

At least one computer-readable storage medium, excluding transitory signals and carrying instructions, which, when executed by at least one data processor of a system, causes the system to:
obtain a presented image shown on a display;
obtain a pattern to detect in the presented image;
detect the pattern within the presented image,
wherein the pattern indicates a modified region,
wherein the pattern indicates a method used to modify the modified region, and
wherein the pattern indicates an expected pixel value of the modified region,
wherein the expected pixel value is a recording of a pixel value presented on the display;
based on the pattern, calculate the expected pixel value of the modified region;
replace the pattern in the presented image with the expected pixel value; and
store the expected pixel value.

### Claim 20 (depends on 19)

The computer-readable storage medium of claim 19, the instructions to calculate the expected pixel value comprising the instructions to:
obtain the pattern to detect;
based on the pattern, obtain a function to apply to the modified region; and
calculate the expected pixel value by applying the function to the modified region.

### Claim 21 (depends on 19)

The computer-readable storage medium of claim 19, the instructions to calculate the expected pixel value comprising the instructions to:
obtain a signature frequency to detect;
detect the signature frequency indicating the modified region;
determine a magnitude of the signature frequency,
wherein the magnitude of the signature frequency indicates the expected pixel value; and
based on the magnitude of the signature frequency, modify a pixel value of the modified region to obtain the expected pixel value.

### Claim 22 (depends on 19)

The computer-readable storage medium of claim 19, comprising instructions to:
record the pattern from the display;
determine a signature pixel value of the recorded pattern; and
store the signature pixel value of the recorded pattern to use in detecting the pattern within the presented image presented on the display.

### Claim 23 (depends on 19)

The computer-readable storage medium of claim 19, comprising instructions to:
detect a second pattern within the presented image,
wherein the second pattern indicates a second region surrounding the modified region,
wherein a pixel value of the second region has been increased in correspondence to a decrease in a pixel value of the modified region,
wherein the second pattern indicates a second expected pixel value associated with the second region;
based on the second pattern, calculate the second expected pixel value of the second region,
wherein the second expected pixel value is lower than the pixel value of the second region; and
replace the second pattern in the presented image with the second expected pixel value.

### Claim 24 (depends on 19)

The computer-readable storage medium of claim 19, comprising the instructions to:
train a machine learning model to detect a discrepancy between a first image and a second image,
wherein the first image excludes the pattern, and
wherein the second image includes the first image modified with the pattern; and
use the machine learning model to detect the pattern within the second image.

### Claim 25 (depends on 19)

The computer-readable storage medium of claim 19, the instructions to detect the pattern comprising the instructions to use a Fast Fourier Transform to detect the pattern within the presented image.

### Claim 26 (depends on 19)

The computer-readable storage medium of claim 19, the instructions to detect the pattern comprising the instructions to use a machine learning model to detect the pattern within the presented image.

### Claim 27 (depends on 19)

The computer-readable storage medium of claim 19, comprising the instructions to:
obtain an input indicating a technique among multiple techniques,
wherein the technique is configured to increase dynamic range of an image recorded from the display; and
apply at least a portion of the technique to the image recorded from the display.