# US 11,451,708 B1 · Increasing dynamic range of a virtual production display

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,451,708 B1
- Document ID: US11451708B1
- Status: Granted
- Kind: B1
- Application number: 17/539,072
- Filed: 2021-11-30
- Granted: 2022-09-20
- Earliest claimed: 2021-11-29
- Assignees: Unity Technologies SF
- Applicants: Unity Technologies SF
- Inventors: Joseph W. Marks; Luca Fascione; Kimball D. Thurston, III; Millie Maier; Kenneth Gimpelson; Dejan Momcilovic; Keith F. Miller; Peter M. Hillman; Jonathan S. Swartz
- Web page: https://patentgrep.com/patent/US11451708B1

## CPC

- H04N 23/741: by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 1/60: Memory management
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06T 5/90: Dynamic range modification of images or parts thereof
- G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215)
- G06T 7/174: involving the use of two or more images
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00)
- G06V 10/22: by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
- G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model
- G06V 10/70: using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88)
- G09G 5/10: Intensity circuits
- H04N 5/04: Synchronising (for television systems using pulse code modulation H04N7/56)
- H04N 5/2224: {related to virtual studio applications}
- H04N 5/265: Mixing
- H04N 5/91: Television signal processing therefor
- H04N 5/92: Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback
- H04N 7/181: {for receiving images from a plurality of remote sources}
- H04N 7/183: {for receiving images from a single remote source}
- H04N 23/60: Control of cameras or camera modules
- H04N 23/71: Circuitry for evaluating the brightness variation
- H04N 23/72: Combination of two or more compensation controls
- H04N 23/73: by influencing the exposure time
- H04N 23/74: by influencing the scene brightness using illuminating means
- H04N 23/76: by influencing the image signals
- H04N 23/815: {for controlling the resolution by using a single image}
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20056: Discrete and fast Fourier transform, [DFT, FFT]
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20208: High dynamic range [HDR] image processing
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30244: Camera pose
- G09G 2310/0224: Details of interlacing
- G09G 2320/0626: for control of overall brightness
- G09G 2320/0686: with two or more screen areas displaying information with different brightness or colours
- G09G 2340/0435: Change or adaptation of the frame rate of the video stream
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)

## Abstract

A processor calibrates the camera by presenting an input image on the display to obtain a presented image. The camera, arbitrarily positioned relative to the display, records the presented image. The processor obtains the input image via a channel different from the display. The processor obtains an indication of a display region associated with the display. The processor determines an input image region corresponding to the display region, and a recorded image region corresponding to the display region. The processor obtains a first pixel value associated with the input image region and a second pixel value associated with the recorded image region. The processor determines a mapping between the first pixel value and the second pixel value, where applying the mapping to the second pixel value substantially produces the first pixel value. The processor stores an identifier associated with the recorded image region and the mapping.

## Claims

### Claim 1

A method to calibrate an interaction between a display and a camera comprising:
presenting an input image on the display to obtain a presented image;
recording, by the camera, the presented image to obtain a recorded image,
wherein the camera is arbitrarily positioned relative to the display;
obtaining the input image via a channel different from the display;
obtaining an indication of a display region associated with the display;
determining an input image region corresponding to the display region, and a recorded image region corresponding to the display region;
obtaining a first pixel value associated with the input image region and a second pixel value associated with the recorded image region,
wherein a pixel value is a set of numbers defining hue, saturation and intensity;
determining a mapping between the first pixel value and the second pixel value,
wherein applying the mapping to the second pixel value substantially produces the first pixel value; and
storing an identifier associated with the recorded image region and the mapping.

### Claim 2 (depends on 1)

The method of claim 1, comprising:
determining a presented image region corresponding to the display region;
obtaining a third pixel value associated with the presented image region;
determining a second mapping between the first pixel value and the third pixel value,
wherein applying the second mapping to the third pixel value substantially produces the first pixel value;
obtaining a desired mapping between the first pixel value and the third pixel value,
wherein the desired mapping indicates a range of values associated with the third pixel value;
determining whether the desired mapping includes the second mapping; and
upon determining the desired mapping does not include the second mapping, adjusting the display region.

### Claim 3 (depends on 2)

The method of claim 2, wherein obtaining the desired mapping comprises:
obtaining a third mapping associated with a third presented image region neighboring the presented image region; and
determining the desired mapping by computing the range of values based on the third mapping,
wherein the desired mapping produces a smooth transition between the presented image region and the third presented image region.

### Claim 4 (depends on 2)

The method of claim 2, wherein obtaining the desired mapping comprises:
obtaining a location of display region on the display; and
based on the location, determining the desired mapping.

### Claim 5 (depends on 1)

The method of claim 1, comprising:
obtaining a relative position and a relative orientation between the display and the camera;
determining a correspondence indicating how the mapping varies based on the relative position and the relative orientation; and
storing the identifier associated with the recorded image region, the mapping, and the correspondence.

### Claim 6 (depends on 1)

The method of claim 1, comprising:
recording, by the camera, a second presented image presented on the display to obtain a second recorded image;
retrieving the identifier associated with the recorded image region and the mapping;
based on the identifier determining a second recorded image region associated with the second recorded image;
applying the mapping to the second recorded image region to obtain an adjusted recorded image region; and
storing the adjusted recorded image region.

### Claim 7

A system comprising:
at least one hardware processor; and
at least one non-transitory memory storing instructions, which, when executed by the at least one hardware processor, cause the system to:
present an input image on a display to obtain a presented image;
record, by a camera, the presented image to obtain a recorded image,
wherein the camera is arbitrarily positioned relative to the display;
obtain the input image via a channel different from the display;
obtain an indication of a display region associated with the display;
determine an input image region corresponding to the display region, and a recorded image region corresponding to the display region;
obtain a first pixel value associated with the input image region and a second pixel value associated with the recorded image region,
wherein a pixel value is a set of numbers defining hue, saturation and intensity;
determine a mapping between the first pixel value and the second pixel value,
wherein applying the mapping to the second pixel value substantially produces the first pixel value; and
store an identifier associated with the recorded image region and the mapping.

### Claim 8 (depends on 7)

The system of claim 7, comprising instructions to:
determine a presented image region corresponding to the display region;
obtain a third pixel value associated with the presented image region;
determine a second mapping between the first pixel value and the third pixel value,
wherein applying the second mapping to the third pixel value substantially produces the first pixel value;
obtain a desired mapping between the first pixel value and the third pixel value,
wherein the desired mapping indicates a range of values associated with the third pixel value;
determine whether the desired mapping includes the second mapping; and
upon determining the desired mapping does not include the second mapping, adjust the display region.

### Claim 9 (depends on 8)

The system of claim 8, wherein instructions to obtain the desired mapping comprise instructions to:
obtain a third mapping associated with a third presented image region neighboring the presented image region; and
determine the desired mapping by computing the range of values based on the third mapping,
wherein the desired mapping produces a smooth transition between the presented image region and the third presented image region.

### Claim 10 (depends on 8)

The system of claim 8, wherein instructions to obtain the desired mapping comprise instructions to:
obtain a location of display region on the display; and
based on the location, determine the desired mapping.

### Claim 11 (depends on 8)

The system of claim 8, wherein the desired mapping indicates a color profile associated with the display region.

### Claim 12 (depends on 7)

The system of claim 7, comprising instructions to:
obtain a relative position and a relative orientation between the display and the camera;
determine a correspondence indicating how the mapping varies based on the relative position and the relative orientation; and
store the identifier associated with the recorded image region, the mapping, and the correspondence.

### Claim 13 (depends on 7)

The system of claim 7, comprising instructions to:
record, by the camera, a second presented image presented on the display to obtain a second recorded image;
retrieve the identifier associated with the recorded image region and the mapping;
based on the identifier determine a second recorded image region associated with the second recorded image;
apply the mapping to the second recorded image region to obtain an adjusted recorded image region; and
store the adjusted recorded image region.

### Claim 14

At least one computer-readable storage medium, excluding transitory signals and carrying instructions, which, when executed by at least one data processor of a system, cause the system to:
present an input image on a display to obtain a presented image;
record, by a camera, the presented image to obtain a recorded image,
wherein the camera is arbitrarily positioned relative to the display;
obtain the input image via a channel different from the display;
obtain an indication of a display region associated with the display;
determine an input image region corresponding to the display region, and a recorded image region corresponding to the display region;
obtain a first pixel value associated with the input image region and a second pixel value associated with the recorded image region,
wherein a pixel value is a set of numbers defining hue, saturation and intensity;
determine a mapping between the first pixel value and the second pixel value,
wherein applying the mapping to the second pixel value substantially produces the first pixel value; and
store an identifier associated with the recorded image region and the mapping.

### Claim 15 (depends on 14)

The computer-readable medium of claim 14, comprising instructions to:
determine a presented image region corresponding to the display region;
obtain a third pixel value associated with the presented image region;
determine a second mapping between the first pixel value and the third pixel value,
wherein applying the second mapping to the third pixel value substantially produces the first pixel value;
obtain a desired mapping between the first pixel value and the third pixel value,
wherein the desired mapping indicates a range of values associated with the third pixel value;
determine whether the desired mapping includes the second mapping; and
upon determining the desired mapping does not include the second mapping, adjust the display region.

### Claim 16 (depends on 15)

The computer-readable medium of claim 15, wherein instructions to obtain the desired mapping comprise instructions to:
obtain a third mapping associated with a third presented image region neighboring the presented image region; and
determine the desired mapping by computing the range of values based on the third mapping,
wherein the desired mapping produces a smooth transition between the presented image region and the third presented image region.

### Claim 17 (depends on 15)

The computer-readable medium of claim 15, wherein instructions to obtain the desired mapping comprise instructions to:
obtain a location of display region on the display; and
based on the location, determine the desired mapping.

### Claim 18 (depends on 15)

The computer-readable medium of claim 15, wherein the desired mapping indicates a color profile associated with the display region.

### Claim 19 (depends on 14)

The computer-readable medium of claim 14, comprising instructions to:
obtain a relative position and a relative orientation between the display and the camera;
determine a correspondence indicating how the mapping varies based on the relative position and the relative orientation; and
store the identifier associated with the recorded image region, the mapping, and the correspondence.

### Claim 20 (depends on 14)

The computer-readable medium of claim 14, comprising instructions to:
record, by the camera, a second presented image presented on the display to obtain a second recorded image;
retrieve the identifier associated with the recorded image region and the mapping;
based on the identifier determine a second recorded image region associated with the second recorded image;
apply the mapping to the second recorded image region to obtain an adjusted recorded image region; and
store the adjusted recorded image region.