# US 12,183,021 B2 · High dynamic range viewpoint synthesis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,183,021 B2
- Document ID: US12183021B2
- Status: Granted
- Kind: B2
- Application number: 17/886,788
- Filed: 2022-08-12
- Published: 2024-02-15
- Granted: 2024-12-31
- Earliest claimed: 2022-08-12
- Assignees: Varjo Technologies Oy
- Applicants: Varjo Technologies Oy
- Inventors: Mikko Ollila
- Web page: https://patentgrep.com/patent/US12183021B2

## CPC

- G06T 7/55: from multiple images
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20208: High dynamic range [HDR] image processing
- G06T 2207/20221: Image fusion; Image merging
- G06V 10/243: {by compensating for image skew or non-uniform image deformations}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An imaging system including processor(s) and data repository. Processor(s) are configured to: receive images of region of real-world environment that are captured by cameras using at least one of: different exposure times, different sensitivities, different apertures; receive depth maps of region that are generated by depth-mapping means; identify different portions of each image that represent objects located at different optical depths; create set of depth planes corresponding to each image; warp depth planes of each set to match perspective of new viewpoint corresponding to which output image is to be generated; fuse sets of warped depth planes corresponding to two or more images to form output set of warped depth planes; and generate output image from output set of warped depth planes.

## Claims

### Claim 1

An imaging system comprising at least one processor and a data repository communicably coupled to the at least one processor, wherein the at least one processor is configured to:
receive a plurality of images of a given region of a real-world environment that are captured by a plurality of cameras, wherein at least two images from amongst the plurality of images are captured using at least one of: different exposure times, different sensitivities, different apertures;
receive corresponding depth maps of the given region of the real-world environment with respect to different viewpoints of the plurality of cameras, wherein the corresponding depth maps are generated by depth-mapping means corresponding to each of the plurality of cameras;
identify different portions of each image that represent objects located at different optical depths with respect to a viewpoint of a given camera that captured said image;
create a set of depth planes corresponding to each image, based on the different optical depths of the objects and relative positions of the different portions of said image representing the objects, wherein a given depth plane is representative of objects or parts of the objects that are located at a same optical depth or within a predefined depth from the same optical depth;
warp depth planes of each set to match a perspective of a new viewpoint corresponding to which an output image is to be generated;
fuse sets of warped depth planes corresponding to two or more images from amongst the plurality of images to form an output set of warped depth planes; and
generate the output image from the output set of warped depth planes.

### Claim 2 (depends on 1)

The imaging system of claim 1, wherein when fusing the sets of warped depth planes, the at least one processor is configured to:
for a given warped depth plane of a given set, select a warped depth plane from amongst warped depth planes of another set whose optical depth is closest to an optical depth of the given warped depth plane; and
fuse the given warped depth plane of the given set with the warped depth plane of the another set to generate a corresponding warped depth plane of the output set.

### Claim 3 (depends on 1)

The imaging system of claim 1, wherein the at least one processor is configured to:
mark the set of depth planes corresponding to each image with information indicative of at least one of: an exposure time, a sensitivity, an aperture used to capture said image; and
select the two or more images from amongst the plurality of images, based on said information.

### Claim 4 (depends on 1)

The imaging system of claim 1, wherein the at least one processor is configured to:
compute weights for pixels of the two or more images whose corresponding sets of warped depth planes are to be fused; and
fuse the sets of warped depth planes to form the output set of warped depth planes, based on the weights computed for the pixels.

### Claim 5 (depends on 4)

The imaging system of claim 4, wherein the weights are computed based on at least one of: brightness, sensitivity, saturation, well-exposedness, contrast, saliency, of the pixels.

### Claim 6 (depends on 4)

The imaging system of claim 4, wherein the weights are computed based on ranking of two or more cameras that captured respective ones of the two or more images.

### Claim 7 (depends on 1)

The imaging system of claim 1, wherein the at least one processor is configured to:
rank the plurality of cameras, based on at least one of: a difference between a viewpoint of a given camera and the new viewpoint, whether there is an overlap between a focusing distance range of the given camera and a focusing distance range of a neural network that is to be employed for processing the output image, an extent of the overlap, an image resolution of the given camera, a focus plane of the given camera, noise characteristics of the given camera, colour characteristics of the given camera, how recent is an image captured by the given camera; and
select two or more cameras from amongst the plurality of cameras that captured respective ones of the two or more images whose corresponding sets of warped depth planes are to be fused, based on the ranking of the plurality of cameras.

### Claim 8 (depends on 1)

The imaging system of claim 1, wherein the at least one processor is configured to:
identify occluded pixels in each warped depth plane, based on colour values and alpha values of pixels of said warped depth plane;
remove the occluded pixels from each warped depth plane; and
fuse the sets of warped depth planes to form the output set of warped depth planes, by utilising non-occluded pixels in the warped depth planes.

### Claim 9 (depends on 1)

The imaging system of claim 1, wherein the at least one processor is configured to inpaint empty pixels in the output image.

### Claim 10

A computer-implemented method comprising:
receiving a plurality of images of a given region of a real-world environment that are captured by a plurality of cameras, wherein at least two images from amongst the plurality of images are captured using at least one of: different exposure times, different sensitivities, different apertures;
receiving corresponding depth maps of the given region of the real-world environment with respect to different viewpoints of the plurality of cameras, wherein the corresponding depth maps are generated by depth-mapping means corresponding to each of the plurality of cameras;
identifying different portions of each image that represent objects located at different optical depths with respect to a viewpoint of a given camera that captured said image;
creating a set of depth planes corresponding to each image, based on the different optical depths of the objects and relative positions of the different portions of said image representing the objects, wherein a given depth plane is representative of objects or parts of the objects that are located at a same optical depth or within a predefined depth from the same optical depth;
warping depth planes of each set for matching a perspective of a new viewpoint corresponding to which an output image is to be generated;
fusing sets of warped depth planes corresponding to two or more images from amongst the plurality of images for forming an output set of warped depth planes; and
generating the output image from the output set of warped depth planes.

### Claim 11 (depends on 10)

The method of claim 10, wherein the step of fusing the sets of warped depth planes comprises:
for a given warped depth plane of a given set, selecting a warped depth plane from amongst warped depth planes of another set whose optical depth is closest to an optical depth of the given warped depth plane; and
fusing the given warped depth plane of the given set with the warped depth plane of the another set for generating a corresponding warped depth plane of the output set.

### Claim 12 (depends on 10)

The method of claim 10, further comprising:
marking the set of depth planes corresponding to each image with information indicative of at least one of: an exposure time, a sensitivity, an aperture used to capture said image; and
selecting the two or more images from amongst the plurality of images, based on said information.

### Claim 13 (depends on 10)

The method of claim 10, further comprising:
computing weights for pixels of the two or more images whose corresponding sets of warped depth planes are to be fused; and
fusing the sets of warped depth planes for forming the output set of warped depth planes, based on the weights computed for the pixels.

### Claim 14 (depends on 13)

The method of claim 13, wherein the weights are computed based on at least one of: brightness, sensitivity, saturation, well-exposedness, contrast, saliency, of the pixels.

### Claim 15 (depends on 13)

The method of claim 13, wherein the weights are computed based on ranking of two or more cameras that captured respective ones of the two or more images.

### Claim 16 (depends on 10)

The method of claim 10, further comprising:
ranking the plurality of cameras, based on at least one of: a difference between a viewpoint of a given camera and the new viewpoint, whether there is an overlap between a focusing distance range of the given camera and a focusing distance range of a neural network that is to be employed for processing the output image, an extent of the overlap, an image resolution of the given camera, a focus plane of the given camera, noise characteristics of the given camera, colour characteristics of the given camera, how recent is an image captured by the given camera; and
selecting two or more cameras from amongst the plurality of cameras that captured respective ones of the two or more images whose corresponding sets of warped depth planes are to be fused, based on the ranking of the plurality of cameras.

### Claim 17 (depends on 10)

The method of claim 10, further comprising:
identifying occluded pixels in each warped depth plane, based on colour values and alpha values of pixels of said warped depth plane;
removing the occluded pixels from each warped depth plane; and
fusing the sets of warped depth planes for forming the output set of warped depth planes, by utilising non-occluded pixels in the warped depth planes.

### Claim 18 (depends on 10)

The method of claim 10, further comprising inpainting empty pixels in the output image.