# US 2023/0143443 A1 · Systems and methods of fusing computer-generated predicted image frames with captured images frames to create a high-dynamic-range video having a high number of frames per second

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0143443 A1
- Document ID: US20230143443A1
- Status: Published application
- Kind: A1
- Application number: 17/978,914
- Filed: 2022-11-01
- Published: 2023-05-11
- Earliest claimed: 2021-11-08
- Assignees: Meta Platforms Technologies, LLC
- Applicants: Meta Platforms Technologies, LLC
- Inventors: Ingrid Anda Cotoros; Vikas CHANDRA
- Web page: https://patentgrep.com/patent/US20230143443A1

## CPC

- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/92: based on global image properties
- G06T 11/00: Two-dimensional [2D] image generation
- H04N 5/265: Mixing
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20208: High dynamic range [HDR] image processing
- G06T 2207/20221: Image fusion; Image merging
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method of using computer-generated predicted image frames to create a high-dynamic-range (HDR) video is described. A method includes receiving first and second captured image frames via an image sensor. The first captured image frame represents a scene in the real-world at a first point in time and the second captured image frame represents the scene in the real-world at a second point in time that is after the first point in time. The method further includes in accordance with a determination that the first captured image frame and the second captured image frame will be used to produce an HDR video, generating a computer-generated predicted image frame representing the scene in the real-world at a time between the first point in time and the second point in time and fusing the second image frame with the computer-generated predicted image frame to generate an HDR frame for the HDR video.

## Claims

### Claim 1

A method of using computer-generated predicted image frames to create a high-dynamic-range (HDR) video having a high number of frames per second (fps), the method comprising:
receiving, at one or more processors that are in communication with an image sensor configured to capture image frames used to produce a high-dynamic range (HDR) video, a first captured image frame and a second captured image frame captured via the image sensor, the first captured image frame representing a scene in the real-world at a first point in time and the second captured image frame representing the scene in the real-world at a second point in time that is after the first point in time;
in accordance with a determination that the first captured image frame and the second captured image frame will be used to produce an HDR video:
generating, via the one or more processors and based on the first captured image frame, a computer-generated predicted image frame representing the scene in the real-world at a time between the first point in time and the second point in time; and
fusing the second captured image frame with the computer-generated predicted image frame to generate an HDR frame for the HDR video.

### Claim 2 (depends on 1)

The method of claim 1, further comprising repeating the receiving captured image frames, the generating computer-generated predicted image frames, and the fusing computer-generated predicted image frames with captured image frames to produce respective HDR frames for the HDR video, such that the HDR video has at least 32 frames per second.

### Claim 3 (depends on 2)

The method of claim 2, wherein the HDR video includes:
(i) a first HDR frame that was created by fusing two captured image frames, and
(ii) a second HDR frame that was created by fusing two computer-generated predicted image frames.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
after producing the HDR video, receiving captured image frames captured via the image sensor and producing a non-HDR video without using any computer-generated predicted image frames, wherein the HDR video includes a first number of frames per second that is greater than or equal to a second number of frames per second for the non-HDR video.

### Claim 5 (depends on 4)

The method of claim 4, wherein the first number of frames per second and the second number of frames per second is 32 frames.

### Claim 6 (depends on 1)

The method of claim 1, wherein the computer-generated predicted image frame is generated while the second captured image frame is being captured by the image sensor.

### Claim 7 (depends on 1)

The method of claim 1, wherein the one or more processors that are in communication with the image sensor receive a third captured image frame representing the scene in the real-world at a third point in time that is after the second point in time, and the third captured image frame is captured in part while the HDR frame is being generated.

### Claim 8 (depends on 1)

The method of claim 1, wherein the computer-generated predicted image frame is a first computer-generated predicted image frame and the HDR frame is a first HDR frame, and the method further comprises:
receiving, at the one or more processors that are in communication with the image sensor, a third captured image frame captured via the image sensor, the third captured image frame representing the scene in the real-world at a third point in time that is after the second point in time; and
in accordance with a determination that the third captured image frame will be used in conjunction with the first captured image frame and the second captured image frame to produce the HDR video:
generating, via the one or more processors and based on the second captured image frame, a second computer-generated predicted image frame representing the scene in the real-world at the time between the second point in time and the third point in time; and
fusing the third image frame with the second computer-generated predicted image frame to generate a second HDR frame for the HDR video.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
receiving, at the one or more processors that are in communication with the image sensor, a third captured image frame captured via the image sensor, the third captured image frame representing the scene in the real-world at a third point in time that is after the second point in time; and
in accordance with a determination that the third captured image frame will be used in conjunction with the first captured image frame and the second captured image frame to produce the HDR video:
generating, via the one or more processors and based on the first captured image frame and the third captured image frame, the computer-generated predicted image frame representing the scene in the real-world at the time between the first point in time and the second point in time; and
fusing the computer-generated predicted image frame with the second image frame and the computer-generated predicted image frame to generate the HDR frame for the HDR video.

### Claim 10 (depends on 9)

The method of claim 9, wherein the computer-generated predicted image frame is a first computer-generated predicted image frame and the HDR frame is a first HDR frame, and the method further comprises:
receiving, at the one or more processors that are in communication with the image sensor, a fourth captured image frame captured via the image sensor, the fourth captured image frame representing the scene in the real-world at a fourth point in time that is after the third point in time; and
in accordance with a determination that the fourth captured image frame will be used in conjunction with the first captured image frame, the second captured image frame, and the third captured image frame to produce the HDR video:
generating, via the one or more processors and based on the second captured image frame and the fourth captured image frame, a second computer-generated predicted image frame representing the scene in the real-world at the time between the third point in time and the second point in time; and
fusing the third image frame with the second computer-generated predicted image frame to generate a second HDR frame for the HDR video.

### Claim 11 (depends on 1)

The method of claim 1, wherein:
the first captured image frame is a first type of image frame,
the second captured image frame is a second type of image frame, and
the first type of image frame is distinct from the second type of image frame.

### Claim 12 (depends on 11)

The method of claim 11, wherein:
the first type of image frame has a short exposure duration; and
the second type of image frame has a long exposure duration that is greater than the short exposure duration.

### Claim 13 (depends on 1)

The method of claim 1, wherein the computer-generated predicted image frame is generated via a machine-learning system that has been trained using a training set consisting of a variety of image frames captured by an image sensor viewing different scenes in the real-world.

### Claim 14 (depends on 1)

The method of claim 1, wherein the HDR video has a number of frames per second (fps) that is at least equal to a maximum fps achievable by the one or more processors when using captured image frames to produce a video.

### Claim 15 (depends on 14)

The method of claim 14, wherein the HDR video has a fps greater than a maximum fps achievable by the one or more processors when using captured image frames to produce a video.

### Claim 16 (depends on 1)

The method of claim 1, wherein the image sensor is part of a security camera, smartphone, smart watch, tablet, or AR glasses.

### Claim 17

A system for generating HDR video, comprising:
an image sensor configured to capture image frames used to produce a high-dynamic range (HDR) video; and
one or more processors that are in communication with the image sensor, the one or more processors configured to:
receive a first captured image frame and a second captured image frame captured via the image sensor, the first captured image frame representing a scene in the real-world at a first point in time and the second captured image frame representing the scene in the real-world at a second point in time that is after the first point in time;
in accordance with a determination that the first captured image frame and the second captured image frame will be used to produce an HDR video:
generate, via the one or more processors and based on the first captured image frame, a computer-generated predicted image frame representing the scene in the real-world at a time between the first point in time and the second point in time; and
fuse the second image frame with the computer-generated predicted image frame to generate an HDR frame for the HDR video.

### Claim 18

A non-transitory computer-readable storage medium including instructions that, when executed by a device that includes an image sensor, cause the device to:
receive a first captured image frame and a second captured image frame captured via the image sensor, the first captured image frame representing a scene in the real-world at a first point in time and the second captured image frame representing the scene in the real-world at a second point in time that is after the first point in time;
in accordance with a determination that the first captured image frame and the second captured image frame will be used to produce an HDR video:
generate, via the one or more processors and based on the first captured image frame, a computer-generated predicted image frame representing the scene in the real-world at a time between the first point in time and the second point in time; and
fuse the second image frame with the computer-generated predicted image frame to generate an HDR frame for the HDR video.

### Claim 19 (depends on 18)

The non-transitory computer-readable storage medium of claim 18, further including instructions that, when executed by the device, cause the device to:
repeat the receiving captured image frames, the generating computer-generated predicted image frames, and the fusing computer-generated predicted image frames with captured image frames to produce respective HDR frames for the HDR video, such that the HDR video has at least 32 frames per second.

### Claim 20 (depends on 18)

The non-transitory computer-readable storage medium of claim 18, wherein the HDR video includes:
(i) a first HDR frame that was created by fusing two captured image frames, and
(ii) a second HDR frame that was created by fusing two computer-generated predicted image frames.
