# US 11,910,122 B1 · Synthesizing intermediary frames for long exposure images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,910,122 B1
- Document ID: US11910122B1
- Status: Granted
- Kind: B1
- Application number: 17/733,923
- Filed: 2022-04-29
- Granted: 2024-02-20
- Earliest claimed: 2020-05-07
- Assignees: Lux Optics Incorporated
- Applicants: Lux Optics Incorporated
- Inventors: Benjamin Keith Sandofsky; Sebastiaan Anne Laurens de With
- Web page: https://patentgrep.com/patent/US11910122B1

## CPC

- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/70: Denoising; Smoothing
- G06T 7/269: using gradient-based methods
- G06T 11/60: Creating or editing images; Combining images with text
- H04N 5/2625: {for obtaining an image which is composed of images from a temporal image sequence, e.g. for a stroboscopic effect (sequence generated by event triggered capturing H04N7/188)}
- H04N 5/265: Mixing
- H04N 23/632: {for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters}
- H04N 23/633: {for displaying additional information relating to control or operation of the camera}
- H04N 23/64: {Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image}
- H04N 23/6812: {based on additional sensors, e.g. acceleration sensors}
- H04N 23/683: {performed by a processor, e.g. controlling the readout of an image memory}
- H04N 23/684: {performed by controlling the image sensor readout, e.g. by controlling the integration time}
- H04N 23/80: Camera processing pipelines; Components thereof
- H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20221: Image fusion; Image merging
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Techniques for synthesizing intermediary frames for long exposure images are disclosed. Video stream data, comprising a plurality of arriving frames, is received. An indication that the video stream data should be processed into a long exposure image that incorporates one or more synthesized frames is received. A set of arriving frames is used to generate at least one synthesized frame. At least one received frame is blended with the at least one generated synthesized frame to form a long exposure image. The long exposure image is provided as output.

## Claims

### Claim 1

A system, comprising:
a processor configured to:
receive, in response to calling a video capture application programming interface (API) of a device, video stream data, wherein the video stream data comprises a plurality of arriving frames captured as video data during a video recording operation of the device;
receive an indication that the received video stream data should be processed into a long exposure image that incorporates one or more synthesized frames;
use a set of frames including at least one arriving frame to generate at least one synthesized frame, including by using a model trained based at least in part on a set of other long exposure images;
blend at least one received frame with the at least one generated synthesized frame to form the long exposure image; and
provide the long exposure image as output; and
a memory communication interface coupled to the processor and configured to provide the processor with instructions.

### Claim 2 (depends on 1)

The system of claim 1, wherein the plurality of arriving frames are obtained from a previously stored video file.

### Claim 3 (depends on 1)

The system of claim 1, wherein the indication is received from a user.

### Claim 4 (depends on 3)

The system of claim 3, wherein the indication is received in response to making a recommendation to the user that the long exposure image should incorporate the one or more synthesized frames.

### Claim 5 (depends on 1)

The system of claim 1, wherein the indication is received in response to applying a set of heuristics to a set of image capture environment information.

### Claim 6 (depends on 1)

The system of claim 1, wherein the indication is received in response to applying an additional trained model to a set of image capture environment information.

### Claim 7 (depends on 1)

The system of claim 1, wherein the indication is received in response to a determination of a scene associated with the video stream data.

### Claim 8 (depends on 1)

The system of claim 1, wherein the indication is received in response to an optical flow analysis.

### Claim 9 (depends on 8)

The system of claim 8, wherein the optical flow analysis is performed using a set of reduced resolution thumbnail images.

### Claim 10 (depends on 1)

The system of claim 1, wherein generating the at least one synthesized frame includes applying a video smoothing technique.

### Claim 11 (depends on 1)

The system of claim 1, wherein generating the at least one synthesized frame includes using an optical flow.

### Claim 12 (depends on 1)

The system of claim 1, wherein the processor is further configured to generate a recommendation to a user to employ the one or more synthesized frames in creating the long exposure image.

### Claim 13 (depends on 1)

The system of claim 1, wherein the set of other long exposure images incorporate one or more other synthesized frames.

### Claim 14

A method, comprising:
receiving, in response to calling a video capture application programming interface (API) of a device, video stream data, wherein the video stream data comprises a plurality of arriving frames captured as video data during a video recording operation of the device;
receiving an indication that the received video stream data should be processed into a long exposure image that incorporates one or more synthesized frames;
using a set of frames including at least one arriving frame to generate at least one synthesized frame, including by using a model trained based at least in part on a set of other long exposure images;
blending at least one received frame with the at least one generated synthesized frame to form the long exposure image; and
providing the long exposure image as output.

### Claim 15 (depends on 14)

The method of claim 14, wherein the plurality of arriving frames are obtained from a previously stored video file.

### Claim 16 (depends on 14)

The method of claim 14, wherein the indication is received from a user.

### Claim 17 (depends on 16)

The method of claim 16, wherein the indication is received in response to making a recommendation to the user that the long exposure image should incorporate the one or more synthesized frames.

### Claim 18 (depends on 14)

The method of claim 14, wherein the indication is received in response to applying a set of heuristics to a set of image capture environment information.

### Claim 19 (depends on 14)

The method of claim 14, wherein the indication is received in response to applying an additional trained model to a set of image capture environment information.

### Claim 20 (depends on 14)

The method of claim 14, wherein the indication is received in response to a determination of a scene associated with the video stream data.

### Claim 21 (depends on 14)

The method of claim 14, wherein the indication is received in response to an optical flow analysis.

### Claim 22 (depends on 21)

The method of claim 21, wherein the optical flow analysis is performed using a set of reduced resolution thumbnail images.

### Claim 23 (depends on 14)

The method of claim 14, wherein generating the at least one synthesized frame includes applying a video smoothing technique.

### Claim 24 (depends on 14)

The method of claim 14, wherein generating the at least one synthesized frame includes using an optical flow.

### Claim 25 (depends on 14)

The method of claim 14, wherein the set of other long exposure images incorporate one or more other synthesized frames.

### Claim 26 (depends on 14)

The method of claim 14, further comprising generating a recommendation to a user to employ the one or more synthesized frames in creating the long exposure image.

### Claim 27

A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:
receiving, in response to calling a video capture application programming interface (API) of a device, video stream data, wherein the video stream data comprises a plurality of arriving frames captured as video data during a video recording operation of the device;
receiving an indication that the received video stream data should be processed into a long exposure image that incorporates one or more synthesized frames;
using a set of frames including at least one arriving frame to generate at least one synthesized frame, including by using a model trained based at least in part on a set of other long exposure images;
blending at least one received frame with the at least one generated synthesized frame to form the long exposure image; and
providing the long exposure image as output.