# US 2022/0044372 A1 · Image Fusion Architecture with Multimode Operations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044372 A1
- Document ID: US20220044372A1
- Status: Published application
- Kind: A1
- Application number: 16/987,210
- Filed: 2020-08-06
- Published: 2022-02-10
- Earliest claimed: 2020-08-06
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Maxim Smirnov; David R. Pope
- Web page: https://patentgrep.com/patent/US20220044372A1

## CPC

- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/73: Deblurring; Sharpening
- G06T 3/18: Image warping, e.g. rearranging pixels individually
- G06T 5/70: Denoising; Smoothing
- G06T 5/94: based on local image properties, e.g. for local contrast enhancement
- G06T 2207/10024: Color image
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20208: High dynamic range [HDR] image processing
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Embodiments relate to circuitry for temporal processing and image fusion. An image fusion circuit receives captured images, and generates corresponding image pyramids. The generated image pyramids are raster or tiled processed, and stored in memory. A fusion module receives a first and second image pyramids from the memory, and warps and fuses image pyramids to generate a fused image pyramid, which may be used for further processing, and may also be stored back into the memory. The image fusion circuitry is configurable to operate in a plurality of different configuration modes corresponding to different image fusion applications for fusing image pyramids of received images, including two-frame fusion, temporal filtering, infinite impulse response (IIR) temporal processing, and/or finite impulse response (FIR) temporal processing.

## Claims

### Claim 1

An image signal processor, comprising:
an image fusion engine comprising:
a pyramid generation circuit configured to receive a plurality of images and to generate, for each image, a respective image pyramid comprising a plurality of downscaled images by sequentially downscaling the image;
an image fusion circuit configured to fuse a first image pyramid with a second image pyramid to generate a fused image; and
a controller coupled to the image fusion engine and configured to switch the image fusion engine between a plurality of configuration modes comprising at least a first configuration mode and a second configuration mode, wherein:
in the first configuration mode, the image fusion engine is configured to fuse an image pyramid of an image of the plurality of images with an image pyramid of a history image corresponding to a fusion of a plurality of previously received images to generate the fused image, and
in the second configuration mode, the image fusion engine is configured to fuse image pyramids of a set of received images by fusing an image pyramid of a reference image of the set of received images with each of one or more remaining images of the set of received images to generate a plurality of partial fusions, and to fuse image pyramids of the plurality of partial fusions to generate the fused image.

### Claim 2 (depends on 1)

The image signal processor of claim 1, wherein the image fusion engine further comprises a warping circuit configured to, responsive to receiving an image pyramid corresponding to an image of the plurality of images, generate a warped pyramid by warping each of the plurality of downscaled images according to one or more warping parameters derived from a model describing correspondence between the image and another image.

### Claim 3 (depends on 1)

The image signal processor of claim 1, wherein image fusion engine further comprises a data routing circuit coupled to the pyramid generation circuit configured to:
receive the plurality of image pyramids from the pyramid generation circuit, and
for each of the plurality of image pyramids, select between storing the image pyramid into a memory or bypassing the memory.

### Claim 4 (depends on 3)

The image signal processor of claim 3, wherein the data routing circuit is configured to, in the first configuration mode:
bypass the memory for each the plurality of pyramids received from the pyramid generation circuit;
receive, from the image fusion circuit, a fused image pyramid corresponding to a fusion of a plurality of previously received images; and
store the received fused image pyramid in the memory as an image pyramid of the history image.

### Claim 5 (depends on 3)

The image signal processor of claim 3, wherein the data routing circuit is configured to, in the second configuration mode:
store an image pyramid corresponding to the reference image of the set of received images in the memory in a raster format; and
store image pyramids corresponding to each of the remaining images of the set of received images in the memory in a tile format.

### Claim 6 (depends on 3)

The image signal processor of claim 3, wherein the data routing circuit is configured to, for an image pyramid of the plurality of image pyramids, bypass the memory by storing the image pyramid in a cache.

### Claim 7 (depends on 1)

The image signal processor of claim 1, wherein, in the first configuration mode, the image fusion engine outputs, for each received image, a respective fused image corresponding to a fusion of an image pyramid of the received image and an image pyramid of the history image.

### Claim 8 (depends on 1)

The image signal processor of claim 1, wherein, in the first configuration mode, the image fusion engine is configured to output the fused image responsive to the image pyramid of the history image corresponding to a fusion of a predetermined number of previously received images.

### Claim 9 (depends on 1)

The image signal processor of claim 1, wherein, in the second configuration mode, the set of received images comprises N images, and the image fusion engine performs N−1 fusion operations to generate N−1 partial fusions, and N−2 fusion operations of the N−1 partial fusions to generate the fused image.

### Claim 10 (depends on 1)

The image signal processor of claim 1, wherein, in the second configuration mode, the image fusion engine is configured to store in a memory up to two image pyramids at a time, and is configured to perform a first fusion of two partial fusions of the plurality of partial fusions before performing a third fusion to generate a third partial fusion of the plurality of partial fusions.

### Claim 11

A method for image fusion, comprising:
configuring an image fusion engine to be in a first configuration mode or a second configuration mode;
receiving a plurality of images at a pyramid generation circuit of an image fusion engine, and
generating, at the pyramid generation circuit of the image fusion engine, for each of the received images, a plurality of pyramids by sequentially downscaling each of the received images;
generating, at an image fusion circuit of the image fusion engine, a fused image, by:
responsive to the image fusion engine being in the first configuration mode, fusing an image pyramid of an image of the plurality of images with an image pyramid of a history image corresponding to a fusion of a plurality of previously received images to generate the fused image
responsive to the image fusion engine being in the second configuration mode, fusing image pyramids of N received images by fusing an image pyramid of a reference image of the N received images with each of one or more remaining images of the N received images to generate N−1 partial fusions, and fusing image pyramids of the N−1 partial fusions to generate the fused image.

### Claim 12 (depends on 11)

The method of claim 11, further comprising, responsive to receiving an image pyramid corresponding to an image of the plurality of images, generating, by a warping circuit of the image fusion engine, a warped pyramid by warping each of the plurality of downscaled images according to one or more warping parameters derived from a model describing correspondence between the image and another image.

### Claim 13 (depends on 11)

The method of claim 11, further comprising, at a data routing circuit coupled to the pyramid generation circuit:
receiving the plurality of image pyramids from the pyramid generation circuit, and
for each of the plurality of image pyramids, selecting between storing the image pyramid into a memory or bypassing the memory.

### Claim 14 (depends on 13)

The method of claim 13, further comprising, by the data routing circuit, responsive to the image fusion engine being in the first configuration mode:
bypassing the memory for each the plurality of pyramids received from the pyramid generation circuit;
receiving, from the image fusion circuit, a fused image pyramid corresponding to a fusion of a plurality of previously received images; and
storing the received fused image pyramid in the memory as an image pyramid of the history image.

### Claim 15 (depends on 13)

The method of claim 13, further comprising, by the data routing circuit, responsive to the image fusion engine being in the second configuration mode:
storing an image pyramid corresponding to the reference image of the set of received images in the memory in a raster format; and
storing image pyramids corresponding to each of the remaining images of the set of received images in the memory in a tile format.

### Claim 16 (depends on 11)

The method of claim 11, further comprising, responsive to the image fusion engine being in the first configuration mode, outputting the fused image responsive to the image pyramid of the history image corresponding to a fusion of a predetermined number of previously received images.

### Claim 17 (depends on 11)

The method of claim 11, wherein the set of received images comprises N images, and further comprising, responsive to the image fusion engine being in the second configuration mode, performing N−1 fusion operations to generate N−1 partial fusions, and N−2 fusion operations of the N−1 partial fusions to generate the fused image.

### Claim 18

An electronic device, comprising:
a memory;
an image signal processor comprising:
an image fusion engine comprising:
a pyramid generation circuit configured to receive a plurality of images and to generate, for each image, a respective image pyramid comprising a plurality of downscaled images by sequentially downscaling the image;
an image fusion circuit configured to fuse a first image pyramid with a second image pyramid to generate a fused image; and
a controller configured to switch the image fusion engine between a plurality of configuration modes comprising at least a first configuration mode and a second configuration mode, wherein
in the first configuration mode, the image fusion engine is configured to fuse an image pyramid of an image of the plurality of images with an image pyramid of a history image corresponding to a fusion of a plurality of previously received images to generate the fused image, and wherein
in the second configuration mode, the image fusion engine is configured to fuse image pyramids of N received images by fusing an image pyramid of a reference image of the N received images with each of one or more remaining images of the N received images to generate N−1 partial fusions, and to fuse image pyramids of the N−1 partial fusions to generate the fused image.

### Claim 19 (depends on 18)

The electronic device of claim 18, wherein image fusion engine further comprises a data routing circuit coupled to the pyramid generation circuit configured to:
receive the plurality of image pyramids from the pyramid generation circuit, and
for each of the plurality of image pyramids, select between storing the image pyramid into a memory or bypassing the memory.

### Claim 20 (depends on 19)

The electronic device of claim 19, wherein the data routing circuit is configured to:
in the first configuration mode:
bypass the memory for each the plurality of pyramids received from the pyramid generation circuit;
receive, from the image fusion circuit, a fused image pyramid corresponding to a fusion of a plurality of previously received images; and
store the received fused image pyramid in the memory as an image pyramid of the history image; and
in the second configuration mode:
store an image pyramid corresponding to the reference image of the set of received images in the memory in a raster format; and
store image pyramids corresponding to each of the remaining images of the set of received images in the memory in a tile format.
