# US 2008/0260265 A1 · Compressed domain image summation apparatus, systems, and methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2008/0260265 A1
- Document ID: US20080260265A1
- Status: Published application
- Kind: A1
- Application number: 11/738,897
- Filed: 2007-04-23
- Published: 2008-10-23
- Earliest claimed: 2007-04-23
- Assignees: Aptina Imaging Corporation
- Inventors: Amnon Silverstein; Sheng Lin; Brian Rodricks
- Web page: https://patentgrep.com/patent/US20080260265A1

## CPC

- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/10: using non-spatial domain filtering
- G06T 5/70: Denoising; Smoothing
- G06T 5/73: Deblurring; Sharpening
- G06T 7/262: using transform domain methods, e.g. Fourier domain methods
- G06T 7/37: using transform domain methods
- H04N 19/48: using compressed domain processing techniques other than decoding, e.g. modification of transform coefficients, variable length coding [VLC] data or run-length data (motion estimation in a transform domain H04N19/547; processing of decoded motion vectors H04N19/513)
- H04N 19/547: Motion estimation performed in a transform domain
- H04N 19/56: Motion estimation with initialisation of the vector search, e.g. estimating a good candidate to initiate a search
- H04N 23/70: Circuitry for compensating brightness variation in the scene
- H04N 23/741: by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
- H04N 23/76: by influencing the image signals
- H04N 23/81: for suppressing or minimising disturbance in the image signal generation
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20201: Motion blur correction
- G06T 2207/20216: Image averaging
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Apparatus, systems, and methods disclosed herein may transpose image blocks from successively-captured versions of an image according to relative movement between an image capture device and the scene being captured. The transposition may provide for alignment of the successively-captured images notwithstanding the movement. The transposed image blocks from the successive images are composited in the frequency domain by integrating frequency domain coefficients from each into a composite final image. Additional embodiments are disclosed.

## Claims

### Claim 1

An apparatus, comprising:
a strip buffer to temporarily store an image data stream associated with a new version of an image as the image data stream is received from an image sensor array (ISA);
an image comparator to compare a selected set of low-frequency coefficients of a new frequency domain (FD) image block associated with the new version of the image to a corresponding set of low-frequency coefficients of an earlier-stored composite FD image block associated with at least one previously received version of the image; and
combinational logic to combine the new FD image block with the earlier-stored composite FD image block to yield a new composite FD image block if the selected set of low-frequency coefficients of the new FD image block match the corresponding set of low-frequency coefficients of the earlier-stored composite FD image block within a match threshold.

### Claim 2 (depends on 1)

The apparatus of claim 1, further comprising:
a first domain translator coupled to the strip buffer to convert an earlier-received image block associated with a first-received image from a spatial domain to a frequency domain to yield an earlier-received FD image block.

### Claim 3 (depends on 2)

The apparatus of claim 2, further comprising:
a first entropy encoder operatively coupled to the first domain translator to entropy-encode the earlier-received FD image block to yield an entropy-encoded earlier-received FD image block and to store the entropy-encoded earlier-received FD image block in a compressed image buffer as an entropy-encoded earlier-stored composite FD image block.

### Claim 4 (depends on 1)

The apparatus of claim 1, further comprising:
a second domain translator coupled to the strip buffer to convert a block of pixel illuminance values selected from the image data stream from a spatial domain (SD) to a frequency domain using an SD to FD transform operation to yield the new FD image block.

### Claim 5 (depends on 1)

The apparatus of claim 1, further comprising:
averaging logic coupled to the combinational logic to compute a weighted average of an FD coefficient from the new FD image block and a corresponding FD coefficient from the earlier-stored composite FD image block, wherein a weight associated with the FD coefficient from the new FD image block comprises a value of one and wherein a weight associated with the corresponding FD coefficient from the earlier-stored composite FD image block comprises a number equal to a number of image frames composited into the earlier-stored composite FD image block.

### Claim 6 (depends on 1)

The apparatus of claim 1, further comprising:
a second entropy encoder coupled to the combinational logic to entropy-encode the new composite FD image block to yield an entropy-encoded new composite FD image block and to store the entropy-encoded new composite FD image block in a compressed image buffer.

### Claim 7 (depends on 1)

The apparatus of claim 1, further comprising:
a first entropy decoder coupled to the image comparator to perform an entropy decoding operation on an entropy-encoded earlier-stored composite FD image block to yield the earlier-stored composite FD image block.

### Claim 8 (depends on 1)

The apparatus of claim 1, further comprising:
transpositional logic operatively coupled to the strip buffer to perform a spatial transposition operation on a set of coordinates associated with a pixel within the earlier-stored composite FD image block to find a set of transposed coordinates associated with a corresponding pixel from the new version of the image.

### Claim 9 (depends on 8)

The apparatus of claim 8, further comprising:
an intra-block coordinate interpolator coupled to the transpositional logic to read a set of block positional coordinates associated with the earlier-stored composite FD image block and to determine the set of coordinates associated with the pixel within the earlier-stored composite FD image block, wherein each coordinate of the set of coordinates is calculated as an offset from a corresponding block positional coordinate.

### Claim 10 (depends on 8)

The apparatus of claim 8, further comprising:
an affine matrix calculator coupled to the transpositional logic to generate an affine matrix such that a set of coordinates of a set of starting points associated with at least one vector multiplied by the affine matrix results in a set of ending points associated with the at least one vector, wherein the at least one vector represents a path between a position in a frame of a previously-received image block associated with the at least one previously received version of the image and a position in the frame of a corresponding subsequently-received image block associated with the new version of the image.

### Claim 11 (depends on 1)

The apparatus of claim 1, further comprising:
motion matching logic operatively coupled to the strip buffer to calculate a vector between a position in a frame of a spatial domain (SD) version of the earlier-stored composite FD image block and a position in the frame of a matching image block found in a search window associated with the new version of the image and to store the vector in a vector table if a strength of a match between the SD version of the earlier-stored composite FD image block and the matching image block is above a selected match threshold.

### Claim 12 (depends on 11)

The apparatus of claim 11, further comprising:
a search window buffer coupled to the strip buffer to store a set of pixels associated with the search window.

### Claim 13 (depends on 11)

The apparatus of claim 11, further comprising:
image feature strength logic operatively coupled to the motion matching logic to scan a selected area of a compressed image buffer to find a highest strength earlier-stored composite FD image block with an image feature strength metric value higher than any other FD image block from the selected area by calculating an image feature strength metric value associated with the earlier-stored composite FD image block using a set of low-frequency coefficients from the earlier-stored composite FD image block; and
a region buffer associated with the selected area to store the highest strength earlier-stored composite FD image block.

### Claim 14 (depends on 13)

The apparatus of claim 13, further comprising:
a second entropy decoder coupled to the image feature strength logic to load an entropy-encoded earlier-stored composite FD image block from the compressed image buffer and to perform an entropy decoding operation on at least a high-order portion of the entropy-encoded earlier-stored composite FD image block to reveal the set of low-frequency coefficients of the earlier-stored composite FD image block.

### Claim 15

A method, comprising:
comparing a selected set of low-frequency coefficients of a new frequency domain (FD) image block from an image data stream associated with a new version of an image to a corresponding set of low-frequency coefficients of an earlier-stored composite FD image block associated with at least one previously received version of the image; and
combining the new FD image block with the earlier-stored composite FD image block to yield a new composite FD image block if the set of low-frequency coefficients of the new FD image block match the set of low-frequency coefficients of the earlier-stored composite FD image block within a match threshold.

### Claim 16 (depends on 15)

The method of claim 15, further comprising:
receiving sequential portions of an image data stream from an image sensor array (ISA); and
storing the sequential portions of the image data stream in a strip buffer as the portions are received from the ISA.

### Claim 17 (depends on 15)

The method of claim 15, wherein the entropy-encoded earlier-stored composite FD image block comprises a joint photographic experts group (JPEG) image block.

### Claim 18 (depends on 15)

The method of claim 15, further comprising:
calculating a weighted average of an FD coefficient from the new FD image block and a corresponding FD coefficient from the earlier-stored composite FD image block, wherein a weight associated with the FD coefficient from the new FD image block comprises a value of one and wherein a weight associated with the corresponding FD coefficient from the earlier-stored composite FD image block comprises a number equal to a number of image frames composited into the earlier-stored composite FD image block; and
using the weighted average to combine the new FD image block with the earlier-stored composite FD image block.

### Claim 19 (depends on 15)

The method of claim 15, wherein the earlier-stored composite FD image block is associated with at least one previously-captured frame of an image and wherein the new FD image block is associated with a next-captured frame of the image.

### Claim 20 (depends on 15)

The method of claim 15, further comprising:
performing an entropy decoding operation on an entropy-encoded earlier-stored composite FD image block to yield the earlier-stored composite FD image block.

### Claim 21 (depends on 15)

The method of claim 15, further comprising:
reading a set of block positional coordinates associated with the earlier-stored composite FD image block; and
determining a set of pixel coordinates associated with at least one pixel within the earlier-stored composite FD image block, the set of pixel coordinates calculated as offsets from the set of block positional coordinates.

### Claim 22 (depends on 21)

The method of claim 21, further comprising:
performing a spatial transposition operation on the set of coordinates associated with the at least one pixel from the earlier-stored composite FD image block to find a set of coordinates associated with a set of pixels from the image data stream such that the set of pixels from the image data stream are in approximate spatial alignment with pixels associated with a spatial domain version of the earlier-stored composite FD image block; and
selecting the set of pixels from the image data stream to form a new image block.

### Claim 23 (depends on 15)

The method of claim 15, further comprising:
calculating an affine matrix such that a set of coordinates of a set of starting points associated with at least one vector multiplied by the affine matrix results in a set of ending points associated with the at least one vector, wherein the at least one vector represents a path between a position in a frame of a previously-received image block and a corresponding position in the frame of a subsequently-received image block.

### Claim 24 (depends on 15)

The method of claim 15, further comprising:
entropy-encoding the new composite FD image block to yield an entropy-encoded new composite FD image block; and
storing the entropy-encoded new composite FD image block in a compressed image buffer.

### Claim 25 (depends on 15)

The method of claim 15, further comprising:
discarding the new FD image block if the set of low-frequency coefficients of the new FD image block do not match the set of low-frequency coefficients of the earlier-stored composite FD image block within the match threshold, leaving the earlier-stored composite FD image block as the new composite FD image block.

### Claim 26 (depends on 15)

The method of claim 15, further comprising:
loading an entropy-encoded earlier-stored composite FD image block from a compressed image buffer;
performing an entropy decoding operation on at least a high-order portion of the entropy-encoded earlier-stored composite FD image block to reveal a set of low-frequency coefficients of the earlier-stored composite FD image block; and
calculating an image feature strength metric value associated with the earlier-stored composite FD image block using the set of low-frequency coefficients.

### Claim 27 (depends on 15)

The method of claim 15, further comprising:
calculating a vector between a position in a frame of a spatial domain (SD) version of the selected composite FD image block and a position in the frame of a matching image block found in a search window of an image represented by the image data stream; and
storing the vector in a vector table if a strength of a match between the SD version of the selected composite FD image block and the matching image block is above a selected match threshold.

### Claim 28 (depends on 27)

The method of claim 27, further comprising:
performing an FD to SD transform operation on the selected composite FD image block to obtain the SD version of the selected composite FD image block;
conducting a search of the search window for a candidate matching image block;
measuring a strength of a match between the SD version of the selected composite FD image block and the candidate matching image block; and
selecting the candidate matching image block as the matching image block if the strength of the match between the SD version of the selected composite FD image block and the candidate matching image block is greater than a strength of a match associated with any other candidate matching image block found in the search window.

### Claim 29 (depends on 28)

The method of claim 28, further comprising:
utilizing at least one of an inverse of a sum of absolute differences operation or an inverse of a sum of squares of an absolute difference operation to measure a match strength characteristic.

### Claim 30 (depends on 28)

The method of claim 28, further comprising:
establishing a set of origination coordinates of the vector as a set of coordinates associated with a pixel in the SD version of the selected composite FD image block; and
establishing a set of destination coordinates of the vector as a set of coordinates associated with a pixel in the matching image block, wherein the pixel in the matching image block is in spatial correspondence with the pixel in the selected composite FD image block.

### Claim 31 (depends on 28)

The method of claim 28, further comprising:
loading a set of high-quality vectors from a table of vectors; and
initiating the search within the search window at an image block closest to a location corresponding to a median value of a set of destination coordinates associated with the set of high-quality vectors.
