# US 8,570,442 B2 · Hyperspectral image reconstruction via a compressed sensing framework

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 8,570,442 B2
- Document ID: US8570442B2
- Status: Granted
- Kind: B2
- Application number: 13/180,895
- Filed: 2011-07-12
- Published: 2013-01-17
- Granted: 2013-10-29
- Earliest claimed: 2011-07-12
- Assignees: Xerox Corporation
- Applicants: Lalit Keshav Mestha; Alvaro Enrique Gil
- Inventors: Lalit Keshav Mestha; Alvaro Enrique Gil
- Web page: https://patentgrep.com/patent/US8570442B2

## CPC

- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 2207/10036: Multispectral image; Hyperspectral image
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

What is disclosed is a system and method for image reconstruction using a compressed sensing framework to increase the number of wavelength bands in hyperspectral video systems. The present method utilizes a restricted representation matrix and sampling matrix to reconstruct bands to a very large number without losing information content. Reference multi-band image vectors are created and those vectors are processed in a block-wise form to obtain custom orthonormal representation matrices. A sampling matrix is also constructed offline in the factory. The compressed sensing protocol is applied using a l1-norm optimization (or relaxation) algorithm to reconstruct large number of wavelength bands with each band being interspersed within the band of interest that are not imaged. The teaching hereof leads to very large number of bands without increasing the hardware cost.

## Claims

### Claim 1

A method for image reconstruction using a compressed sensing framework for a hyperspectral video system, the method comprising:
constructing, for each of a plurality of sub-data cubes, a N×N representation matrix ψ, each of said sub-data cubes containing a 2D array of pixels and collectively comprising a first hyperspectral image data cube constructed from a first set of different spectral planes of a hyperspectral image captured using a reference sensor, where N is a total number of bands in said reference sensor, wherein constructing each representation matrix comprises:
creating a matrix of random numbers;
deriving a set of natural basis vectors in a spectral direction;
selecting a first few of said natural basis vectors to restrict said representation matrix to a space of interest;
replacing a first few columns of said random number matrix with said selected first few natural basis vectors; and
performing Gram-Schmidt ortho-normalization to obtain said representation matrix;
receiving a M×N sampling matrix Φ comprising a non-square matrix filled with 1's at locations corresponding to the peak wavelengths of filters in a multi-filter grid of a target hyperspectral camera and with remaining elements filled with 0's, where M a number of bands in said target camera such that M<<N; and
using said constructed representation matrices and said sampling matrix to reconstruct a full signal ƒ* of a hyperspectral image captured by said target hyperspectral camera.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
capturing a second set of different spectral planes using said target hyperspectral camera;
constructing a second hyperspectral image data cube from said second set of spectral planes; and
partitioning said second hyperspectral image data cube into a plurality of sub-data cubes each containing a 2D array of pixels.

### Claim 3 (depends on 2)

The method of claim 2, wherein reconstructing said full signal ƒ* of said hyperspectral image comprises:
for all sub-data cubes:
for all pixels in the current sub-data cube:
performing L1-norm minimization to obtain a vector x* using the representation matrix ψ corresponding to this sub-data cube; and
using said vector x* and said corresponding representation matrix ψ to reconstruct said full signal ƒ*, such that: ƒ*=ψx*.

### Claim 4 (depends on 3)

The method of claim 3, wherein L1-norm minimization comprises solving, for every pixel in each spectral plane, a constrained minimization given by:
x*=min∥x∥L1, such that Ax=y,
where matrix A is a product of Φ and ψ, x is a column vector of size N×1, and y is equal to the intensity vector for a given pixel in a sub-data cube from said target hyperspectral image.

### Claim 5 (depends on 4)

The method of claim 4, wherein said constrained minimization is solved using any of: a LASSO (least absolute shrinkage and selection operator) algorithm, and a Basis Pursuit algorithm.

### Claim 6 (depends on 2)

The method of claim 2, further comprising, for each sub-data cube obtained from having partitioned said second hyperspectral image data cube, filtering said second set of spectral planes in a spectral direction of at least one filter of said target camera's hyperspectral image data cube.

### Claim 7 (depends on 2)

The method of claim 2, further comprising using a Gram-Schmidt Orthogonalization to restrict matrix ψ to said space of interest.

### Claim 8 (depends on 1)

The method of claim 1, wherein said target hyperspectral camera has a multi-filter grid for multi-band capture of different spectral planes at wavelengths of interest with each filter being tuned to a specified wavelength of interest in the electromagnetic spectrum.

### Claim 9 (depends on 2)

The method of claim 2, wherein the 2D spatial resolutions of said first hyperspectral image data cube is the same as the 2D spatial resolution of said second hyperspectral image data cube.

### Claim 10 (depends on 2)

The method of claim 2, wherein, in response to the 2D spatial resolutions of said first hyperspectral image data cube not being the same as the 2D spatial resolutions of said second hyperspectral image data cube, resizing said image to match said resolutions.

### Claim 11

A hyperspectral video system for image reconstruction using a compressed sensing framework, the system comprising:
a reference sensor having a total of N bands;
a target hyperspectral camera having a multi-filter grid of a total of M bands where M<<N;
a memory and a storage medium; and
a processor in communication with said storage medium and said memory, said processor executing machine readable instructions for performing:
constructing, for each of a plurality of sub-data cubes, a N×N representation matrix ψ, each of said sub-data cubes containing a 2D array of pixels and collectively comprising a first hyperspectral image data cube constructed from a first set of different spectral planes of a hyperspectral image captured using said reference sensor, wherein constructing each representation matrix comprises:
creating a matrix of random numbers;
deriving a set of natural basis vectors in a spectral direction;
selecting a first few of said natural basis vectors to restrict said representation matrix to a space of interest;
replacing a first few columns of said random number matrix with said selected first few natural basis vectors; and
performing Gram-Schmidt ortho-normalization to obtain said representation matrix;
retrieving, from said memory, a M×N sampling matrix φ comprising a non-square matrix filled with 1's at locations corresponding to the peak wavelengths of filters in said multi-filter grid and with remaining elements filled with 0's;
using said constructed representation matrices and said sampling matrix to reconstruct a full signal ƒ* of a hyperspectral image captured by said target hyperspectral camera; and
storing said reconstructed hyperspectral image to said storage medium.

### Claim 12 (depends on 11)

The system of claim 11, further comprising:
capturing a second set of different spectral planes using said target hyperspectral camera;
constructing a second hyperspectral image data cube from said second set of spectral planes; and
partitioning said second hyperspectral image data cube into a plurality of sub-data cubes each containing a 2D array of pixels.

### Claim 13 (depends on 12)

The system of claim 12, wherein reconstructing said full signal ƒ* of said hyperspectral image comprises:
for all sub-data cubes:
for all pixels in the current sub-data cube:
performing L1-norm minimization to obtain a vector x* using the representation matrix ψ corresponding to this sub-data cube; and
using said vector x* and said corresponding representation matrix ψ to reconstruct said full signal ƒ*, such that: ƒ*=ψx*.

### Claim 14 (depends on 13)

The system of claim 13, wherein L1-norm minimization comprises solving, for every pixel in each spectral plane, a constrained minimization given by:
x*=min∥x∥L1, such that Ax=y,
where matrix A is a product of φ and ψ, x is a column vector of size N×1, and y is equal to the intensity vector for a given pixel in a sub-data cube from said target hyperspectral image.

### Claim 15 (depends on 14)

The system of claim 14, wherein said constrained minimization is solved using any of: a LASSO (least absolute shrinkage and selection operator) algorithm, and a Basis Pursuit algorithm.

### Claim 16 (depends on 12)

The system of claim 12, further comprising, for each sub-data cube obtained from having partitioned said second hyperspectral image data cube, filtering said second set of spectral planes in a spectral direction of at least one filter of said target camera's hyperspectral image data cube.

### Claim 17 (depends on 12)

The system of claim 12, further comprising using a Gram-Schmidt Orthogonalization to restrict matrix ψ to said space of interest.

### Claim 18 (depends on 11)

The system of claim 11, wherein said target hyperspectral camera has a multi-filter grid for multi-band capture of different spectral planes at wavelengths of interest with each filter being tuned to a specified wavelength of interest in the electromagnetic spectrum.

### Claim 19 (depends on 12)

The system of claim 12, wherein the 2D spatial resolutions of said first hyperspectral image data cube is the same as the 2D spatial resolution of said second hyperspectral image data cube.

### Claim 20 (depends on 12)

The system of claim 12, wherein, in response to the 2D spatial resolutions of said first hyperspectral image data cube not being the same as the 2D spatial resolutions of said second hyperspectral image data cube, resizing said image to match said resolutions.

### Claim 21

A computer implemented method for image reconstruction using a compressed sensing framework for a hyperspectral video system, the method comprising:
constructing, for each of a plurality of sub-data cubes a N×N representation matrix ψ, each of said sub-data cubes containing a 2D array of pixels and collectively comprising a first hyperspectral image data cube constructed from a first set of different spectral planes of a hyperspectral image captured using a reference sensor, where N is a total number of bands in said reference sensor, wherein constructing each representation matrix comprises:
creating a matrix of random numbers;
deriving a set of natural basis vectors in a spectral direction;
selecting a first few of said natural basis vectors to restrict said representation matrix to a space of interest;
replacing a first few columns of said random number matrix with said selected first few natural basis vectors; and
performing Gram-Schmidt ortho-normalization to obtain said representation matrix;
receiving a M×N sampling matrix φ comprising a non-square matrix filled with 1's at locations corresponding to the peak wavelengths of filters in a multi-filter grid of a target hyperspectral camera and with remaining elements filled with 0's, where M is a number of bands in said target camera such that M<<N; and
using said constructed representation matrices and said sampling matrix to reconstruct a full signal ƒ* of a hyperspectral image captured by said target hyperspectral camera.

### Claim 22 (depends on 21)

The computer implemented method of claim 21, further comprising:
capturing a second set of different spectral planes using said target hyperspectral camera;
constructing a second hyperspectral image data cube from said second set of spectral planes; and
partitioning said second hyperspectral image data cube into a plurality of sub-data cubes each containing a 2D array of pixels.

### Claim 23 (depends on 21)

The computer implemented method of claim 21, wherein reconstructing said full signal ƒ* of said hyperspectral image comprises:
for all sub-data cubes:
for all pixels in the current sub-data cube:
performing L1-norm minimization to obtain a vector x using the representation matrix ψ corresponding to this sub-data cube, wherein L1-norm minimization comprises solving, for every pixel in each spectral plane, a constrained minimization given by: x*=min∥x∥L1, such that Ax=y, where A is a product of φ and ψ, x is a column vector of size N×1, and y is equal to the intensity vector for a given pixel in a sub-data cube from target hyperspectral image; and
using said vector x* and said corresponding representation matrix ψ to reconstruct said full signal ƒ*, such that: ƒ*=ψx*.

### Claim 24 (depends on 1)

The method of claim 1, wherein said reference sensor and said target hyperspectral camera have different 2D spatial resolutions.

### Claim 25 (depends on 11)

The system of claim 11, wherein said reference sensor and said target hyperspectral camera have different 2D spatial resolutions.