# US 2018/0324359 A1 · Image processing for turbulence compensation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0324359 A1
- Document ID: US20180324359A1
- Status: Published application
- Kind: A1
- Application number: 15/956,109
- Filed: 2018-04-18
- Published: 2018-11-08
- Earliest claimed: 2017-05-02
- Assignees: Canon Kabushiki Kaisha
- Applicants: Canon Kabushiki Kaisha
- Inventors: Ruimin Pan; Matthew Raphael Arnison; David Robert James Monaghan
- Web page: https://patentgrep.com/patent/US20180324359A1

## CPC

- H04N 23/683: {performed by a processor, e.g. controlling the readout of an image memory}
- G06T 7/571: from focus
- G06T 7/593: from stereo images
- H04N 13/207: using a single two-dimensional [2D] image sensor
- H04N 13/246: Calibration of cameras
- H04N 23/672: {based on the phase difference signals}
- H04N 23/80: Camera processing pipelines; Components thereof
- H04N 23/95: Computational photography systems, e.g. light-field imaging systems
- H04N 25/704: Pixels specially adapted for focusing, e.g. phase difference pixel sets
- G06T 2207/10012: Stereo images
- G06T 2207/20201: Motion blur correction
- G06T 2207/20228: Disparity calculation for image-based rendering
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

One or more embodiments of an apparatus, system and method of compensating image data for phase fluctuations caused by a wave deforming medium, and storage or recording mediums for use therewith, are provided herein. At least one embodiment of the method comprises capturing, by a sensor of an imaging system, first image data and second image data for each of a plurality of pixel positions of the sensor, the sensor capturing an object through a wave deforming medium causing a defocus disparity between the first image data and second image data; and determining the defocus disparity between the first image data and the second image data, the defocus disparity corresponding to a defocus wavefront deviation of the wave deforming medium. The method may further comprise compensating the image data captured by the sensor for phase fluctuations caused by the wave deforming medium using the determined defocus disparity.

## Claims

### Claim 1

A method of compensating image data for phase fluctuations caused by a wave deforming medium, the method comprising:
capturing, by a sensor of an imaging system, first image data and second image data for each of a plurality of pixel positions of the sensor, the sensor capturing an object through a wave deforming medium causing a defocus disparity between the first image data and second image data;
determining the defocus disparity between the first image data and the second image data, the defocus disparity corresponding to a defocus wavefront deviation of the wave deforming medium; and
compensating the image data captured by the sensor for phase fluctuations caused by the wave deforming medium using the determined defocus disparity.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the first image data and the second image data is captured using a dual-pixel autofocus sensor.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the defocus disparity between the first image data and the second image data relates to displacement between left pixel data and right pixel data of the dual-pixel autofocus sensor.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the first image data and the second image data is a captured by a stereo camera, and the defocus disparity between the first image data and the second image data relates to displacement between left and right image data captured by the stereo camera.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the first image data and the second image data comprises a first image and a second image captured at different times, and the defocus disparity between the first image data and the second image data relates to a relative blur between the first image and the second image.

### Claim 6 (depends on 5)

The method according to claim 5, wherein the defocus wavefront deviation is determined based on a defocus distance, the defocus distance being determined using the relative blur between the first image and the second image.

### Claim 7 (depends on 1)

The method according to claim 1, wherein processing at least one of the first image data and second image data comprises:
determining a look-up table comprising a mapping between a strength of phase fluctuations caused by the wave deforming medium and a tile size for a tile-based turbulence compensation method;
selecting a tile size from the look-up table based on the determined strength of phase fluctuations; and
applying the tile-based compensation method using the selected tile size to correct for the phase fluctuations.

### Claim 8 (depends on 1)

The method according to claim 1, wherein the received first image data and second image data comprises a plurality of frames.

### Claim 9 (depends on 8)

The method according to claim 8, wherein strength of phase fluctuations in a region of a frame is determined based on a plurality of samples of the strength of phase fluctuations determined within the region of the frame.

### Claim 10 (depends on 8)

The method according to claim 8, further comprising:
compensating for phase fluctuations caused by the wave deforming medium based on comparison of the strength of phase fluctuations associated with the plurality of frames.

### Claim 11 (depends on 8)

The method according to claim 8, further comprising:
compensating for phase fluctuations caused by the wave deforming medium by fusing the plurality of frames based on values of the strength of phase fluctuations determined within each one of the plurality of frames.

### Claim 12 (depends on 11)

The method according to claim 11, wherein the fusing comprises:
for each region in a fused image, determining a plurality of corresponding regions from the plurality of frames, each corresponding region being associated with a strength of phase fluctuations;
comparing the strength of phase fluctuations for the determined corresponding regions; and
forming a region of the fused image based on the comparison.

### Claim 13 (depends on 12)

The method according to claim 12, wherein forming a region of the fused image comprises selecting a region from the determined plurality of corresponding regions having a strength of phase fluctuation below a predetermined threshold.

### Claim 14 (depends on 12)

The method according to claim 12, wherein forming a region of the fused image comprises selecting a region from the determined plurality of corresponding regions having a lowest strength of phase fluctuation.

### Claim 15 (depends on 1)

The method according to claim 1, wherein processing at least one of the first image data and second image data comprises deconvolving the image data using a point spread function, the size of the point spread function at a particular position being determined using the strength of phase fluctuation determined at the particular pixel position.

### Claim 16 (depends on 1)

The method according to claim 1, wherein the defocus wavefront deviation is determined based on disparity in the first image data and the second image data with respect to reference image data at a predetermined pixel position, and the reference image data is determined by convolving the first image data with a kernel having a predetermined width.

### Claim 17 (depends on 16)

The method according to claim 16, wherein the disparity is a ratio of a gradient magnitude of the second image data over a gradient magnitude of the reference image data, the ratio determined at a pixel with maximum gradient across an edge.

### Claim 18 (depends on 1)

The method according to claim 1, wherein determining the defocus wavefront deviation comprises estimating the wavefront deviation using one dimensional signals captured using an autofocus sensor of a device capturing the image.

### Claim 19 (depends on 1)

The method according to claim 1, wherein processing at least one of the first image data and second image data comprises:
determining a region associated with the object in a plurality of frames of the received image;
determining an average strength of phase fluctuations for the region in each of the plurality of frames, and
selecting a frame based on the average strength of phase fluctuations for the region.

### Claim 20 (depends on 19)

The method according to claim 19, further comprising generating a fused high-resolution image from the regions based on the average strength of phase fluctuations.

### Claim 21 (depends on 1)

The method according to claim 1, further comprising determining a strength of phase fluctuations caused by the wave deforming medium using the defocus wavefront deviation and lens intrinsic characteristics, the strength of phase fluctuations being determined with reference to a defocus Zemike coefficient.

### Claim 22

A computer readable medium having at least one computer program stored thereon for causing at least one processor to perform a method for determining a turbulence strength for processing image data, the method comprising:
receiving image data for a portion of an image captured by a dual-pixel sensor through a wave deforming medium;
determining a defocus disparity between left pixel data and right pixel data of the dual-pixel sensor, the defocus disparity corresponding to a defocus wavefront deviation of the wave deforming medium; and
determining the turbulence strength caused by the wave deforming medium using the determined defocus disparity between left pixel data and right pixel data to process the image data.

### Claim 23

An image capture apparatus configured to determine phase fluctuations of a wave deforming medium, the image capturing apparatus comprising:
a memory;
a lens system focusing light travelling from an imaging scene through the wave deforming medium on an image sensor;
the image sensor configured to capture image data from the lens system as first pixel data and second pixel data for each of a plurality of pixel positions, the image sensor being coupled to the memory, the memory storing the captured first pixel data and second pixel data; and
a processor coupled to the memory and configured to determine phase fluctuations caused by the wave deforming medium using a defocus disparity between the first pixel data and the second pixel data captured by the image sensor.

### Claim 24

A system, comprising:
an image capture sensor,
a memory, and
a processor, wherein the processor executes code stored on the memory to:
receive, from the image capture sensor, first image data and second image data for each of a plurality of pixel positions of the image capture sensor, the image data capturing an object through a wave deforming medium causing a defocus disparity between the first image data and second image data;
determine the defocus disparity between the first image data and the second image data, the defocus disparity corresponding to a defocus wavefront deviation of the wave deforming medium; and
compensate the image data captured by the image capture sensor for phase fluctuations caused by the wave deforming medium using the determined defocus disparity.
