# US 2015/0145988 A1 · Image processing apparatus, imaging apparatus, and image processing method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0145988 A1
- Document ID: US20150145988A1
- Status: Published application
- Kind: A1
- Application number: 14/538,156
- Filed: 2014-11-11
- Published: 2015-05-28
- Earliest claimed: 2013-11-28
- Assignees: Canon Kabushiki Kaisha
- Applicants: Canon Kabushiki Kaisha
- Inventors: Aihiko Numata
- Web page: https://patentgrep.com/patent/US20150145988A1

## CPC

- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G02B 27/0075: {with means for altering, e.g. increasing, the depth of field or depth of focus}
- G03B 13/36: Autofocus systems
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/70: Denoising; Smoothing
- H04N 23/959: by adjusting depth of field during image capture, e.g. maximising or setting range based on scene characteristics
- G02B 7/34: using different areas in a pupil plane
- G06T 2200/21: involving computational photography
- H04N 23/67: Focus control based on electronic image sensor signals
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G03B: APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR (optical parts of such apparatus G02B; photosensitive materials or processes for photographic purposes G03C; apparatus for processing exposed photographic materials G03D)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An image processing apparatus includes: a distance calculation unit configured to calculate distance information on the basis of a first image and a second image; and a blur addition unit configured to add a blur to original images based on the first image and the second image, using the distance information calculated by the distance calculation unit and to move a focus plane by a predetermined refocus distance. The blur addition unit adds the blur to a first original image when the refocus distance is equal to or less than a threshold and adds the blur to a second original image, which is an image having an effective F-number greater than an effective F-number of the first original image, when the refocus distance is greater than the threshold.

## Claims

### Claim 1

An image processing apparatus comprising:
a distance calculation unit configured to calculate distance information on the basis of a first image based on a light beam from a first pupil region of an imaging optical system and a second image based on a light beam from a second pupil region of the imaging optical system; and
a blur addition unit configured to add a blur to original images based on the first image and the second image, using the distance information calculated by the distance calculation unit and to move a focus plane by a predetermined refocus distance, wherein
the blur addition unit adds the blur to a first original image when the refocus distance is equal to or less than a threshold and adds the blur to a second original image, which is an image having an effective F-number greater than an effective F-number of the first original image, when the refocus distance is greater than the threshold.

### Claim 2 (depends on 1)

The image processing apparatus according to claim 1, wherein
the first image and the second image are images having a same effective F-number,
the first original image is an image obtained by adding up the first image and the second image at a first ratio, and
the second original image is any one of an image obtained by adding up the first image and the second image at a second ratio, an absolute value of a logarithm of which is greater than an absolute value of a logarithm of the first ratio, the first image, and the second image.

### Claim 3 (depends on 2)

The image processing apparatus according to claim 2, wherein the first original image is an image obtained by adding up the first image and the second image with a same ratio.

### Claim 4 (depends on 3)

The image processing apparatus according to claim 3, wherein the second original image is the first image or the second image.

### Claim 5 (depends on 3)

The image processing apparatus according to claim 3, wherein the threshold is a depth of field of the first original image.

### Claim 6 (depends on 3)

The image processing apparatus according to claim 3, wherein the threshold is a refocus distance at which resolution of the first original image and resolution of the second original image are equal.

### Claim 7 (depends on 3)

The image processing apparatus according to claim 3, wherein the blur addition unit shifts center-of-gravity positions of the first image and the second image according to the distance calculated by the distance calculation unit and generates the second original image on the basis of the first image and the second image after the shift.

### Claim 8 (depends on 3)

The image processing apparatus according to claim 3, wherein the blur addition unit implements image correction processing on the first image, using a transmittance distribution of the first pupil region, implements the image correction processing on the second image, using a transmittance distribution of the second pupil region, and generates the second original image on the basis of the first image and the second image after the correction.

### Claim 9 (depends on 1)

The image processing apparatus according to claim 1, wherein
the first image and the second image are images having different effective F-numbers,
the first original image is an image having a smaller effective F-number of the first image and the second image, and
the second original image is an image having a greater effective F-number of the first image and the second image.

### Claim 10 (depends on 9)

The image processing apparatus according to claim 9, wherein the threshold is a refocus distance at which resolution of the first image and resolution of the second image are equal.

### Claim 11 (depends on 9)

The image processing apparatus according to claim 9, further comprising a control unit configured to control, according to the refocus distance, at least one of the effective F-number of the first image and the effective F-number of the second image.

### Claim 12 (depends on 1)

The image processing apparatus according to claim 1, wherein
both of the first image and the second image are images having different effective F-numbers according to pixel positions,
the first original image is an image formed of, in each pixel position, a pixel value having a smaller effective F-number of the first image and the second image, and
the second original image is an image formed of, in each pixel position, a pixel value having a greater effective F-number of the first image and the second image.

### Claim 13 (depends on 1)

An imaging apparatus including:
an imaging optical system;
an imaging unit configured to acquire a signal based on a light beam transmitted through an exit pupil of the imaging optical system; and
the image processing apparatus according to claim 1.

### Claim 14 (depends on 13)

The imaging apparatus according to claim 13, wherein
the imaging unit includes a solid-state imaging sensor including a plurality of ranging pixels, and
the ranging pixel includes a first photoelectric conversion unit configured to receive a light beam from a first pupil region of the imaging optical system and acquire a first image and a second photoelectric conversion unit configured to receive a light beam from a second pupil region of the imaging optical system and acquire a second image.

### Claim 15 (depends on 14)

The imaging apparatus according to claim 14, wherein the ranging pixel includes a light beam separation unit configured to separate incident light according to an incident direction of light incident on the ranging pixel.

### Claim 16 (depends on 15)

The imaging apparatus according to claim 15, wherein the light beam separation unit is a microlens.

### Claim 17 (depends on 15)

The imaging apparatus according to claim 15, wherein the light beam separation unit is a waveguide.

### Claim 18 (depends on 14)

The imaging apparatus according to claim 14, wherein, in the ranging pixel, an aperture ratio of the first photoelectric conversion unit and an aperture ratio of the second photoelectric conversion unit are different.

### Claim 19 (depends on 18)

The imaging apparatus according to claim 18, wherein the ranging pixel further includes an aperture ratio control unit configured to control at least one of the aperture ratio of the first photoelectric conversion unit and the aperture ratio of the second photoelectric conversion unit.

### Claim 20

An image processing method in an image processing apparatus,
the method comprising:
calculating distance information on the basis of a first image based on a light beam from a first pupil region of an imaging optical system and a second image based on a light beam from a second pupil region of the imaging optical system; and
adding a blur to original images based on the first image and the second image, using the calculated distance information and moving a focus plane by a predetermined refocus distance, wherein
in adding the blur, the blur is added to a first original image when the refocus distance is equal to or less than a threshold and the blur is added to a second original image, which is an image having an effective F-number greater than an effective F-number of the first original image, when the refocus distance is greater than the threshold.
