# US 9,521,320 B2 · Image processing apparatus, image capturing apparatus, image processing method, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,521,320 B2
- Document ID: US9521320B2
- Status: Granted
- Kind: B2
- Application number: 14/984,071
- Filed: 2015-12-30
- Published: 2016-05-12
- Granted: 2016-12-13
- Earliest claimed: 2013-03-05
- Assignees: Canon Kabushiki Kaisha
- Applicants: Canon Kabushiki Kaisha
- Inventors: Shohei Tsutsumi
- Web page: https://patentgrep.com/patent/US9521320B2

## CPC

- H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio
- H04N 5/142: {Edging; Contouring}
- H04N 5/2624: {for obtaining an image which is composed of whole input images, e.g. splitscreen}
- H04N 5/2628: {Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation}
- H04N 5/265: Mixing
- H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay
- H04N 23/62: Control of parameters via user interfaces
- H04N 23/632: {for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters}
- H04N 23/635: {Region indicators; Field of view indicators}
- H04N 23/959: by adjusting depth of field during image capture, e.g. maximising or setting range based on scene characteristics
- G01S 17/08: for measuring distance only (indirect measurement G01S17/46; active triangulation systems G01S17/48)
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES

## Abstract

Conventionally, on a subject in which range, refocus can be performed at the time of image capturing or at the time of editing an image is not specified clearly, and therefore, it is difficult for a user to capture an image or to edit an image in a manner the user intends. An image processing apparatus has an acquisition unit configured to acquire an image including a plurality of subject areas and distance information corresponding to the plurality of subject areas and a generation unit configured to generate a shifted image in which positions of the plurality of subject areas are shifted in the image based on the distance information.

## Claims

### Claim 1

An image processing apparatus comprising:
an acquisition unit configured to acquire an image including a plurality of subject areas and distance information corresponding to the plurality of subject areas;
a first generation unit configured to generate a rearranged image in which the plurality of subject areas is rearranged in the image based on the distance information; and
a second generation unit configured to generate an image comprising the rearranged image in which information on a focus state for image combination processing to change the focus state of the image including the plurality of subject areas is reflected in the rearranged image.

### Claim 2 (depends on 1)

The image processing apparatus according to claim 1, wherein
the information on the focus state includes at least one of a focus position, a depth of field, and a focus control range.

### Claim 3 (depends on 1)

The image processing apparatus according to claim 1, wherein
the first generation unit performs the rearrangement so that the plurality of subject areas is arranged in order in a depth direction in accordance with the distance information.

### Claim 4 (depends on 1)

The image processing apparatus according to claim 1, wherein
the distance information is information indicative of the respective distance to the plurality of subject areas from the image capturing apparatus having captured the image including the plurality of subject areas, and
the image processing apparatus further comprises a distance derivation unit configured to derive the distance information.

### Claim 5 (depends on 4)

The image processing apparatus according to claim 4, wherein
the acquisition unit acquires parallax image data, which is image data indicative of the image including the plurality of subject areas, which is obtained by capturing images from a plurality of viewpoints, and which is configured by a plurality of single-viewpoint images each corresponding to respective viewpoints, and
the distance derivation unit derives the distance information by performing stereo matching between the plurality of single-viewpoint images.

### Claim 6 (depends on 1)

The image processing apparatus according to claim 1, wherein
the acquisition unit acquires parallax image data, which is image data indicative of the image including the plurality of subject areas, which is obtained by capturing images from a plurality of viewpoints, and which is configured by a plurality of single-viewpoint images each corresponding to the respective viewpoints, and
the second generation unit generates an image comprising the rearranged image in which information on the focus state for the image combination processing is reflected in the rearranged image based on one single-viewpoint image of the plurality of single-viewpoint images.

### Claim 7 (depends on 1)

The image processing apparatus according to claim 1, wherein
the acquisition unit acquires parallax image data, which is image data indicative of the image including the plurality of subject areas, which is obtained by capturing images from a plurality of viewpoints, and which is configured by a plurality of single-viewpoint images each corresponding to the respective viewpoints, and
the second generation unit generates an image comprising the rearranged image in which information on the focus state for the image combination processing is reflected in the rearranged image based on the image generated by the image combination processing.

### Claim 8 (depends on 1)

The image processing apparatus according to claim 1, wherein the image combination processing is processing to combine a plurality of parallax images obtained by capturing the plurality of subject areas from a plurality of different viewpoints.

### Claim 9 (depends on 1)

The image processing apparatus according to claim 1, wherein the image combination processing is filtering processing on the image including the plurality of subject areas.

### Claim 10 (depends on 1)

The image processing apparatus according to claim 1, further comprising an extraction unit configured to extract subject areas from the image including the plurality of subject areas based on the distance information, wherein the first generation unit generates the rearranged image by rearranging subject areas extracted by the extraction unit.

### Claim 11 (depends on 10)

The image processing apparatus according to claim 10, wherein the extraction unit extracts an area along a contour of a subject or an area in an arbitrary shape including a subject as the subject area.

### Claim 12 (depends on 1)

An image capturing apparatus having the image processing apparatus according to claim 1, the image capturing apparatus comprising:
a reception unit configured to receive a user's instructions;
a display unit configured to display an image in which information on the focus state is reflected; and
an adjustment unit configured to adjust the focus state of the image displayed in the display unit based on the user's instructions.

### Claim 13 (depends on 12)

The image capturing apparatus having the image processing apparatus according to claim 12, wherein the display unit has a live view function and displays an image in which information on the focus state is reflected in a live view.

### Claim 14

An image processing method comprising:
acquiring an image including a plurality of subject areas and distance information corresponding to the plurality of subject areas;
generating a rearranged image in which the plurality of subject areas is rearranged in the image based on the distance information; and
generating an image comprising the rearranged image in which information on a focus state for image combination processing to change the focus state of the image including the plurality of subject areas is reflected in the rearranged image.

### Claim 15 (depends on 14)

A non-transitory computer readable storage medium storing a program for causing a computer to perform the image processing method according to claim 14.