# US 9,002,128 B2 · Image processing apparatus, integrated circuit, program, imaging apparatus, and display apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,002,128 B2
- Document ID: US9002128B2
- Status: Granted
- Kind: B2
- Application number: 13/981,734
- Filed: 2012-12-26
- Published: 2014-03-06
- Granted: 2015-04-07
- Earliest claimed: 2012-03-15
- Assignees: Panasonic Intellectual Property Corporation of America
- Applicants: Panasonic Intellectual Property Corporation of America
- Inventors: Yoshiteru Tanaka; Ikuo Fuchigami; Tsuyoshi Nakamura
- Web page: https://patentgrep.com/patent/US9002128B2

## CPC

- H04N 5/2621: {Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability}
- G03B 15/00: Special procedures for taking photographs; Apparatus therefor
- G06T 5/70: Denoising; Smoothing
- H04N 23/60: Control of cameras or camera modules
- H04N 23/80: Camera processing pipelines; Components thereof
- H04N 23/958: for extended depth of field imaging
- G03B 35/08: by simultaneous recording
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20012: Locally adaptive
- H04N 5/2226: {Determination of depth image, e.g. for foreground/background separation (determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10)}
- H04N 25/61: the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G03B: APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR (optical parts of such apparatus G02B; photosensitive materials or processes for photographic purposes G03C; apparatus for processing exposed photographic materials G03D)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An image processing apparatus separates a captured image into a region of object (B) 1001B and a region of object (A) 1001A shallower than the region of object (B) 1001B in depth indicated by depth information, duplicates pixels constituting the region of object (B) 1001B and positioned in neighborhood of boundary between the region of object (B) 1001B and the region of object (A) 1001A, onto the neighborhood of the boundary outside the region of object (B) 1001B, thereby generating an extended region (B) 5001B, performs blur processing on the extended region (B) 5001B and the region of object (A) 1001A based on the depth of the region of object (A) 1001A indicated by the depth information, and after the processing, composites a value of each pixel constituting the extended region (B) 5001B and a value of one of pixels constituting the region of object (A) 1001A corresponding in position.

## Claims

### Claim 1

An image processing apparatus comprising:
a separation unit configured to separate a captured image into a plurality of regions including a first region and a second region that is shallower than the first region in terms of depth indicated by depth information of the captured image;
an extension unit configured to duplicate pixels that constitute the first region and are positioned in a neighborhood of a boundary between the first region and the second region, onto the neighborhood of the boundary outside the first region, thereby generating an extended region;
a blur processing unit configured to perform blur processing on the extended region and the second region based on the depth of the second region indicated by the depth information; and
a composition unit configured to, after the blur processing, composite a value of each of pixels constituting the extended region and a value of one of pixels constituting the second region that correspond in position with each other.

### Claim 2 (depends on 1)

The image processing apparatus of claim 1, wherein
the composition unit sets a composition ratio of a value of each of the pixels constituting the extended region to a value of a corresponding one of the pixels constituting the second region, such that the composition ratio is increasingly lower as the pixel constituting the extended region is more distant from the first region.

### Claim 3 (depends on 1)

The image processing apparatus of claim 1, wherein
the composition unit performs the composition at a composition ratio that is calculated based on an area ratio, in a range where a blur filter has been applied to each target pixel in the blur processing, of the first region to the second region in the captured image.

### Claim 4 (depends on 1)

The image processing apparatus of claim 1, wherein
the separation unit, the extension unit, the blur processing unit, and the composition unit repeatedly perform the separation processing, the extension processing, the blur processing, and the composition processing, respectively, thereby adjusting a blur effect.

### Claim 5 (depends on 1)

The image processing apparatus of claim 1, wherein
the blur processing unit repeatedly performs the blur processing, thereby adjusting a blur effect.

### Claim 6 (depends on 1)

The image processing apparatus of claim 1, wherein
with respect to one target region among the regions,
the extension unit judges whether the target region is adjacent to other of the regions that is shallower than the target region in terms of depth indicated by the depth information, and
when judging that the target region is adjacent to the other region,
the extension unit duplicates pixels that constitute the target region and are positioned in a neighborhood of a boundary between the target region and the adjacent region, onto the neighborhood of the boundary outside the target region, thereby generating an extended region.

### Claim 7

An integrated circuit comprising:
a separation unit configured to separate a captured image into a plurality of regions including a first region and a second region that is shallower than the first region in terms of depth indicated by depth information of the captured image;
an extension unit configured to duplicate pixels that constitute the first region and are positioned in a neighborhood of a boundary between the first region and the second region, onto the neighborhood of the boundary outside the first region, thereby generating an extended region;
a blur processing unit configured to perform blur processing on the extended region and the second region based on the depth of the second region indicated by the depth information; and
a composition unit configured to, after the blur processing, composite a value of each of pixels constituting the extended region and a value of one of pixels constituting the second region that correspond in position with each other.

### Claim 8

A non-transitory computer readable recording medium having recorded therein a program that causes a computer to perform blur effect processing, the blur effect processing comprising:
a separation step of separating a captured image into a plurality of regions including a first region and a second region that is shallower than the first region in terms of depth indicated by depth information of the captured image;
an extension step of duplicating pixels that constitute the first region and are positioned in a neighborhood of a boundary between the first region and the second region, onto the neighborhood of the boundary outside the first region, thereby generating an extended region;
a blur processing step of performing blur processing on the extended region and the second region based on the depth of the second region indicated by the depth information; and
a composition step of, after the blur processing, compositing a value of each of pixels constituting the extended region and a value of one of pixels constituting the second region that correspond in position with each other.

### Claim 9

An imaging apparatus comprising:
an imaging unit having a stereo camera;
a display unit;
a memory configured to store therein a program for blur effect processing and a left image and a right image captured by the stereo camera;
a DSP configured to perform generation processing of generating a depth image indicating depth information for each pixel based on the captured left and right images stored in the memory, and perform the blur effect processing on at least one of the captured left and right images; and
a CPU configured to control processing of storing the left image and the right image captured by the stereo camera in the memory, control processing of causing the DSP to perform the generation processing and the blur effect processing, and control processing of causing the display unit to display a result of the blur effect processing, wherein
the blur effect processing comprises:
a separation step of separating a target image into a plurality of regions including a first region and a second region that is shallower than the first region in terms of depth indicated by depth information of the target image;
an extension step of duplicating pixels that constitute the first region and are positioned in a neighborhood of a boundary between the first region and the second region, onto the neighborhood of the boundary outside the first region, thereby generating an extended region;
a blur processing step of performing blur processing on the extended region and the second region based on the depth of the second region indicated by the depth information; and
a composition step of, after the blur processing, compositing a value of each of pixels constituting the extended region and a value of one of pixels constituting the second region that correspond in position with each other.

### Claim 10

An imaging apparatus comprising:
an imaging unit having a monocular camera;
a display unit;
a distance sensor unit configured to measure, for each pixel, a distance to a subject captured by the imaging unit;
a memory configured to store therein a program for blur effect processing, an image captured by the monocular camera, and the distance measured for each pixel by the distance sensor unit;
a DSP configured to perform generation processing of generating a depth image indicating depth information for each pixel based on the captured image stored in the memory and the measured distance for each pixel, and perform the blur effect processing on the captured image; and
a CPU configured to control processing of storing the left image and the right image captured by the monocular camera in the memory, control processing of causing the DSP to perform the generation processing and the blur effect processing, and control processing of causing the display unit to display a result of the blur effect processing, wherein
the blur effect processing comprises:
a separation step of separating a target image into a plurality of regions including a first region and a second region that is shallower than the first region in terms of depth indicated by depth information of the target image;
an extension step of duplicating pixels that constitute the first region and are positioned in a neighborhood of a boundary between the first region and the second region, onto the neighborhood of the boundary outside the first region, thereby generating an extended region;
a blur processing step of performing blur processing on the extended region and the second region based on the depth of the second region indicated by the depth information; and
a composition step of, after the blur processing, compositing a value of each of pixels constituting the extended region and a value of one of pixels constituting the second region that correspond in position with each other.

### Claim 11

A display apparatus comprising:
a display unit;
a memory configured to store therein a program for blur effect processing, a captured image, and a depth image indicating depth information for each pixel of the captured image;
a DSP configured to perform the blur effect processing on the captured image; and
a CPU configured to control processing of causing the DSP to perform the blur effect processing, and control processing of causing the display unit to display a result of the blur effect processing, wherein
the blur effect processing comprises:
a separation step of separating a target image into a plurality of regions including a first region and a second region that is shallower than the first region in terms of depth indicated by depth information of the target image;
an extension step of duplicating pixels that constitute the first region and are positioned in a neighborhood of a boundary between the first region and the second region, onto the neighborhood of the boundary outside the first region, thereby generating an extended region;
a blur processing step of performing blur processing on the extended region and the second region based on the depth of the second region indicated by the depth information; and
a composition step of, after the blur processing, compositing a value of each of pixels constituting the extended region and a value of one of pixels constituting the second region that correspond in position with each other.

### Claim 12

An image processing apparatus comprising:
a blur processing unit configured to, based on a captured image and a depth image indicating depth information of the captured image, perform blur processing on the captured image, thereby generating a blurred image; and
a repetition unit configured to, based on the generated blurred image and the depth information, repeatedly perform the blur effect processing on the generated blurred image to generate a new blurred image until a number of repetitions of the blur effect processing reaches a predetermined number, wherein
the blur effect processing unit includes:
a separation unit configured to separate a target image into a plurality of regions including a first region and a second region that is shallower than the first region in terms of depth indicated by depth information of the target image;
an extension unit configured to duplicate pixels that constitute the first region and are positioned in a neighborhood of a boundary between the first region and the second region, onto the neighborhood of the boundary outside the first region, thereby generating an extended region;
a blur processing unit configured to perform blur processing on the extended region and the second region based on the depth of the second region indicated by the depth information; and
a composition unit configured to, after the blur processing, composite a value of each of pixels constituting the extended region and a value of one of pixels constituting the second region that correspond in position with each other.

### Claim 13

A non-transitory computer readable recording medium having recorded therein a program that causes a computer to perform predetermined image processing, the predetermined image processing comprising:
a blur effect processing step of, based on a captured image and a depth image indicating depth information of the captured image, performing blur effect processing on the captured image, thereby generating a blurred image; and
a repetition step of, based on the generated blurred image and the depth information, repeatedly performing the blur effect processing on the generated blurred image to generate a new blurred image until a number of repetitions of the blur effect processing reaches a predetermined number, wherein
the blur effect processing comprises:
a separation step of separating a target image into a plurality of regions including a first region and a second region that is shallower than the first region in terms of depth indicated by depth information of the target image;
an extension step of duplicating pixels that constitute the first region and are positioned in a neighborhood of a boundary between the first region and the second region, onto the neighborhood of the boundary outside the first region, thereby generating an extended region;
a blur processing step of performing blur processing on the extended region and the second region based on the depth of the second region indicated by the depth information; and
a composition step of, after the blur processing, compositing a value of each of pixels constituting the extended region and a value of one of pixels constituting the second region that correspond in position with each other.