# US 2016/0261852 A1 · Image processing device, image processing method, and imaging device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0261852 A1
- Document ID: US20160261852A1
- Status: Published application
- Kind: A1
- Application number: 15/032,549
- Filed: 2014-11-17
- Published: 2016-09-08
- Earliest claimed: 2013-12-16
- Assignees: Sony Corporation
- Applicants: Sony Corporation
- Inventors: Yasutaka HIRASAWA
- Web page: https://patentgrep.com/patent/US20160261852A1

## CPC

- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- G01B 11/24: for measuring contours or curvatures
- G01C 3/08: Use of electric radiation detectors
- G01C 11/30: by triangulation
- G06T 7/55: from multiple images
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 13/25: using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
- H04N 13/128: Adjusting depth or disparity
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A depth map generation unit (15) generates a depth map through a matching process using a first image generated by a first imaging unit which has a pixel configuration including pixels having different polarization directions and a second image generated by a second imaging unit which has a different pixel configuration from the pixel configuration of the first imaging unit. A normal-line map generation unit (17) generates a normal-line map based on a polarization state of a polarized image of at least one of the first and second images. A map unifying unit (19) performs a process of unifying the generated depth map and the generated normal-line map and acquires an image in which the number of pixels is not reduced while generating the depth map with precision equal to or greater than the generated depth map. The image in which the number of pixels is not reduced can be acquired while generating the highly precise depth map.

## Claims

### Claim 1

An image processing device comprising:
a depth map generation unit configured to generate a depth map through a matching process using a first image generated by a first imaging unit which has a pixel configuration including pixels having different polarization directions and a second image generated by a second imaging unit which has a different pixel configuration from the pixel configuration of the first imaging unit;
a normal-line map generation unit configured to generate a normal-line map based on a polarization state of a polarized image of at least one of the first and second images generated by the depth map generation unit; and
a map unifying unit configured to perform a process of unifying the depth map generated by the depth map generation unit and the normal-line map generated by the normal-line map generation unit.

### Claim 2 (depends on 1)

The image processing device according to claim 1,
wherein the normal-line map generation unit generates the normal-line map based on luminance of the polarized image having three or more polarization directions.

### Claim 3 (depends on 2)

The image processing device according to claim 2,
wherein the map unifying unit estimates a depth value not indicated in the depth map from a shape determined based on a depth value indicated in the depth map and the normal-line map.

### Claim 4 (depends on 3)

The image processing device according to claim 3,
wherein the first image is an image generated by the first imaging unit including the pixels having three or more polarization directions,
wherein the second image is an image generated by the second imaging unit including pixels with no polarization characteristics, and
wherein the normal-line map generation unit generates the normal-line map based on the first image.

### Claim 5 (depends on 4)

The image processing device according to claim 4,
wherein the depth map generation unit generates an unpolarized image from the first image and performs the matching process using the unpolarized image and the second image.

### Claim 6 (depends on 5)

The image processing device according to claim 5,
wherein the depth map generation unit performs edge extraction on each of the unpolarized image and the second image and performs the matching process using an edge-extracted image of the unpolarized image and an edge-extracted image of the second image.

### Claim 7 (depends on 4)

The image processing device according to claim 4,
wherein the second image is an image generated by the second imaging unit in which color filters of the same color are provided in all of the pixels or the second imaging unit in which the color filters are not provided, and
wherein the depth map generation unit performs the matching process using an unpolarized image generated from the second image.

### Claim 8 (depends on 3)

The image processing device according to claim 3,
wherein the first in is an image generated by the first imaging unit which has a pixel configuration including a first pixel group formed by pixels with polarization characteristics and a second pixel group formed by pixels having a polarization direction different from a polarization direction of the first pixel group or pixels with no polarization characteristics, and
wherein the second image is an image generated by the second imaging unit which has a pixel configuration including a third pixel group formed by pixels having a different polarization direction from the polarization direction of the first image at a position corresponding to the first pixel group and a fourth pixel group formed by pixels with the same configuration as a configuration of the second pixel group at a position corresponding to the second pixel group.

### Claim 9 (depends on 8)

The image processing device according to claim 8,
wherein the depth map generation unit performs the matching process between images having the same polarization direction or images with no polarization characteristics by using an image of the second pixel group in the first image and an image of the fourth pixel group in the second image.

### Claim 10 (depends on 8)

The image processing device according to claim 8, further comprising:
an image phase adjustment unit configured to generate a polarized image having a plurality of polarization directions by matching phases of an image of the first pixel group in the first image and an image of the third pixel group in the second image based on a parallax amount of the first image and the second image,
wherein the normal-line map generation unit generates the normal-line map based on a polarization state of the polarized image generated by the image phase adjustment unit.

### Claim 11 (depends on 8)

The image processing device according to claim 8,
wherein, in a case of an image in which the second and fourth pixel groups have no polarization characteristics, an image having three or more polarization directions of the first and third pixel groups in total is set.

### Claim 12 (depends on 8)

The image processing device according to claim 8,
wherein, in a case of an image in which the second and fourth pixel groups have no polarization characteristics, the image phase adjustment unit generates an image of the second pixel group through an interpolation process using an image of the first pixel group, generates an image of the fourth pixel group through an interpolation process using an image of the third pixel group, and generates the polarized image using the interpolated images.

### Claim 13 (depends on 8)

The image processing device according to claim 8,
wherein the first and third pixel groups are pixels of a predetermined color and the second and fourth pixel groups are pixels of different colors.

### Claim 14 (depends on 3)

The image processing device according to claim 3,
wherein the normal-line map generation unit generates the normal-line map using the depth map generated by the depth map generation unit.

### Claim 15 (depends on 1)

The image processing device according to claim 1, further comprising:
the first imaging unit configured to generate the first image; and
the second imaging unit configured to generate the second image.

### Claim 16 (depends on 1)

The image processing device according to claim 1, further comprising:
a communication unit configured to perform communication with an external device in which one of the first and second imaging units is provided and acquire an image generated by the imaging unit provided in the external device; and
another imaging unit different from the imaging unit provided in the external device.

### Claim 17

An image processing method comprising:
a step of generating, by a depth map generation unit, a depth map by performing a matching process using a first image generated by a first imaging unit which has a pixel configuration including pixels having different polarization directions and a second image generated by a second imaging unit which has a different pixel configuration from the pixel configuration of the first imaging unit;
a step of generating, by a normal-line map generation unit, a normal-line map based on a polarization state of a polarized image of at least one of the generated first and second images; and
a step of performing, by a map unifying unit, a process of unifying the depth map and the normal-line map.

### Claim 18

An imaging device comprising:
a first imaging unit configured to have a pixel configuration including pixels having different polarization directions;
a second imaging unit configured to have a different pixel configuration from the pixel configuration of the first imaging unit; and
an image processing unit configured to perform image processing using a first image generated by the first imaging unit and a second image generated by the second imaging unit.

### Claim 19 (depends on 18)

The imaging device according to claim 18,
wherein the first imaging unit has a pixel configuration including a first pixel group formed by pixels with polarization characteristics and a second pixel group formed by pixels having a different polarization direction from a polarization direction of the first pixel group or pixels with no polarization characteristics, and
wherein the second imaging unit has a pixel configuration including a third pixel group formed by pixels having a different polarization direction from a polarization direction of the first image at a position corresponding to the first pixel group and a fourth pixel group formed by pixels with the same configuration as a configuration of the second pixel group at a position corresponding to the second pixel group.

### Claim 20

An image processing device comprising:
a reception unit configured to receive a first image from an image processing device including a first imaging unit that has a pixel configuration including pixels having different polarization directions and a transmission unit that transmits the first image generated by the first imaging unit;
a second imaging unit configured to have a different pixel configuration from the pixel configuration of the first imaging unit; and
an image processing unit configured to perform image processing using the first image received by the reception unit and a second image generated by the second imaging unit.
