# US 2021/0243437 A1 · Image processing apparatus, image processing method, and program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0243437 A1
- Document ID: US20210243437A1
- Status: Published application
- Kind: A1
- Application number: 17/273,891
- Filed: 2019-06-06
- Published: 2021-08-05
- Earliest claimed: 2018-09-12
- Assignees: Sony Corporation
- Applicants: Sony Corporation
- Inventors: Yuhi KONDO; Yasutaka HIRASAWA; Takefumi NAGUMO; Toshinori IHARA; Teppei KURITA; Legong SUN
- Web page: https://patentgrep.com/patent/US20210243437A1

## CPC

- H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
- H04N 19/50: using predictive coding (H04N19/61 takes precedence)
- H04N 19/136: Incoming video signal characteristics or properties
- H04N 19/182: the unit being a pixel
- H04N 19/186: the unit being a colour or a chrominance component
- H04N 19/423: characterised by memory arrangements (H04N19/433 takes precedence)
- H04N 19/90: using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
- H04N 19/597: specially adapted for multi-view video sequence encoding
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A polarization imaging unit 20 acquires a polarized image including polarization pixels with a plurality of polarization directions. An information compression unit 40 sets reference image information based on polarized image information of reference polarization pixels with at least a plurality of polarization directions in the polarized image and generates difference information between the polarized image information of each of polarization pixels different from the reference polarization pixels in the polarized image and the reference image information. In addition, the information compression unit 40 reduces the amount of information of the difference information generated for each of the polarization pixels with the plurality of polarization directions to generate compressed image information including the reference image information and the difference information with the reduced amount of information. An information decoding unit 70 applies a decoding process corresponding to the compression process of the information compression unit 40 to the compressed image information acquired through a recording medium 50 or a transmission path 60 to generate the polarized image and outputs the polarized image to a polarized image using unit 80.

## Claims

### Claim 1

An image processing apparatus comprising:
an information compression unit that generates compressed image information by reducing an amount of information of polarized image information of each of polarization pixels with at least a plurality of polarization directions in a polarized image including polarization pixels with a plurality of polarization directions.

### Claim 2 (depends on 1)

The information processing apparatus according to claim 1, wherein
the information compression unit sets reference image information based on polarized image information of a reference polarization pixel in the polarized image, generates difference information between polarized image information of each polarization pixel different from the reference polarization pixel in the polarized image and the reference image information, and reduces an amount of information of the difference information generated for each polarization pixel to generate the compressed image information including the reference image information and the difference information with the reduced amount of information.

### Claim 3 (depends on 2)

The image processing apparatus according to claim 2, wherein
the reference image information includes information indicating an intensity value of the reference polarization pixel of each color and each polarization direction, and
the difference information includes information indicating a difference between the intensity value of the reference polarization pixel and an intensity value of a polarization pixel of a same color and a same polarization direction different from the reference polarization pixel.

### Claim 4 (depends on 2)

The image processing apparatus according to claim 2, wherein
the information compression unit generates the reference image information and the difference information based on a nature of polarization.

### Claim 5 (depends on 4)

The image processing apparatus according to claim 4, wherein
the reference image information includes information indicating a reference average intensity value calculated for each color by using intensity values of reference polarization pixels with a plurality of polarization directions and includes information indicating a reference intensity value that is an intensity value of the reference polarization pixel with a polarization direction designated in advance from combinations of polarization pixels with polarization directions that allow to calculate an average intensity value, and
the difference information includes information indicating a difference between an average intensity value, which is calculated for each color by using intensity values of polarization pixels different from the reference polarization pixels, and the reference average intensity value of a same color and includes information indicating a difference between the intensity value of the polarization pixel different from the reference polarization pixel and the reference intensity value of a same color and a same polarization direction.

### Claim 6 (depends on 4)

The image processing apparatus according to claim 4, wherein
the reference image information includes information indicating a reference average intensity value calculated for each color by using intensity values of reference polarization pixels with a plurality of polarization directions, and
the difference information includes information indicating a difference between an average intensity value, which is calculated for each color by using intensity values of polarization pixels with a plurality of polarization directions different from the reference polarization pixels, and the reference average intensity value of a same color and includes information indicating a difference between a ratio of an intensity value in a pixel block of the polarization pixels different from the reference polarization pixels and a ratio of an intensity value of polarization pixels in an adjacent pixel block.

### Claim 7 (depends on 4)

The image processing apparatus according to claim 4, wherein
the reference image information includes information indicating a reference average intensity value calculated for each color by using intensity values of reference polarization pixels with a plurality of polarization directions,
the difference information includes information indicating a difference between an average intensity value, which is calculated for each color by using intensity values of polarization pixels with a plurality of polarization directions different from the reference polarization pixels, and the reference average intensity value of a same color and includes information indicating a difference between each of the intensity values of the polarization pixels with the plurality of polarization directions different from the reference polarization pixels and a predicted value of a same polarization direction, and
the predicted value is calculated for each polarization direction based on the average intensity value calculated in a pixel block of the polarization pixels different from the reference polarization pixels and based on a ratio of an intensity value of each polarization pixel in a pixel block adjacent to the pixel block.

### Claim 8 (depends on 4)

The image processing apparatus according to claim 4, wherein
the polarized image includes an unpolarized normal pixel,
the reference image information includes information indicating an unpolarized reference intensity value that is an intensity value of a reference normal pixel of each color and includes information indicating a reference intensity value that is an intensity value of a reference polarization pixel with a polarization direction designated in advance from combinations of polarization pixels with polarization directions that allow to calculate the average intensity value, and
the difference information includes information indicating a difference between an intensity value of a normal pixel different from the reference normal pixel and an unpolarized reference intensity value of a same color and includes information indicating a difference between an intensity value of a polarization pixel with a same polarization direction that is a polarization pixel different from the reference polarization pixel and a reference intensity value with the same polarization direction.

### Claim 9 (depends on 4)

The image processing apparatus according to claim 4, wherein
the polarized image includes an unpolarized normal pixel,
the reference image information includes information indicating an unpolarized reference intensity value that is an intensity value of a reference normal pixel of each color, and
the difference information includes information indicating a difference between an intensity value of a normal pixel different from the reference normal pixel and an unpolarized reference intensity value of a same color and includes information indicating a difference between a ratio of an intensity value of a designated polarization pixel in a pixel block, which includes the designated polarization pixel with a polarization direction designated in advance from combinations of normal pixels different from the reference normal pixels and polarization pixels with polarization directions that allow to calculate an average intensity value, and a ratio of an intensity value of the designated polarization pixel in an adjacent pixel block.

### Claim 10 (depends on 4)

The information processing apparatus according to claim 4, wherein
the polarized image includes an unpolarized normal pixel,
the reference image information includes information indicating an unpolarized reference intensity value that is an intensity value of a reference normal pixel of each color,
the difference information includes information indicating a difference between an intensity value of a normal pixel different from the reference normal pixel and an unpolarized reference intensity value of a same color and includes information indicating a difference between an intensity value of a designated polarization pixel in a pixel block, which includes the designated polarization pixel with a polarization direction designated in advance from combinations of normal pixels different from the reference normal pixels and polarization pixels with polarization directions that allow to calculate an average intensity value, and a predicted value of a same polarization direction, and
the predicted value is calculated for each polarization direction based on a ratio of the intensity value of the normal pixel different from the reference normal pixel and an intensity value of the designated polarization pixel in a pixel block adjacent to the pixel block.

### Claim 11 (depends on 2)

The image processing apparatus according to claim 2, further comprising:
a characteristic information storage unit that stores characteristic information regarding acquisition of the polarized image including the polarization pixels with the plurality of polarization directions, wherein
the information compression unit uses the characteristic information stored in the characteristic information storage unit to generate the compressed image information.

### Claim 12 (depends on 11)

The image processing apparatus according to claim 11, wherein
the characteristic information indicates an extinction ratio or a transmission axis transmittance of a polarization imaging unit that has acquired the polarized image, and
the information compression unit uses the extinction ratio or the transmission axis transmittance to correct the polarized image and uses the polarized image after the correction to generate the compressed image information.

### Claim 13 (depends on 11)

The image processing apparatus according to claim 11, wherein
the characteristic information indicates a polarization direction of polarization illumination light used to acquire the polarized image, and
the information compression unit generates the reference image information and the difference information based on the polarization direction of the polarization illumination light.

### Claim 14 (depends on 2)

The image processing apparatus according to claim 2, wherein
the information compression unit sets a bit width of the difference information to a width smaller than a bit width of the reference image information to reduce the amount of information of the difference information.

### Claim 15

An image processing method comprising:
by an information compression unit, generating compressed image information by reducing an amount of information of polarized image information of each of polarization pixels with at least a plurality of polarization directions in a polarized image including polarization pixels with a plurality of polarization directions.

### Claim 16

A program causing a computer to execute a compression process of a polarized image, the program causing the computer to execute:
a procedure of acquiring the polarized image including polarization pixels with a plurality of polarization directions; and
a procedure of generating compressed image information by reducing an amount of information of polarized image information of each of polarization pixels with at least a plurality of polarization directions in the polarized image.

### Claim 17

An image processing apparatus comprising:
an information decoding unit that generates a polarized image by executing a decoding process of compressed image information generated by an information compression unit reducing an amount of information of polarized image information of each of polarization pixels with at least a plurality of polarization directions in the polarized image including polarization pixels with a plurality of polarization directions.

### Claim 18 (depends on 17)

The image processing apparatus according to claim 17, wherein
the compressed image information includes reference image information and difference information with reduced amount of information, the compressed image information being obtained by setting the reference image information based on polarized image information of a reference polarization pixel in the polarized image, generating the difference information between polarized image information of each polarization pixel different from the reference polarization pixel in the polarized image and the reference image information, and reducing the amount of information of the difference information generated for each polarization pixel, and
the information decoding unit generates an intensity value of each pixel of the polarized image based on the reference image information and the difference information.
