# US 9,438,880 B2 · Image processing apparatus, image processing method, and program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,438,880 B2
- Document ID: US9438880B2
- Status: Granted
- Kind: B2
- Application number: 14/591,046
- Filed: 2015-01-07
- Published: 2015-07-16
- Granted: 2016-09-06
- Earliest claimed: 2014-01-14
- Assignees: Sony Corporation
- Applicants: Sony Corporation
- Inventors: Yasuhide Hyodo; Shuichi Takahashi; Satoshi Shioiri
- Web page: https://patentgrep.com/patent/US9438880B2

## CPC

- H04N 13/15: for colour aspects of image signals
- H04N 13/128: Adjusting depth or disparity
- H04N 13/122: Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

There is provided an image processing apparatus that includes a depth-adjustment-amount determination section configured to determine a depth adjustment amount of an input image, an evaluation function storage section configured to store an evaluation function, the evaluation function representing a relationship of a subjective depth amount of the image to a color-saturation component thereof, the subjective depth amount being a depth amount subjectively felt by a user, an input-value-adjustment-amount determination section configured to determine an input-value adjustment amount based on the evaluation function, the input-value adjustment amount representing a retina's response to the color-saturation component corresponding to the determined depth adjustment amount, an adjustment gain calculation section configured to calculate each spatial-frequency-component adjustment gain based on the determined input-value adjustment amount, and a gain superimposition section configured to superimpose each of the calculated spatial-frequency-component adjustment gains on the color-saturation component of the input image.

## Claims

### Claim 1

An image processing apparatus, comprising:
a depth-adjustment-amount determination section configured to determine a depth adjustment amount of an input image;
an evaluation function storage section configured to store an evaluation function, the evaluation function representing a relationship of a subjective depth amount of the image to a color-saturation component thereof, the subjective depth amount being a depth amount subjectively felt by a user;
an input-value-adjustment-amount determination section configured to determine an input-value adjustment amount based on the evaluation function, the input-value adjustment amount representing a retina's response to the color-saturation component corresponding to the determined depth adjustment amount;
an adjustment gain calculation section configured to calculate each spatial-frequency-component adjustment gain based on the determined input-value adjustment amount; and
a gain superimposition section configured to superimpose each of the spatial-frequency-component adjustment gains on the color-saturation component of the input image, the spatial-frequency-component adjustment gains being calculated by the adjustment gain calculation section.

### Claim 2 (depends on 1)

The image processing apparatus according to claim 1, wherein
the evaluation function storage section stores the evaluation function, the evaluation function representing not only the relationship of the subjective depth amount to the color-saturation component but also to a luminance-contrast component of the image,
in addition to the input-value adjustment amount representing the retina's response to the color-saturation component, the input-value-adjustment-amount determination section determines another input-value adjustment amount based on the evaluation function, the input-value-adjustment amount representing a retina's response to a luminance-contrast component corresponding to the determined depth adjustment amount,
for each of the luminance-contrast component and the color-saturation component, the adjustment gain calculation section calculates each of the spatial-frequency-component adjustment gains based on the input-value adjustment amounts, and
for each of the luminance-contrast component and the color-saturation component, the gain superimposition section superimposes each of the spatial-frequency-component adjustment gains on the input image, the spatial-frequency-component adjustment gains being calculated by the adjustment gain calculation section.

### Claim 3 (depends on 2)

The image processing apparatus according to claim 2, wherein
the input-value-adjustment-amount determination section includes a parameter for setting an adjustment balance between the luminance-contrast component and the color-saturation component, and with the adjustment balance set by the parameter, determines the input-value adjustment amounts representing the retina's response to the luminance-contrast component and to the color-saturation component.

### Claim 4 (depends on 2)

The image processing apparatus according to claim 2, wherein
the evaluation function representing the relationship of the subjective depth amount to the color-saturation component and to the luminance-contrast component satisfies
D
⁡
(
I
lum
,
I
col
)
=
⁢
{
D
lum
+
B
,
D
col
<
D
lum
D
col
+
B
,
D
col
≥
D
lum
D
lum
=
⁢
A
lum
⁢
log
10
⁢
I
lum
,
D
col
=
A
col
⁢
log
10
⁢
I
col
Dlum=Alum log10Ilum, Dcol=Acol log10Icol
where lcol denotes the color-saturation component, llum denotes the luminance-contrast component, Dcol denotes the subjective depth amount for the color-saturation component, Dlum denotes the subjective depth amount for the luminance-contrast component, and D(llum,lcol) denotes the subjective depth amount in combination of the color-saturation component and the luminance-contrast component (Alum,Acoland B are each a given constant).

### Claim 5 (depends on 2)

The image processing apparatus according to claim 2, wherein
the evaluation function representing the relationship of the subjective depth amount to the color-saturation component and to the luminance-contrast component satisfies
D(Ilum, Icol)=((Alum log10Ilum)n+(Acol log10Icol)n)1/n+B
where lcoldenotes the color-saturation component,llum denotes the luminance-contrast component, and D(llum,lcol) denotes the subjective depth amount in combination of the color-saturation component and the luminance-contrast component (Alum, Acoland B are each a given constant).

### Claim 6 (depends on 1)

The image processing apparatus according to claim 1, wherein
the evaluation function representing the relationship of the subjective depth amount to the color-saturation component satisfies
Dcol=Acol log10Icol+Bcol
where Icol denotes the color-saturation component, and Dcol denotes the subjective depth amount (Acol and Bcol are each a given constant).

### Claim 7 (depends on 1)

The image processing apparatus according to claim 1, wherein
the input image is a three-dimensional (3D) image.

### Claim 8 (depends on 7)

The image processing apparatus according to claim 7, wherein
the 3D image is input as a stereo image including a right-eye image and a left-eye image.

### Claim 9 (depends on 7)

The image processing apparatus according to claim 7, wherein
the 3D image is input as a multi-view image including three or more images viewed from three or more positions.

### Claim 10 (depends on 7)

The image processing apparatus according to claim 7, wherein
the 3D image is input in a data format being a combination of a two-dimensional (2D) image and depth information thereabout.

### Claim 11 (depends on 1)

The image processing apparatus according to claim 1, wherein
the input image is a 2D image.

### Claim 12 (depends on 1)

The image processing apparatus according to claim 1, further comprising
a color conversion section configured to convert a color system of the input image from RGB to CIE L*a*b*.

### Claim 13 (depends on 1)

The image processing apparatus according to claim 1, further comprising:
a depth information estimation section configured to estimate depth information from the input image; and
a depth-adjustment-amount defining function storage section configured to store a depth-adjustment-amount defining function defining a depth adjustment amount with respect to the depth information, the depth-adjustment-amount determination section determining the depth adjustment amount using the estimated depth information based on the depth-adjustment-amount defining function.

### Claim 14 (depends on 13)

The image processing apparatus according to claim 13, wherein
when the input image is a 2D image, the depth information estimation section estimates the depth information based on an analysis result of the 2D image.

### Claim 15 (depends on 1)

The image processing apparatus according to claim 1, further comprising
a spatial frequency weight setting section configured to set a function of a weight of each of the spatial-frequency-component adjustment gains, the adjustment gain calculation section calculating the adjustment gains by multiplying the input-value adjustment amount by the function.

### Claim 16 (depends on 1)

The image processing apparatus according to claim 1, wherein
the gain superimposition section converts the input image into a frequency domain, and converts a result of convolution integral back into an image domain, the result of convolution integral being obtained by performing the convolution integral between the frequency-domain image and each of the spatial-frequency-component adjustment gains.

### Claim 17 (depends on 16)

The image processing apparatus according to claim 16, wherein
the gain superimposition section converts the input image into the frequency domain by Fourier transform, and converts the frequency-domain image back into the image domain by inverse Fourier transform.

### Claim 18 (depends on 1)

The image processing apparatus according to claim 1, wherein
when the depth adjustment amount is positive, the input-value adjustment amount determined by the input-value-adjustment-amount determination section is also positive, and when the depth adjustment amount is negative, the input-value adjustment amount determined by the input-value-adjustment-amount determination section is also negative.

### Claim 19

An image processing method, comprising:
by an image processing apparatus configured to process an input image
determining a depth adjustment amount of an input image;
determining an input-value adjustment amount based on an evaluation function, the evaluation function representing a relationship of a subjective depth amount to a color-saturation component of the image, the subjective depth amount being a depth amount subjectively felt by a user, the input-value adjustment amount representing a retina's response to the color-saturation component corresponding to the determined depth adjustment amount;
calculating each spatial-frequency-component adjustment gain based on the determined input-value adjustment amount; and
superimposing each of the calculated spatial-frequency-component adjustment gains on the color-saturation component of the input image.

### Claim 20

At least one non-transitory computer-readable storage medium having encoded thereon executable instructions that, when executed by at least one processor, cause the at least one processor to carry out a method comprising:
determining a depth adjustment amount of an input image;
determining an input-value adjustment amount based on an evaluation function, the evaluation function representing a relationship of a subjective depth amount to a color-saturation component of the image, the subjective depth amount being a depth amount subjectively felt by a user, the input-value adjustment amount representing a retina's response to the color-saturation component corresponding to the determined depth adjustment amount;
calculating each spatial-frequency-component adjustment gain based on the determined input-value adjustment amount; and
superimposing each of the calculated spatial-frequency-component adjustment gains on the color-saturation component of the input image.