US 9,754,357 B2Granted

Image processing device, stereoscoopic device, integrated circuit, and program for determining depth of object in real space generating histogram from image obtained by filming real space and performing smoothing of histogram

AssigneePanasonic Intellectual Property Corporation of America

InventorsTsuyoshi Nakamura, Tadanori Tezuka, Yoshiteru Tanaka, Masahiro Ohashi

Life of this patent

Estimating the term…

Loading patent…

Abstract

A histogram generator generates a histogram that indicates a count of pixels of a designated color in association with coordinates along a basic axis of a screen, for frame image data obtained by filming a real space. A histogram smoother performs smoothing of the generated histogram. A three-dimensional coordinates generator selects a value associated with particular coordinates from among the counts indicated in the smoothed histogram, and performs depth value determination for a target object using the selected value.

Claims12

Dependent claims

An image processing device determining a depth of a target object in real space by performing image processing, the image processing device comprising:

a generator generating an X-axis histogram for a designated color and a Y-axis histogram for the designated color, the X-axis histogram indicating a pixel count at each of a plurality of coordinates along an X-axis, the Y-axis histogram indicating a pixel count at each of a plurality of coordinates along a Y-axis, the X-axis and the Y-axis being axes of a screen, a pixel count at a given coordinate indicating the number of pixels having the designated color at the given coordinate in frame image data obtained by filming the real space;
a smoother performing smoothing of the X-axis histogram and the Y-axis histogram to acquire a smoothed X-axis histogram and a smoothed Y-axis histogram, respectively;
and
a depth determiner determining three-dimensional coordinates of the target object using the smoothed X-axis histogram and the smoothed Y-axis histogram, wherein
the three-dimensional coordinates are expressed as (Xu, Yu, f(h1(Xu), h2(Yu))) when Xu is an X coordinate in the smoothed X-axis histogram, h1(Xu) is a pixel count at the coordinate Xu, Yu is a Y coordinate in the smoothed Y-axis histogram, h2(Yu) is a pixel count at the coordinate Yu, and f(h1(Xu), h2(Yu)) is a function for determining a Z coordinate from the pixel count h1(Xu) and the pixel count h2(Yu),
the pixel count h1(Xu) and the pixel count h2(Yu) determine a centroid intersecting at a center of the target object,
the function f(h1(Xu), h2(Yu)) determines the Z coordinate of the target object based on a length of the centroid determined by the X-axis histogram and the Y-axis histogram,
the smoothing for the smoothed X-axis histogram and the smoothed Y-axis histogram involves applying an impulse response convolution to pixel counts at the coordinates in the X-axis histogram and pixel counts at the coordinates in the Y-axis histogram,
the smoothed X-axis histogram includes a peak pixel count at the center along the X-axis and pixel counts approaching zero at edges along the X-axis, and the smoothed Y-axis histogram includes a peak pixel count at the center along the Y-axis and pixel counts approaching zero at edges along the Y-axis,
an impulse response I[n] used in the impulse response convolution is obtained by converting a frequency response H(w),
the impulse response I[n] is obtained by:
(Math 1)
I[n]=DTFT−1[H(w)]
where DTFT denotes a discrete-time Fourier transform, and
the impulse response continues from 0 to M in terms of the X coordinates and an output I[n]*x[n] of the impulse response convolution applied to an nth X coordinate x[n] among coordinates 0 through M is calculated as:
(Math 2)
I
⁡
[
n
]
*
x
⁡
[
n
]
=
∑
m
=
0
M
⁢
⁢
I
⁡
[
m
]
⁢
x
⁡
[
n
-
m
]
,
(
Math
⁢
⁢
2
)
and I[n] is defined as:
(Math 3)
I
⁡
[
n
]
=
{
I
n
,
n
≥
0
0
,
otherwise
.
(
Math
⁢
⁢
3
)

77 more elements

An image processing device, comprising

a pixel group extractor extracting, among pixels in the frame image data, a group of pixels of a designated color from each of an X-axis and a Y-axis, the X-axis and the Y-axis being axes of a screen;
a smoother performing smoothing of pixel values of the pixels in the pixel group from each of the X-axis and the Y-axis to acquire a smoothed pixel group;
a generator generating an X-axis histogram indicating a pixel count at each of a plurality of coordinates along the X-axis and a Y-axis histogram indicating a pixel count at each of a plurality of coordinates along the Y-axis, a pixel count at a given coordinate indicating the number of pixels at the given coordinate in the smoothed pixel group;
and
a depth determiner determining three-dimensional coordinates of the target object using the X-axis histogram and the Y-axis histogram, wherein
the three-dimensional coordinates are expressed as (Xu, Yu, f(h1(Xu), h2(Yu))) when Xu is an X coordinate in the X-axis histogram, h1(Xu) is a pixel count at the coordinate Xu, Yu is a Y coordinate in the Y-axis histogram, h2(Yu) is a pixel count at the coordinate Yu, and f(h1(Xu), h2(Yu)) is a function for determining a Z coordinate from the pixel count h1(Xu) and the pixel count h2(Yu),
the pixel count h1(Xu) and the pixel count h2(Yu) determine a centroid intersecting at a center of the target object,
the function f(h1(Xu), h2(Yu)) determines the Z coordinate of the target object based on a length of the centroid determined by the X-axis histogram and the Y-axis histogram,
the smoothing for the smoothed pixel group involves applying an impulse response convolution to the pixel values of the pixels in the pixel group from each of the X-axis and the Y-axis,
an impulse response I[n] used in the impulse response convolution is obtained by converting a frequency response H(w),
the impulse response I[n] is obtained by:
(Math 1)
I[n]=DTFT−1[H(w)]
where DTFT denotes a discrete-time Fourier transform, and
the impulse response continues from 0 to M in terms of the X coordinates and an output I[n]*x[n] of the impulse response convolution applied to an nth X coordinate x[n] among coordinates 0 through M is calculated as:
(Math 2)
I
⁡
[
n
]
*
x
⁡
[
n
]
=
∑
m
=
0
M
⁢
⁢
I
⁡
[
m
]
⁢
x
⁡
[
n
-
m
]
,
(
Math
⁢
⁢
2
)
and I[n] is defined as:
(Math 3)
I
⁡
[
n
]
=
{
I
n
,
n
≥
0
0
,
otherwise
.
(
Math
⁢
⁢
3
)

77 more elements

An integrated circuit determining a depth of a target object in real space by performing image processing, the integrated circuit comprising:

a generator generating an X-axis histogram for a designated color and a Y-axis histogram for the designated color, the X-axis histogram indicating a pixel count at each of a plurality of coordinates along an X-axis, the Y-axis histogram indicating a pixel count at each of a plurality of coordinates along a Y-axis, the X-axis and the Y-axis being axes of a screen, a pixel count at a given coordinate indicating the number of pixels having the designated color at the given coordinate in frame image data obtained by filming the real space;
a smoother performing smoothing of the X-axis histogram and the Y-axis histogram to acquire a smoothed X-axis histogram and a smoothed Y-axis histogram, respectively;
and
a depth determiner determining three-dimensional coordinates of the target object using the smoothed X-axis histogram and the smoothed Y-axis histogram, wherein
the three-dimensional coordinates are expressed as (Xu, Yu, f(h1(Xu), h2(Yu))) when Xu is an X coordinate in the smoothed X-axis histogram, h1(Xu) is a pixel count at the coordinate Xu, Yu is a Y coordinate in the smoothed Y-axis histogram, h2(Yu) is a pixel count at the coordinate Yu, and f(h1(Xu), h2(Yu)) is a function for determining a Z coordinate from the pixel count h1(Xu) and the pixel count h2(Yu),
the pixel count h1(Xu) and the pixel count h2(Yu) determine a centroid intersecting at a center of the target object,
the function f(h1(Xu), h2(Yu)) determines the Z coordinate of the target object based on a length of the centroid determined by the X-axis histogram and the Y-axis histogram,
the smoothing for the smoothed X-axis histogram and the smoothed Y-axis histogram involves applying an impulse response convolution to pixel counts at the coordinates in the X-axis histogram and pixel counts at the coordinates in the Y-axis histogram,
the smoothed X-axis histogram includes a peak pixel count at the center along the X-axis and pixel counts approaching zero at edges along the X-axis, and the smoothed Y-axis histogram includes a peak pixel count at the center along the Y-axis and pixel counts approaching zero at edges along the Y-axis,
an impulse response I[n] used in the impulse response convolution is obtained by converting a frequency response H(w),
the impulse response I[n] is obtained by:
(Math 1)
I[n]=DTFT−1[H(w)]
where DTFT denotes a discrete-time Fourier transform, and
the impulse response continues from 0 to M in terms of the X coordinates and an output I[n]*x[n] of the impulse response convolution applied to an nth X coordinate x[n] among coordinates 0 through M is calculated as:
(Math 2)
I
⁡
[
n
]
*
x
⁡
[
n
]
=
∑
m
=
0
M
⁢
⁢
I
⁡
[
m
]
⁢
x
⁡
[
n
-
m
]
,
(
Math
⁢
⁢
2
)
and I[n] is defined as:
(Math 3)
I
⁡
[
n
]
=
{
I
n
,
n
≥
0
0
,
otherwise
.
(
Math
⁢
⁢
3
)

77 more elements

A non-transitory recording medium recording thereon an image processing program causing a computer to execute processing for determining a depth of a target object in real space by performing image processing, the image processing program causing the computer to execute:

generating an X-axis histogram for a designated color and a Y-axis histogram for the designated color, the X-axis histogram indicating a pixel count at each of a plurality of coordinates along an X-axis, the Y-axis histogram indicating a pixel count at each of a plurality of coordinates along a Y-axis, the X-axis and the Y-axis being axes of a screen, a pixel count at a given coordinate indicating the number of pixels having the designated color at the given coordinate in frame image data obtained by filming the real space;
smoothing the X-axis histogram and the Y-axis histogram to acquire a smoothed X-axis histogram and a smoothed Y-axis histogram, respectively;
and
determining three-dimensional coordinates of the target object using the smoothed X-axis histogram and the smoothed Y-axis histogram, wherein
the three-dimensional coordinates are expressed as (Xu, Yu, f(h1(Xu), h2(Yu))) when Xu is an X coordinate in the smoothed X-axis histogram, h1(Xu) is a pixel count at the coordinate Xu, Yu is a Y coordinate in the smoothed Y-axis histogram, h2(Yu) is a pixel count at the coordinate Yu, and f(h1(Xu), h2(Yu)) is a function for determining a Z coordinate from the pixel count h1(Xu) and the pixel count h2(Yu),
the pixel count h1(Xu) and the pixel count h2(Yu) determine a centroid intersecting at a center of the target object,
the function f(h1(Xu), h2(Yu)) determines the Z coordinate of the target object based on a length of the centroid determined by the X-axis histogram and the Y-axis histogram,
the smoothing for the smoothed X-axis histogram and the smoothed Y-axis histogram involves applying an impulse response convolution to pixel counts at the coordinates in the X-axis histogram and pixel counts at the coordinates in the Y-axis histogram,
the smoothed X-axis histogram includes a peak pixel count at the center along the X-axis and pixel counts approaching zero at edges along the X-axis, and the smoothed Y-axis histogram includes a peak pixel count at the center along the Y-axis and pixel counts approaching zero at edges along the Y-axis,
an impulse response I[n] used in the impulse response convolution is obtained by converting a frequency response H(w),
the impulse response I[n] is obtained by:
(Math 1)
I[n]=DTFT−1[H(w)]
where DTFT denotes a discrete-time Fourier transform, and
the impulse response continues from 0 to M in terms of the X coordinates and an output I[n]*x[n] of the impulse response convolution applied to an nth X coordinate x[n] among coordinates 0 through M is calculated as:
(Math 2)
I
⁡
[
n
]
*
x
⁡
[
n
]
=
∑
m
=
0
M
⁢
⁢
I
⁡
[
m
]
⁢
x
⁡
[
n
-
m
]
,
(
Math
⁢
⁢
2
)
and I[n] is defined as:
(Math 3)
I
⁡
[
n
]
=
{
I
n
,
n
≥
0
0
,
otherwise
.
(
Math
⁢
⁢
3
)

77 more elements

Dependencies are inferred from the claim text.

Facts

Application: 14/003,084Filed 5 Mar 2013
Priority: JP 2012-06661223 Mar 2012
Published: US 2014/0071251 A113 Mar 2014
Granted: 5 Sep 2017
Term ends: Not estimatedThe source records don’t support an estimate.

Classification10 codes

G06T 5/70
Image enhancement or restoration: denoising; Smoothing
G06F 3/0346
Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor: with detection of the…
G06F 3/03545
With detection of two-dimensional [2D] relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D…
G06F 3/04815
Based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like…:…