Device having image reconstructing function, method, and storage medium

An image reconstructing device comprising:

• an image information acquiring section that acquires information forming a lens array image, the lens array image being formed by a lens array comprising an array of sub-lenses including a first sub-lens and a second sub-lens, for an optical image of an object formed by a main lens;
• an interpolating section that interpolates a third sub-image formed by a third sub-lens based on a first sub-image and a second sub-image constituting the lens array image, wherein the third sub-lens is an imaginary sub-lens positioned between the first sub-lens forming the first sub-image and the second sub-lens forming the second sub-image;
• a distance information acquiring section that acquires distance information indicating a distance to a point on which the main lens focuses;
• a reconstructing section that reconstructs an image of the object formed by the main lens focused on the point at the distance indicated by the acquired distance information from the interpolated third sub-image, the first sub-image, and the second sub-image;
• and
• an image information outputting section that outputs information forming the reconstructed image,
• wherein the interpolating section comprises:
• an extracting section that extracts a first pixel line and a second pixel line on a same epipolar plane from the first sub-image and the second sub-image indicated by the acquired information;
• a creating section that creates an epipolar image in which the extracted first pixel line, a third pixel line on the epipolar plane in the third sub-image, and the extracted second pixel line are arranged in sequence in a direction of a normal line to the epipolar plane;
• a line presuming section that presumes, among lines passing through a first pixel on the first pixel line constituting the epipolar image, a line passing through a second pixel indicating the same point on the object as indicated by the first pixel among pixels constituting the second pixel line based on a difference between a pixel value of the first pixel and a pixel value of the second pixel;
• a pixel presuming section that presumes a pixel through which the presumed line passes among pixels constituting the third pixel line to be a pixel indicating the same point on the object as indicated by the first pixel and the second pixel;
• and
• a pixel value determining section that determines a pixel value of the presumed pixel based on the pixel value of the first pixel.

An image reconstructing device comprising:

• an image information acquiring section that acquires information forming a lens array image, the lens array image being formed by a lens array comprising an array of sub-lenses including a first sub-lens and a second sub-lens, for an optical image of an object formed by a main lens;
• an interpolating section that interpolates a third sub-image formed by a third sub-lens based on a first sub-image and a second sub-image constituting the lens array image, wherein the third sub-lens is an imaginary sub-lens positioned between the first sub-lens forming the first sub-image and the second sub-lens forming the second sub-image;
• a distance information acquiring section that acquires distance information indicating a distance to a point on which the main lens focuses;
• a reconstructing section that reconstructs an image of the object formed by the main lens focused on the point at the distance indicated by the acquired distance information from the interpolated third sub-image, the first sub-image, and the second sub-image;
• and
• an image information outputting section that outputs information forming the reconstructed image,
• wherein the interpolating section comprises:
• an extracting section that extracts a first pixel line and a second pixel line on a same epipolar plane from the first sub-image and the second sub-image indicated by the acquired information;
• a creating section that creates an epipolar image in which the extracted first pixel line, a third pixel line on the epipolar plane in the third sub-image, and the extracted second pixel line are arranged in sequence in a direction of a normal line to the epipolar plane;
• a line presuming section that presumes a line passing through a first pixel on the first pixel line and a second pixel on the second pixel line indicating the same point on the object among lines passing through a given pixel on the third pixel line constituting the epipolar image based on a difference between a pixel value of the first pixel and a pixel value of the second pixel;
• and
• a pixel value determining section that determines a pixel value of the given pixel based on the pixel value of the first pixel and the pixel value of the second pixel through which the presumed line passes.

An image reconstructing method, comprising:

• acquiring information forming a lens array image, the lens array image being formed by a lens array comprising an array of sub-lenses including a first sub-lens and a second sub-lens, for an optical image of an object formed by a main lens;
• interpolating a third sub-image formed by a third sub-lens based on a first sub-image and a second sub-image constituting the lens array image, wherein the third sub-lens is an imaginary sub-lens positioned between the first sub-lens forming the first sub-image and the second sub-lens forming the second sub-image;
• acquiring distance information indicating a distance to a point on which the main lens focuses;
• reconstructing an image of the object formed by the main lens focused on the point at the distance indicated by the acquired distance information from the interpolated third sub-image, the first sub-image, and the second sub-image;
• and
• outputting information forming the reconstructed image,
• wherein the interpolating comprises identifying a parallax between the first sub-image and the second sub-image based on a pixel value of the first sub-image and a pixel value of the second sub-image, and interpolating the third sub-image so that a ratio of a parallax between the first sub-image and the third sub-image to the identified parallax is equal to a ratio of a distance between the first sub-lens and the third sub-lens to a distance between the first sub-lens and the second sub-lens.

An image reconstructing method, comprising:

• acquiring information forming a lens array image, the lens array image being formed by a lens array comprising an array of sub-lenses including a first sub-lens and a second sub-lens, for an optical image of an object formed by a main lens;
• interpolating a third sub-image formed by a third sub-lens based on a first sub-image and a second sub-image constituting the lens array image, wherein the third sub-lens is an imaginary sub-lens positioned between the first sub-lens forming the first sub-image and the second sub-lens forming the second sub-image;
• acquiring distance information indicating a distance to a point on which the main lens focuses;
• reconstructing an image of the object formed by the main lens focused on the point at the distance indicated by the acquired distance information from the interpolated third sub-image, the first sub-image, and the second sub-image;
• and
• outputting information forming the reconstructed image,
• wherein the interpolating comprises:
• extracting a first pixel line and a second pixel line on a same epipolar plane from the first sub-image and the second sub-image indicated by the acquired information;
• creating an epipolar image in which the extracted first pixel line, a third pixel line on the epipolar plane in the third sub-image, and the extracted second pixel line are arranged in sequence in a direction of a normal line to the epipolar plane;
• presuming, among lines passing through a first pixel on the first pixel line constituting the epipolar image, a line passing through a second pixel indicating the same point on the object as indicated by the first pixel among pixels constituting the second pixel line based on a difference between a pixel value of the first pixel and a pixel value of the second pixel;
• presuming a pixel through which the presumed line passes among pixels constituting the third pixel line to be a pixel indicating the same point on the object as indicated by the first pixel and the second pixel;
• and
• determining a pixel value of the presumed pixel based on the pixel value of the first pixel.

An image reconstructing method, comprising:

• acquiring information forming a lens array image, the lens array image being formed by a lens array comprising an array of sub-lenses including a first sub-lens and a second sub-lens, for an optical image of an object formed by a main lens;
• interpolating a third sub-image formed by a third sub-lens based on a first sub-image and a second sub-image constituting the lens array image, wherein the third sub-lens is an imaginary sub-lens positioned between the first sub-lens forming the first sub-image and the second sub-lens forming the second sub-image;
• acquiring distance information indicating a distance to a point on which the main lens focuses;
• reconstructing an image of the object formed by the main lens focused on the point at the distance indicated by the acquired distance information from the interpolated third sub-image, the first sub-image, and the second sub-image;
• and
• outputting information forming the reconstructed image,
• wherein the interpolating comprises:
• extracting a first pixel line and a second pixel line on a same epipolar plane from the first sub-image and the second sub-image indicated by the acquired information;
• creating an epipolar image in which the extracted first pixel line, a third pixel line on the epipolar plane in the third sub-image, and the extracted second pixel line are arranged in sequence in a direction of a normal line to the epipolar plane;
• presuming a line passing through a first pixel on the first pixel line and a second pixel on the second pixel line indicating the same point on the object among lines passing through a given pixel on the third pixel line constituting the epipolar image based on a difference between a pixel value of the first pixel and a pixel value of the second pixel;
• and
• determining a pixel value of the given pixel based on the pixel value of the first pixel and the pixel value of the second pixel through which the presumed line passes.

A non-transitory computer readable storage medium having stored thereon a program that is executable by a computer to cause the computer to perform functions comprising:

• acquiring information forming a lens array image, the lens array image being formed by a lens array comprising an array of sub-lenses including a first sub-lens and a second sub-lens, for an optical image of an object formed by a main lens;
• interpolating a third sub-image formed by a third sub-lens based on a first sub-image and a second sub-image constituting the lens array image, wherein the third sub-lens is an imaginary sub-lens positioned between the first sub-lens forming the first sub-image and the second sub-lens forming the second sub-image;
• acquiring distance information indicating a distance to a point on which the main lens focuses;
• reconstructing an image of the object formed by the main lens focused on the point at the distance indicated by the acquired distance information from the interpolated third sub-image, the first sub-image, and the second sub-image;
• and
• outputting information forming the reconstructed image,
• wherein the interpolating comprises identifying a parallax between the first sub-image and the second sub-image based on a pixel value of the first sub-image and a pixel value of the second sub-image, and interpolating the third sub-image so that a ratio of a parallax between the first sub-image and the third sub-image to the identified parallax is equal to a ratio of a distance between the first sub-lens and the third sub-lens to a distance between the first sub-lens and the second sub-lens.

A non-transitory computer readable storage medium having stored thereon a program that is executable by a computer to cause the computer to perform functions comprising:

• acquiring information forming a lens array image, the lens array image being formed by a lens array comprising an array of sub-lenses including a first sub-lens and a second sub-lens, for an optical image of an object formed by a main lens;
• interpolating a third sub-image formed by a third sub-lens based on a first sub-image and a second sub-image constituting the lens array image, wherein the third sub-lens is an imaginary sub-lens positioned between the first sub-lens forming the first sub-image and the second sub-lens forming the second sub-image;
• acquiring distance information indicating a distance to a point on which the main lens focuses;
• reconstructing an image of the object formed by the main lens focused on the point at the distance indicated by the acquired distance information from the interpolated third sub-image, the first sub-image, and the second sub-image;
• and
• outputting information forming the reconstructed image,
• wherein the interpolating comprises:
• extracting a first pixel line and a second pixel line on a same epipolar plane from the first sub-image and the second sub-image indicated by the acquired information;
• creating an epipolar image in which the extracted first pixel line, a third pixel line on the epipolar plane in the third sub-image, and the extracted second pixel line are arranged in sequence in a direction of a normal line to the epipolar plane;
• presuming, among lines passing through a first pixel on the first pixel line constituting the epipolar image, a line passing through a second pixel indicating the same point on the object as indicated by the first pixel among pixels constituting the second pixel line based on a difference between a pixel value of the first pixel and a pixel value of the second pixel;
• presuming a pixel through which the presumed line passes among pixels constituting the third pixel line to be a pixel indicating the same point on the object as indicated by the first pixel and the second pixel;
• and
• determining a pixel value of the presumed pixel based on the pixel value of the first pixel.

A non-transitory computer readable storage medium having stored thereon a program that is executable by a computer to cause the computer to perform functions comprising:

• acquiring information forming a lens array image, the lens array image being formed by a lens array comprising an array of sub-lenses including a first sub-lens and a second sub-lens, for an optical image of an object formed by a main lens;
• interpolating a third sub-image formed by a third sub-lens based on a first sub-image and a second sub-image constituting the lens array image, wherein the third sub-lens is an imaginary sub-lens positioned between the first sub-lens forming the first sub-image and the second sub-lens forming the second sub-image;
• acquiring distance information indicating a distance to a point on which the main lens focuses;
• reconstructing an image of the object formed by the main lens focused on the point at the distance indicated by the acquired distance information from the interpolated third sub-image, the first sub-image, and the second sub-image;
• and
• outputting information forming the reconstructed image,
• wherein the interpolating comprises:
• extracting a first pixel line and a second pixel line on a same epipolar plane from the first sub-image and the second sub-image indicated by the acquired information;
• creating an epipolar image in which the extracted first pixel line, a third pixel line on the epipolar plane in the third sub-image, and the extracted second pixel line are arranged in sequence in a direction of a normal line to the epipolar plane;
• presuming a line passing through a first pixel on the first pixel line and a second pixel on the second pixel line indicating the same point on the object among lines passing through a given pixel on the third pixel line constituting the epipolar image based on a difference between a pixel value of the first pixel and a pixel value of the second pixel;
• and
• determining a pixel value of the given pixel based on the pixel value of the first pixel and the pixel value of the second pixel through which the presumed line passes.