# US 2013/0278596 A1 · Image generation apparatus, image display apparatus, image generation method and non-transitory computer readable medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2013/0278596 A1 - Document ID: US20130278596A1 - Status: Published application - Kind: A1 - Application number: 13/866,548 - Filed: 2013-04-19 - Published: 2013-10-24 - Earliest claimed: 2012-04-20 - Assignees: NLT Technologies, Ltd. - Applicants: Nlt Technologies, Ltd. - Inventors: Di WU; Michiaki SAKAMOTO - Web page: https://patentgrep.com/patent/US20130278596A1 ## CPC - G06T 15/00: Three-dimensional [3D] image rendering - H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation - H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Provided are an image generation apparatus, three-dimensional image display apparatus, image generation method and non-transitory computer readable medium. An image generation apparatus for generating a virtual-viewpoint image, includes a disparity-image-generating section which generates a disparity image corresponding to a two-dimensional image, by using a depth image and a parameter of a virtual camera; a disparity-image-correcting section which corrects the disparity image based on contour lines extracted from the two-dimensional image and the disparity image, or corrects the depth image to be used for generating the disparity image based on contour lines extracted from the two-dimensional image and the depth image; an pixel-shift-processing section which shifts each pixel of the two-dimensional image according to the corrected disparity image, to generate an intermediate image; and a unfixed-pixel-area-interpolating section which interpolates unfixed pixels to which the shifted pixels have been assigned in the intermediate image, to generate the virtual-viewpoint image. ## Claims ### Claim 1 An image generation apparatus which uses a two-dimensional image, a depth image representing depth information of the two-dimensional image, and a parameter of a virtual camera arranged at a virtual location in a three-dimensional space, to generate a virtual-viewpoint image to be captured with the virtual camera at the virtual location, the image generation apparatus comprising: a disparity-image-generating section which generates a disparity image representing a disparity value of each pixel of the two-dimensional image, by using the depth image and the parameter of the virtual camera; a disparity-image-correcting section which performs one of correcting the disparity value of each of pixels of the disparity image generated by the disparity-image-generating section, based on a contour line extracted from the two-dimensional image and a contour line extracted from the disparity image, and correcting the depth information of each of pixels of the depth image to be used for generating the disparity image in the disparity-image-generating section, based on a contour line extracted from the two-dimensional image and a contour line extracted from the depth image; an pixel-shift-processing section which shifts each pixel of the two-dimensional image according to the disparity image which has been corrected, to generate an intermediate image; and a unfixed-pixel-area-interpolating section which interpolates unfixed pixels to which shifted pixels of the two-dimensional image have not been assigned in the intermediate image, to generate the virtual-viewpoint image. ### Claim 2 (depends on 1) The image generation apparatus of claim 1, wherein the disparity-image-correcting section extracts the contour line from the two-dimensional image and the contour line from the disparity image, estimates an error area out of the disparity image based on the contour line of the two-dimensional image and the contour line of the disparity image and corrects the disparity value of each pixel in the error area, and wherein the disparity-image-correcting section, for estimating the error area, defines a pixel of the contour line of the two-dimensional image as a reference pixel, the pixel being located on a predetermined one of lines which are one of rows and columns, determines two pixels of the contour line of the disparity image such that the two pixels are separated in a positive direction and a negative direction along the other of the rows and columns at a maximum distance within an area including lines which are the one of rows and columns in a vicinity of the predetermined one of lines, and estimates an area located between the two pixels and extending along the other of rows and columns as the error area. ### Claim 3 (depends on 1) The image generation apparatus of claim 1, wherein the disparity-image-correcting section extracts the contour line from the two-dimensional image and the contour line from the depth image, estimates an error area out of the depth image based on the contour line of the two-dimensional image and the contour line of the depth image and corrects the depth information of each pixel in the error area, and wherein the disparity-image-correcting section, for estimating the error area, defines a pixel of the contour line of the two-dimensional image as a reference pixel, the pixel being located on a predetermined one of lines which are one of rows and columns, determines two pixels of the contour line of the depth image such that the two pixels are separated in a positive direction and a negative direction along the other of the rows and columns at a maximum distance within an area including lines which are the one of rows and columns in a vicinity of the predetermined one of lines, and estimates an area located between the two pixels and extending along the other of rows and columns as the error area. ### Claim 4 An image generation apparatus which uses a plurality of two-dimensional images to generate a virtual-viewpoint image to be captured with a virtual camera arranged at a virtual location in a three-dimensional space, the image generation apparatus comprising: a disparity-image-generating section which generates a disparity image representing a disparity value of each pixel of one of the plurality of two-dimensional image, by using the plurality of two-dimensional images; a disparity-image-correcting section which performs correcting the disparity value of each of pixels of the disparity image generated by the disparity-image-generating section, based on a contour line extracted from the one of the plurality of two-dimensional images and a contour line extracted from the disparity image; a pixel-shift-processing section which shifts each pixel in the one of the plurality of two-dimensional images according to the disparity image which has been corrected, to generate an intermediate image; and a unfixed-pixel-area-interpolating section which interpolates unfixed pixels to which shifted pixels of the one of the two-dimensional image have not been assigned in the intermediate image, to generate the virtual-viewpoint image. ### Claim 5 (depends on 4) The image generation apparatus of claim 4, wherein the disparity-image-correcting section extracts a contour line from the one of the two-dimensional images and a contour line from the disparity image, estimates an error area out of the disparity image based on the contour line of the one of the two-dimensional images and the contour line of the disparity image and corrects the disparity value of each pixel in the error area, and wherein the disparity-image-correcting section, for estimating the error area, defines a pixel of the contour line of the one of the two-dimensional images as a reference pixel, the pixel being located on a predetermined one of lines which are one of rows and columns, determines two pixels of the contour line of the disparity image such that the two pixels are separated in a positive direction and a negative direction along the other of the rows and columns at a maximum distance within an area including lines which are the one of rows and columns in a vicinity of the predetermined one of lines, and estimates an area located between the two pixels and extending along the other of rows and columns as the error area. ### Claim 6 An image generation apparatus which uses a two-dimensional image, a depth image representing depth information of the two-dimensional image, and a parameter of a virtual camera arranged at a virtual location in a three-dimensional space, to generate a virtual-viewpoint image to be captured with the virtual camera at the virtual location, the image generation apparatus comprising: a disparity-image-generating section which generates a disparity image representing a disparity value of each pixel of the two-dimensional image, by using the depth image and the parameter of the virtual camera; a pixel-shift-processing section which shifts each pixel of the two-dimensional image according to the disparity image, to generate an intermediate image; a noise-in-unfixed-pixel-area-eliminating section which eliminates noise in an area of unfixed pixels to which shifted pixels of the two-dimensional image have not been assigned in the intermediate image; and a unfixed-pixel-area-interpolating section which interpolates the unfixed pixels in the intermediate image wherein the noise has been eliminated, to generate the virtual-viewpoint image. ### Claim 7 (depends on 6) The image generation apparatus of claim 6, wherein the noise-in-unfixed-pixel-area-eliminating section comprises a noise-judging section which compares a length of a pixel area located between two areas of unfixed pixels and a predetermined threshold for judging noise, to judge whether the pixel area corresponds to noise or not, and a noise-eliminating section which performs a noise elimination in the pixel area when the pixel area is judged as noise. ### Claim 8 (depends on 6) The image generation apparatus of claim 6, wherein the noise-in-unfixed-pixel-area-eliminating section comprises a noise-judging section which compares a ratio of a total length of two areas of unfixed pixels to a length of a pixel area located between the two areas of unfixed pixels, and a predetermined threshold for judging noise, to judge whether the pixel area corresponds to noise or not, and a noise-eliminating section which performs a noise elimination in the pixel area when the pixel area is judged as noise. ### Claim 9 An image generation apparatus which uses a plurality of two-dimensional images to generate a virtual-viewpoint image to be captured with a virtual camera arranged at a virtual location in a three-dimensional space, the image generation apparatus comprising: a disparity-image-generating section which generates a disparity image representing a disparity value of each pixel of one of the plurality of two-dimensional image, by using the plurality of two-dimensional images; a pixel-shift-processing section which shifts each pixel in the one of the plurality of two-dimensional images according to the disparity image, to generate an intermediate image; a noise-in-unfixed-pixel-area-eliminating section which eliminates noise in an area of unfixed pixels to which shifted pixels of the one of the two-dimensional images have not been assigned in the intermediate image; and a unfixed-pixel-area-interpolating section which interpolates the unfixed pixels in the intermediate image wherein the noise has been eliminated, to generate the virtual-viewpoint image. ### Claim 10 (depends on 9) The image generation apparatus of claim 9, wherein the noise-in-unfixed-pixel-area-eliminating section comprises a noise-judging section which compares a length of a pixel area located between two areas of unfixed pixels and a predetermined threshold for judging noise, to judge whether the pixel area corresponds to noise or not, and a noise-eliminating section which performs a noise elimination in the pixel area when the pixel area is judged as noise. ### Claim 11 (depends on 9) The image generation apparatus of claim 9, wherein the noise-in-unfixed-pixel-area-eliminating section comprises a noise-judging section which compares a ratio of a total length of two areas of unfixed pixels to a length of a pixel area located between the two areas of unfixed pixels, and a predetermined threshold for judging noise, to judge whether the pixel area corresponds to noise or not, and a noise-eliminating section which performs a noise elimination in the pixel area when the pixel area is judged as noise. ### Claim 12 (depends on 1) A three-dimensional image display apparatus comprising the image generation apparatus of claim 1 and a stereoscopic display panel. ### Claim 13 An image generation method which uses a two-dimensional image, a depth image representing depth information of the two-dimensional image, and a parameter of a virtual camera arranged at a virtual location in a three-dimensional space, to generate a virtual-viewpoint image to be captured with the virtual camera at the virtual location, the image generation method comprising: a disparity-image-generating step of generating a disparity image representing a disparity value of each pixel of the two-dimensional image, by using the depth image and the parameter of the virtual camera; a disparity-image-correcting step of performing one of correcting the disparity value of each of pixels of the disparity image generated in the disparity-image-generating step, based on a contour line extracted from the two-dimensional image and a contour line extracted from the disparity image, and correcting the depth information of each of pixels of the depth image to be used for generating the disparity image in the disparity-image-generating step, based on a contour line extracted from the two-dimensional image and a contour line extracted from the depth image; an pixel-shift-processing step of shifting each pixel of the two-dimensional image according to the disparity image which has been corrected, to generate an intermediate image; and a unfixed-pixel-interpolating step of interpolating unfixed pixels to which shifted pixels of the two-dimensional image have not been assigned in the intermediate image, to generate the virtual-viewpoint image. ### Claim 14 A non-transitory computer readable medium storing an image generation program for use in an apparatus which uses a two-dimensional image, a depth image representing depth information of the two-dimensional image, and a parameter of a virtual camera arranged at a virtual location in a three-dimensional space, to generate a virtual-viewpoint image to be obtained when an object is captured with the virtual camera at the virtual location, the image generation program causing the apparatus to execute processing comprising: a disparity-image-generating step of generating a disparity image representing a disparity value of each pixel of the two-dimensional image, by using the depth image and the parameter of the virtual camera; a disparity-image-correcting step of performing one of correcting the disparity value of each of pixels of the disparity image generated in the disparity-image-generating step, based on a contour line extracted from the two-dimensional image and a contour line extracted from the disparity image, and correcting the depth information of each of pixels of the depth image to be used for generating the disparity image in the disparity-image-generating step, based on a contour line extracted from the two-dimensional image and a contour line extracted from the depth image; an pixel-shift-processing step of shifting each pixel of the two-dimensional image according to the disparity image which has been corrected, to generate an intermediate image; and a unfixed-pixel-interpolating step of interpolating unfixed pixels to which shifted pixels of the two-dimensional image have not been assigned in the intermediate image, to generate the virtual-viewpoint image.