# US 10,122,986 B2 · Image processing device and image processing method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,122,986 B2 - Document ID: US10122986B2 - Status: Granted - Kind: B2 - Application number: 13/981,650 - Filed: 2012-02-16 - Published: 2013-12-19 - Granted: 2018-11-06 - Earliest claimed: 2011-02-18 - Assignees: Sony Corporation - Applicants: Shinobu Hattori; Yoshitomo Takahashi - Inventors: Shinobu Hattori; Yoshitomo Takahashi - Web page: https://patentgrep.com/patent/US10122986B2 ## CPC - H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation - G06T 7/13: Edge detection - H04N 13/106: Processing image signals (for multi-view video sequence encoding H04N19/597) - H04N 13/139: Format conversion, e.g. of frame-rate or size - H04N 13/161: Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597) - G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining - H04N 13/243: using three or more two-dimensional [2D] image sensors - H04N 2213/005: Aspects relating to the "3D+depth" image format - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract The present technique relates to an image processing device and an image processing method for realizing accurate depth image warping operations at the receiving side when depth images with reduced resolutions are transmitted. A parallax image generating unit reduces the resolutions of parallax images B1′ and C1′. A boundary information generating unit generates boundary information about the parallax images B1′ and C1′. A multiplexing unit transmits a multiplexed stream containing the parallax images B1′ and C1′ with the reduced resolutions and the boundary information. The present technique can be applied to encoding devices that encode glasses-free 3D images, for example. ## Claims ### Claim 1 An image processing device, comprising: at least one processor configured to: reduce a resolution of a depth image; detect a boundary position on the depth image, where pixel values in the depth image vary, based on a pixel value difference that is larger than a threshold value, wherein the pixel value difference is between a plurality of first adjacent pixels in a horizontal direction in the depth image; generate boundary information on a pixel basis, wherein the boundary information indicates the plurality of first adjacent pixels adjacent to the boundary position, based on the boundary position; generate left-side boundary information on the pixel basis, wherein the left-side boundary information indicates a plurality of second adjacent pixels located on a left side of the boundary position among the plurality of first adjacent pixels indicated by the boundary information, wherein, in an increased resolution depth image derived from the depth image having reduced resolution, a first pixel value of a plurality of first pixels adjacent to a plurality of second pixels is changed to a second pixel value of the plurality of second pixels, and wherein the first pixel value is changed to the second pixel value based on the boundary information and the left-side boundary information; and transmit the depth image having the reduced resolution, the boundary information, and the left-side boundary information. ### Claim 2 (depends on 1) The image processing device according to claim 1, wherein the at least one processor are further configured to transmit the depth image having the reduced resolution, the boundary information, and resolution conversion information. ### Claim 3 (depends on 1) The image processing device according to claim 1, wherein the at least one processor is further configured to: encode the depth image, having the reduced resolution, on a block basis, generate the boundary information on one of the pixel basis or on the block basis, and transmit the encoded depth image, and the generated boundary information. ### Claim 4 (depends on 1) The image processing device according to claim 1, wherein the at least one processor is further configured to transmit a viewpoint image, the depth image having the reduced resolution, and the boundary information. ### Claim 5 (depends on 1) The image processing device according to claim 1, wherein the at least one processor is further configured to set the generated boundary information to “1” based on the pixel value difference that is larger than the threshold value. ### Claim 6 (depends on 1) The image processing device according to claim 1, wherein the at least one processor is further configured to set the generated boundary information to “0” based on the pixel value difference that is smaller than the threshold value. ### Claim 7 (depends on 1) The image processing device according to claim 1, wherein a plurality of pixels adjacent to the boundary position include at least one pattern, wherein the at least one pattern is different for the plurality of first adjacent pixels. ### Claim 8 (depends on 7) The image processing device according to claim 7, wherein the at least one processor is further configured to detect the boundary position based on a difference of the at least one pattern for the plurality of first adjacent pixels. ### Claim 9 An image processing method, comprising: in an image processing device: reducing a resolution of a depth image; detecting a boundary position on the depth image, where pixel values in the depth image vary, based on a pixel value difference that is larger than a threshold value, wherein the pixel value difference is between a plurality of first adjacent pixels in a horizontal direction of the depth image; generating boundary information on a pixel basis, wherein the boundary information indicates the plurality of first adjacent pixels adjacent to the boundary position, based on the boundary position; generating left-side boundary information on the pixel basis, wherein the left-side boundary information indicates a plurality of second adjacent pixels located on a left side of the boundary position among the plurality of first adjacent pixels indicated by the boundary information, wherein, in an increased resolution depth image derived from the depth image having reduced resolution, a first pixel value of a plurality of first pixels adjacent to a plurality of second pixels is changed to a second pixel value of the plurality of second pixels, and wherein the first pixel value is changed to the second pixel value based on the boundary information and the left-side boundary information; and transmitting the depth image having the reduced resolution, the generated boundary information, and the left-side boundary information. ### Claim 10 An image processing device, comprising: at least one processor configured to: receive a first depth image having a reduced resolution, and boundary information which indicates a plurality of first pixels adjacent to a boundary position where a plurality of pixel values in the first depth image vary; increase resolution of the first depth image; generate a second depth image of a virtual viewpoint based on the first depth image having the increased resolution, the boundary information, and a position of the virtual viewpoint; and based on the boundary information, change a first pixel value of the plurality of first pixels indicated by the boundary information to a second pixel value of a plurality of second pixels adjacent to the plurality of first pixels in the first depth image having the increased resolution. ### Claim 11 (depends on 10) The image processing device according to claim 10, wherein the at least one processor is further configured to: receive the first depth image having the reduced resolution, the boundary information on a pixel basis, and left-side boundary information that indicates a plurality of third pixels located on a left side of the boundary position among the plurality of first pixels indicated by the boundary information, and based on the boundary information, the left-side boundary information, and the position of the virtual viewpoint, process the first depth image having the increased resolution. ### Claim 12 (depends on 11) The image processing device according to claim 11, wherein, based on the boundary information and the left-side boundary information, the at least one processor is further configured to change a third pixel value of the plurality of first pixels indicated by the boundary information and the left-side boundary information to a fourth pixel value of left-side pixels adjacent to the plurality of first pixels in the first depth image having the increased resolution. ### Claim 13 (depends on 12) The image processing device according to claim 12, wherein the at least one processor is further configured to change a fifth pixel value of a plurality of fourth pixels different from the first pixel value of the plurality of first pixels indicated by the boundary information and the left-side boundary information to a sixth pixel value of right-side pixels adjacent to the plurality of first pixels. ### Claim 14 (depends on 10) The image processing device according to claim 10, wherein the at least one processor is further configured to: receive the first depth image having the reduced resolution, the boundary information, and resolution conversion information for the reduced resolution, and increase resolution of the received first depth image based on the resolution conversion information. ### Claim 15 (depends on 14) The image processing device according to claim 14, wherein the at least one processor is further configured to increase resolution of the first depth image based on interpolation of a seventh pixel value of each pixel in the first depth image having the increased resolution with an eight pixel value of each pixel in the first depth image having the reduced resolution. ### Claim 16 (depends on 14) The image processing device according to claim 14, wherein the at least one processor is further configured to increase resolution of the first depth image based on linear interpolation of the first depth image having the reduced resolution. ### Claim 17 (depends on 10) The image processing device according to claim 10, wherein the at least one processor is further configured to: decode an encoded first depth image, wherein an encode of the first depth image is executed on a block basis, receive the encoded first depth image, and the boundary information on one of a pixel basis or on the block basis, decode the encoded first depth image based on the block basis, and increase resolution of the decoded first depth image. ### Claim 18 (depends on 10) The image processing device according to claim 10, wherein the at least one processor is further configured to generate a viewpoint image of the virtual viewpoint, based on the second depth image of the virtual viewpoint. ### Claim 19 (depends on 18) The image processing device according to claim 18, wherein the at least one processor is further configured to interpolate an occlusion region in the viewpoint image of the virtual viewpoint, wherein the occlusion region is a first region which exists in the viewpoint image of the virtual viewpoint while the occlusion region is a second region non-existent in the viewpoint image corresponding to the second depth image. ### Claim 20 (depends on 19) The image processing device according to claim 19, wherein the at least one processor is further configured to: receive a first viewpoint image of a first viewpoint, a second viewpoint image of a second viewpoint, the first depth image having the reduced resolution, and the boundary information, generate a third depth image of the virtual viewpoint based on a fourth depth image of the first viewpoint having an increased resolution, the boundary information about the first viewpoint, and the position of the virtual viewpoint, and generate a fifth depth image of the virtual viewpoint based on a sixth depth image of the second viewpoint having the increased resolution, the boundary information about the second viewpoint, and the position of the virtual viewpoint, generate the first viewpoint image of the virtual viewpoint based on the first viewpoint and the third depth image of the viewpoint image, generate the second viewpoint image of the virtual viewpoint based on the second viewpoint and the fifth depth image, and interpolate the occlusion region of one of the first viewpoint image or the second viewpoint image of the virtual viewpoint, with the occlusion region of the other one of the first viewpoint image or the second viewpoint image. ### Claim 21 An image processing method, comprising: receiving a first depth image having a reduced resolution, and boundary information which indicates a plurality of first pixels adjacent to a boundary position where a plurality of pixel values in the first depth image vary; increasing resolution of the received first depth image; and generating a second depth image of a virtual viewpoint based on the first depth image having the increased resolution, the boundary information and a position of the virtual viewpoint, wherein, based on the boundary information, changing a first pixel value of the plurality of first pixels indicated by the boundary information to a second pixel value of a plurality of second pixels adjacent to the plurality of first pixels in the first depth image having the increased resolution.