# US 10,122,988 B2 · Image encoding apparatus, image decoding apparatus, methods of controlling the same, and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,122,988 B2 - Document ID: US10122988B2 - Status: Granted - Kind: B2 - Application number: 15/126,326 - Filed: 2015-03-31 - Published: 2017-03-23 - Granted: 2018-11-06 - Earliest claimed: 2014-04-17 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Masaki Kitago - Web page: https://patentgrep.com/patent/US10122988B2 ## CPC - H04N 13/161: Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597) - H04N 1/41: Bandwidth or redundancy reduction (by scanning H04N1/17 {; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00}) - H04N 1/648: {Transmitting or storing the primary (additive or subtractive) colour signals; Compression thereof (H04N1/642 - H04N1/646 take precedence)} - H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation - H04N 13/178: Metadata, e.g. disparity information - H04N 13/232: using fly-eye lenses, e.g. arrangements of circular lenses - H04N 13/257: Colour aspects - H04N 19/597: specially adapted for multi-view video sequence encoding - H04N 19/635: characterised by filter definition or implementation details - H04N 23/67: Focus control based on electronic image sensor signals - H04N 23/80: Camera processing pipelines; Components thereof - H04N 23/843: {Demosaicing, e.g. interpolating colour pixel values} - H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio - H04N 23/957: Light-field or plenoptic cameras or camera modules - H04N 25/134: {based on three different wavelength filter elements} - H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract This invention enables more efficiently storing and managing light field data. To accomplish this, when encoding N (N>1) viewpoint images captured by a light field imaging unit, an encoding apparatus calculates the average value of the values of N corresponding pixels corresponding to each other in the N viewpoint images and generates an image having the average value as the value of a pixel and focused at a predetermined focal position. The encoding apparatus generates (N−1) types of difference values used to derive the values of the corresponding pixels in the N viewpoint images from the value of the pixel of the generated image focused at the predetermined focal position. The encoding apparatus encodes the generated image and difference information formed from the (N−1) types of difference values. ## Claims ### Claim 1 An image encoding apparatus that encodes N (N is an integer: N≥2) viewpoint images captured at viewpoints different from each other, the apparatus comprising: at least one processor; and a memory storing a program to be executed by the at least one processor; wherein the at least one processor functions as units comprising: (1) a first generation unit configured to calculate an average value of values of N corresponding pixels corresponding to each other in the N viewpoint images and generating an image having the average value as a value of a pixel; (2) a second generation unit configured to generate (N−1) types of difference values from the corresponding pixels in the N viewpoint images, wherein the (N−1) types of difference values are used to derive the values of the corresponding pixels in the N viewpoint images from the value of the pixel in the average image generated by the first generation unit; and (3) an encoding unit configured to encode the image generated by the first generation unit and difference information formed from the (N−1) types of difference values generated by the second generation unit. ### Claim 2 (depends on 1) The apparatus according to claim 1, wherein the first generation unit and the second generation unit perform generation using Haar wavelet transform. ### Claim 3 (depends on 1) The apparatus according to claim 1, wherein the first generation unit includes a determination unit configured to determine, based on a focus position designated by a user, positions of pixels serving as the corresponding pixels in the N viewpoint images. ### Claim 4 (depends on 1) The apparatus according to claim 1, wherein the encoding unit generates a file whose file header stores information representing a storing position of each of encoded data generated from the image generated by the first generation unit and (N−1) encoded data generated from the (N−1) types of difference information generated by second generation unit. ### Claim 5 (depends on 1) The apparatus according to claim 1, further comprising: an input unit configured to input light field RAW data before demosaicing, which is captured by a light field imaging unit; and a demosaicing unit configured to generate an image of each viewpoint from the light field RAW data input by the input unit. ### Claim 6 (depends on 1) The apparatus according to claim 1, further comprising: an input unit configured to input light field RAW data before demosaicing, which is obtained by imaging of a light field imaging unit; and an image generation unit configure to generate, if a sensor for each color filter provided in the light field imaging unit has a Bayer arrangement, for each color filter, an image having a value detected by each sensor as a value of a pixel, wherein the first generation unit and the second generation unit process the image of each color filter generated by the image generation unit. ### Claim 7 (depends on 1) The apparatus according to claim 1, wherein the encoding unit entropy-encodes, by spatial frequency conversion, each of the image generated by the first generation unit and the (N−1) pieces of difference information generated by the second generation unit. ### Claim 8 A method of controlling an image encoding apparatus that encodes N (N is an integer: N≥2) viewpoint images captured at viewpoints different from each other, the method comprising: a first generation step of calculating an average value of values of N corresponding pixels corresponding to each other in the N viewpoint images and generating an image having the average value as a value of a pixel; a second generation step of generating (N−1) types of difference values from the corresponding pixels in the N viewpoint images, wherein the (N−1) types of difference values are used to derive the values of the corresponding pixels in the N viewpoint images from the value of the pixel generated in the first generation step; and an encoding step of encoding the image generated in the first generation step and difference information formed from the (N−1) types of difference values generated in the second generation step. ### Claim 9 A computer-readable storage medium storing a program which causes a computer to execute steps of a method of encoding N (N is an integer: N≥2) viewpoint images captured at viewpoints different from each other, the method comprising: a first generation step of calculating an average value of values of N corresponding pixels corresponding to each other in the N viewpoint images and generating an image having the average value as a value of a pixel; a second generation step of generating (N−1) types of difference values from the corresponding pixels in the N viewpoint images, wherein the (N−1) types of difference values are used to derive the values of the corresponding pixels in the N viewpoint images from the value of the pixel generated in the first generation step; and an encoding step of encoding the image generated in the first generation step and difference information formed from the (N−1) types of difference values generated in the second generation step. ### Claim 10 An image decoding apparatus that decodes encoded data of N (N is an integer: N≥2) viewpoint images captured at viewpoints different from each other, the apparatus comprising: at least one processor; and a memory storing a program to be executed by the at least one processor, wherein the at least one processor functions as units comprising: (1) a storage unit configured to store first encoded data obtained by encoding an image focused at a predetermined focal position and having, as a value of a pixel, an average value of values of N corresponding pixels corresponding to each other in the N viewpoint images and second encoded data obtained by encoding difference information formed from (N−1) types of difference values from the corresponding pixels in the N viewpoint images, wherein the (N−1) types of difference values are used to derive the values of the corresponding pixels in the N viewpoint images from the image having the average value of the values of the N corresponding pixels as the value of the pixel; and (2) a decoding unit configured to decode at least one of the encoded data stored in storage unit. ### Claim 11 (depends on 10) The apparatus according to claim 10, further comprising an image processing unit configured to perform image processing for one of the image and the difference information decoded by the decoding unit and to generate image data. ### Claim 12 (depends on 10) The apparatus according to claim 10, wherein the decoding unit switches between whether to decode only the first encoded data, whether to decode encoded data corresponding to some types of the first encoded data and the second encoded data, and whether to decode all of the first encoded data and the second encoded data. ### Claim 13 (depends on 10) The apparatus according to claim 10, further comprising an input unit configured to input an instruction of refocus processing by a user, wherein the decoding unit decodes only the first encoded data when the input unit has input refocus processing without changing a focus position. ### Claim 14 (depends on 10) The apparatus according to claim 10, further comprising an input unit configured to input an instruction of refocus processing by a user, wherein the decoding unit decodes all of the first encoded data and the second encoded data when refocus processing by changing a focus position is input. ### Claim 15 (depends on 10) The apparatus according to claim 10, further comprising an input unit configured to input an instruction of stereo image generation by a user, wherein the decoding unit decodes only encoded data corresponding to some pieces of difference information out of the first encoded data and the second encoded data when the input unit has input the stereo image generation. ### Claim 16 A method of controlling an image decoding apparatus that decodes encoded data of N (N is an integer: N≥2) viewpoint images captured at viewpoints different from each other, the method comprising: storing, in a predetermined storage unit, first encoded data obtained by encoding an image focused at a predetermined focal position and having, as a value of a pixel, an average value of values of N corresponding pixels corresponding to each other in the N viewpoint images and second encoded data obtained by encoding difference information formed from (N−1) types of difference values from the corresponding pixels in the N viewpoint images, wherein the (N−1) types of difference values are used to derive the values of the corresponding pixels in the N viewpoint images from the image having the average value of the values of the N corresponding pixels as the value of the pixel; and decoding at least one of the encoded data stored in the predetermined storage unit.