# US 10,122,939 B2 · Image processing device for processing image data and map data with regard to depth distribution of a subject, image processing system, imaging apparatus, image processing method, and recording medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,122,939 B2 - Document ID: US10122939B2 - Status: Granted - Kind: B2 - Application number: 15/189,145 - Filed: 2016-06-22 - Published: 2016-12-29 - Granted: 2018-11-06 - Earliest claimed: 2015-06-26 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Masaaki Matsuoka - Web page: https://patentgrep.com/patent/US10122939B2 ## CPC - H04N 5/2621: {Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability} - G06T 5/70: Denoising; Smoothing - G06T 7/571: from focus - H04N 5/2226: {Determination of depth image, e.g. for foreground/background separation (determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10)} - H04N 19/60: using transform coding - G06T 2200/21: involving computational photography - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An image processing device processes image data and map data with regard to depth distribution of subjects. A decoding unit decodes encoded image data obtained by encoding image data of a subject image, and thereby generates decoded image data. Another decoding unit decodes encoded map data obtained by encoding map data and thereby generates decoded map data. A defocus map shaping unit acquires decoded image data and decoded map data, performs a shaping process of the decoded map data with reference to the decoded image data, and thereby generates shaped map data. A background blurring unit refers to the shaped map data and performs image processing of adding blur to a background of the decoded image. ## Claims ### Claim 1 An image processing device that processes image data and map data with regard to depth distribution of a subject, the image processing device comprising: at least one processor or at least one circuit programmed to function as: a first decoding unit configured to decode the image data that has been encoded and thereby generate decoded image data; a second decoding unit configured to decode the map data that has been encoded and thereby generate decoded map data; a first shaping unit configured to perform a shaping process of the decoded map data with reference to the decoded image data, wherein the shaping process includes an edge-preserved filtering process using coefficients generated based on the decoded image data; and a first image processing unit configured to perform image processing to the decoded image data with reference to the decoded map data that has been processed for shaping by the first shaping unit. ### Claim 2 (depends on 1) The image processing device according to claim 1, wherein the first shaping unit perform the shaping process to the decoded map data with reference to the decoded image data having higher resolution than that of the decoded map data. ### Claim 3 (depends on 1) The image processing device according to claim 1, wherein: the at least one processor or at least one circuit is further programmed to function as a second shaping unit configured to acquire the map data that has not been encoded and perform a shaping process to the data, wherein the first shaping unit refers to the decoded image data having first resolution to shape the decoded map data and outputs map data having the first resolution, and wherein the second shaping unit refers to image data having second resolution that is lower than the first resolution to shape the map data and outputs map data having the second resolution. ### Claim 4 (depends on 3) The image processing device according to claim 3, wherein: the at least one processor or at least one circuit is further programmed to function as a map generation unit configured to generate the map data that has not been encoded and output the data to the second shaping unit. ### Claim 5 (depends on 3) The image processing device according to claim 3, wherein the at least one processor or at least one circuit is further programmed to function as: a third decoding unit configured to decode the image data that has been encoded and thereby generate decoded image data; a fourth decoding unit configured to decode the map data that has been processed for shaping by the second shaping unit and then encoded, and thereby generate decoded map data; and a second image processing unit configured to perform image processing to the decoded image data generated by the third decoding unit with reference to the decoded map data generated by the fourth decoding unit. ### Claim 6 (depends on 5) The image processing device according to claim 5, wherein the first image processing unit performs image processing to the decoded image data with reference to the decoded map data having the first resolution, and wherein the second image processing unit performs image processing to the decoded image data with reference to the decoded map data having the second resolution. ### Claim 7 (depends on 6) The image processing device according to claim 6, wherein the first or the second image processing unit performs a process of blurring a background image. ### Claim 8 (depends on 1) The image processing device according to claim 1, wherein the at least one processor or at least one circuit is further programmed to function as: a first encoding unit configured to encode the image data and output the data to the first decoding unit; and a second encoding unit configured to encode the map data and output the data to the second decoding unit. ### Claim 9 (depends on 5) The image processing device according to claim 5, wherein the at least one processor or at least one circuit is further programmed to function as: a first encoding unit configured to encode the image data and output the data to the first decoding unit; a second encoding unit configured to encode the map data and output the data to the second decoding unit; a third encoding unit configured to encode the image data and output the data to the third decoding unit; and a fourth encoding unit configured to encode the map data that has been processed for shaping by the second shaping unit and output the data to the fourth decoding unit. ### Claim 10 (depends on 3) The image processing device according to claim 3, wherein the at least one processor or at least one circuit is further programmed to function as: a resizing unit configured to resize the image data, wherein the second shaping unit acquires image data that has been processed for resizing by the resizing unit and perform a shaping process to the map data. ### Claim 11 (depends on 9) The image processing device according to claim 9, wherein the at least one processor or at least one circuit is further programmed to function as: a first selection unit configured to select and output the map data encoded by the second encoding unit or the fourth encoding unit; and a second selection unit configured to select and output the image data encoded by the first encoding unit or the third encoding unit. ### Claim 12 (depends on 3) The image processing device according to claim 3, wherein the at least one processor or at least one circuit is further programmed to function as: a first encoding unit configured to encode the image data and output the data to the first decoding unit; and a second encoding unit configured to encode the map data that has not been shaped by the second shaping unit and output the data to the second decoding unit. ### Claim 13 (depends on 1) The image processing device according to claim 1, wherein, when the pixel value of a pixel of interest and the pixel value of a peripheral pixel of the foregoing pixel are to be smoothed with respect to the pixel of interest of the map data, the first shaping unit increases a weight of smoothing for an output of the peripheral pixel as the pixel value of the pixel of interest gets closer to the pixel value of the peripheral pixel. ### Claim 14 (depends on 1) The image processing device according to claim 1, wherein the shaping process includes a Joint Bilateral process. ### Claim 15 An image processing system comprising: at least one processor or at least one circuit programmed to function as an encoding device and a decoding device, wherein the encoding device comprising: a first encoding unit configured to encode image data; and a second encoding unit configured to encode map data with regard to depth distribution of a subject, and wherein the decoding device comprising: a first decoding unit configured to decode the image data encoded by the first encoding unit and thereby generate decoded image data; a second decoding unit configured to decode the map data encoded by the second encoding unit and thereby generate decoded map data; a first shaping unit configured to perform a shaping process to the decoded map data with reference to the decoded image data having higher resolution than that of the decoded map data, wherein the shaping process includes an edge-preserved filtering process using coefficients generated based on the decoded image data; and a first image processing unit configured to perform image processing to the decoded image data with reference to the decoded map data that has been processed for shaping by the first shaping unit. ### Claim 16 (depends on 15) The image processing system according to claim 15, wherein the encoding device further comprising: a second shaping unit configured to perform a shaping process to the map data that has not been encoded, wherein the first shaping unit refers to the decoded image data having first resolution to shape the decoded map data and output map data having the first resolution, and wherein the second shaping unit refers to image data having second resolution that is lower than the first resolution to shape the map data and output map data having the second resolution. ### Claim 17 (depends on 16) The image processing system according to claim 16, wherein, when the pixel value of a pixel of interest and the pixel value of a peripheral pixel of the pixel are to be smoothed with respect to the pixel of interest of the map data, the first or the second shaping unit increases a weight of smoothing for an output of the peripheral pixel as the pixel value of the pixel of interest gets closer to the pixel value of the peripheral pixel. ### Claim 18 (depends on 16) The image processing system according to claim 16, wherein the encoding device further comprising: a third encoding unit configured to encode the image data; and a fourth encoding unit configured to encode the map data that has been processed for shaping by the second shaping unit, and wherein the decoding device further comprising: a third decoding unit configured to decode the image data encoded by the third encoding unit and thereby generate decoded image data; and a fourth decoding unit configured to decode the map data encoded by the fourth encoding unit and thereby generate decoded map data. ### Claim 19 (depends on 18) The image processing system according to claim 18, wherein the decoding device further comprising: a second image processing unit configured to perform image processing to the decoded image data generated by the third decoding unit with reference to the decoded map data generated by the fourth decoding unit. ### Claim 20 (depends on 19) The image processing system according to claim 19, wherein the first image processing unit performs image processing to the decoded image data with reference to the decoded map data having first resolution, and wherein the second image processing unit performs image processing to the decoded image data with reference to the decoded map data having second resolution that is lower than the first resolution. ### Claim 21 An imaging apparatus that includes an image processing device that processes image data and map data with regard to depth distribution of a subject, wherein the image processing device comprising at least one processor or at least one circuit programmed to function as: a first decoding unit configured to decode the image data that has been encoded and thereby generate decoded image data; a second decoding unit configured to decode the map data that has been encoded and thereby generate decoded map data; a first shaping unit configured to perform a shaping process to the decoded map data with reference to the decoded image data having higher resolution than that of the decoded map data, wherein the shaping process includes an edge-preserved filtering process using coefficients generated based on the decoded image data; and a first image processing unit configured to perform image processing to the decoded image data with reference to the decoded map data that has been processed for shaping by the first shaping unit. ### Claim 22 (depends on 21) The imaging apparatus according to claim 21, comprising: an image sensor configured to image a subject through an imaging optical system, wherein the image sensor has a plurality of photoelectric conversion units that receive light that has passed through each of different pupil areas of the imaging optical system, and acquires information of depth distribution of a subject from a signal output by the plurality of photoelectric conversion units. ### Claim 23 An image processing method implemented by an image processing device that processes image data and map data of depth distribution of a subject, comprising: first decoding of decoding the image data that has been encoded and thereby generating decoded image data; second decoding of decoding the map data that has been encoded and thereby generating decoded map data; shaping by performing a shaping process to the decoded map data with reference to the decoded image data, wherein the shaping process includes an edge-preserved filtering process using coefficients generated based on the decoded image data; and image processing of performing image processing to the decoded image data with reference to the decoded map data that has been processed for shaping in the shaping. ### Claim 24 A non-transitory recording medium in which a program for causing a computer of the image processing device to implement each step of an image processing method is recorded, the image processing method comprising: first decoding of decoding the image data that has been encoded and thereby generating decoded image data; second decoding of decoding the map data that has been encoded and thereby generating decoded map data; shaping by performing a shaping process to the decoded map data with reference to the decoded image data, wherein the shaping process includes an edge-preserved filtering process using coefficients generated based on the decoded image data; and image processing of performing image processing to the decoded image data with reference to the decoded map data that has been processed for shaping in the shaping. ### Claim 25 (depends on 1) The image processing device according to claim 1, wherein the map data recorded by the second decoding unit is not performed by the shaping process before decoding.