# US 2021/0243351 A1 · Image processing device, image processing method, and program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0243351 A1 - Document ID: US20210243351A1 - Status: Published application - Kind: A1 - Application number: 17/052,657 - Filed: 2019-04-23 - Published: 2021-08-05 - Earliest claimed: 2018-05-15 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: MASATOSHI YOKOKAWA; HIDEYUKI ICHIHASHI; YIWEN ZHU; TOMOHIRO NISHI - Web page: https://patentgrep.com/patent/US20210243351A1 ## CPC - H04N 17/002: {for television cameras} - H04N 23/741: by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors - G06T 5/92: based on global image properties - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - H04N 13/25: using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor - H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images - H04N 23/667: Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes - H04N 23/69: Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming - H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture - H04N 23/71: Circuitry for evaluating the brightness variation - H04N 23/73: by influencing the exposure time - H04N 23/74: by influencing the scene brightness using illuminating means - H04N 23/95: Computational photography systems, e.g. light-field imaging systems - H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract To realize a device and a method capable of generating a high-quality image similar to that of a same viewing angle compound eye camera by using captured images of a tele/wide compound eye camera. A wide image capturing unit which captures an image of a wide viewing angle, a tele image capturing unit which captures an image of a narrow viewing angle, and an image processing unit which receives an input of the captured images of the respective imaging units and executes signal processing are included. In a case where a wide conversion lens is attached to the tele image capturing unit, the image processing unit executes any one of high-sensitivity image generation processing, HDR image generation processing, or high-resolution image generation processing based on the captured images of the wide image capturing unit and the tele image capturing unit according to a situation at the time of imaging with a wide conversion lens attachment detection unit which detects whether or not the wide conversion lens is attached. ## Claims ### Claim 1 An image processing device comprising: a wide image capturing unit that captures an image of a wide viewing angle; a tele image capturing unit that captures an image of a narrower viewing angle than the viewing angle of the wide image capturing unit; and an image processing unit that receives an input of captured images of the wide image capturing unit and the tele image capturing unit and executes signal processing, wherein the image processing unit includes a wide conversion lens attachment detection unit that detects whether or not a wide conversion lens capable of capturing an image of substantially the same viewing angle as the viewing angle of the wide image capturing unit is attached to the tele image capturing unit, and a signal processing unit that executes different signal processing according to a detection result of the wide conversion lens attachment detection unit. ### Claim 2 (depends on 1) The image processing device according to claim 1, wherein, in a case where the wide conversion lens is attached to the tele image capturing unit, the signal processing unit executes any one of high-sensitivity image generation processing, high dynamic range (HDR) image generation processing, or high-resolution image generation processing based on the captured images of the wide image capturing unit and the tele image capturing unit according to a situation at the time of image capturing. ### Claim 3 (depends on 1) The image processing device according to claim 1, wherein, in a case where the wide conversion lens is attached to the tele image capturing unit, the signal processing unit executes any one of high-sensitivity image generation processing, high dynamic range (HDR) image generation processing, or high-resolution image generation processing based on the captured images of the wide image capturing unit and the tele image capturing unit according to mode setting information of a user. ### Claim 4 (depends on 1) The image processing device according to claim 1, wherein the wide conversion lens attachment detection unit receives an input of the captured images of the wide image capturing unit and the tele image capturing unit, calculates similarity between the respective imaging units and determines whether or not the wide conversion lens is attached to the tele image capturing unit on a basis of the calculated similarity. ### Claim 5 (depends on 4) The image processing device according to claim 4, wherein the wide conversion lens attachment detection unit uses a luminance histogram or a sum of absolute differences (SAD) as an index value of the similarity. ### Claim 6 (depends on 1) The image processing device according to claim 1, wherein, in a case where the wide conversion lens is attached to the tele image capturing unit, the signal processing unit receives an input of illuminance information, backlight presence/absence information, and zoom setting information as imaging state information at the time of image capturing and executes any one of high-sensitivity image generation processing, high dynamic range (HDR) image generation processing, or high-resolution image generation processing based on the captured images of the wide image capturing unit and the tele image capturing unit according to input information. ### Claim 7 (depends on 1) The image processing device according to claim 1, wherein, in a case where the wide conversion lens is attached to the tele image capturing unit, and illuminance at the time of image capturing is lower than a predetermined threshold, the signal processing unit executes high-sensitivity image generation processing based on the captured images of the wide image capturing unit and the tele image capturing unit. ### Claim 8 (depends on 1) The image processing device according to claim 1, wherein, in a case where the wide conversion lens is attached to the tele image capturing unit, and illuminance at the time of image capturing is equal to or higher than a predetermined threshold, and it is determined that there is backlight, the signal processing unit executes HDR image generation processing based on the captured images of the wide image capturing unit and the tele image capturing unit. ### Claim 9 (depends on 1) The image processing device according to claim 1, wherein, in a case where the wide conversion lens is attached to the tele image capturing unit, and illuminance at the time of image capturing is equal to or higher than a predetermined threshold, there is no backlight, and zoom magnification is equal to or larger than a predetermined threshold, the signal processing unit executes high-resolution image generation processing based on the captured images of the wide image capturing unit and the tele image capturing unit. ### Claim 10 (depends on 1) The image processing device according to claim 1, wherein the signal processing unit executes subject distance calculation processing based on the captured images of the wide image capturing unit and the tele image capturing unit. ### Claim 11 (depends on 1) The image processing device according to claim 1, wherein, in a case where the wide conversion lens is attached to the tele image capturing unit, the signal processing unit selects optimal calibration data according to an attaching position of the wide conversion lens, and executes calibration processing based on the selected optimal calibration data to execute subject distance calculation processing based on the captured images of the wide image capturing unit and the tele image capturing unit. ### Claim 12 An image processing method executed in an image processing device, the method executing: a step of capturing an image by each of a wide image capturing unit that captures an image of a wide viewing angle and a tele image capturing unit that captures an image of a narrower viewing angle than the viewing angle of the wide image capturing unit; and an image processing step in which the image processing unit receives an input of captured images of the wide image capturing unit and the tele image capturing unit and executes signal processing, wherein the image processing step executes wide conversion lens attachment detection processing of detecting whether or not a wide conversion lens capable of capturing an image of substantially the same viewing angle as the viewing angle of the wide image capturing unit is attached to the tele image capturing unit, and different signal processing according to a result of the wide conversion lens attachment detection. ### Claim 13 A program that allows an image processing device to execute image processing, the program allowing the image processing device to execute: a step of capturing an image by each of a wide image capturing unit that captures an image of a wide viewing angle and a tele image capturing unit that captures an image of a narrower viewing angle than the viewing angle of the wide image capturing unit; and an image processing step in which an image processing unit receives an input of captured images of the wide image capturing unit and the tele image capturing unit and executes signal processing, wherein, at the image processing step, a wide conversion lens attachment detection processing of detecting whether or not a wide conversion lens capable of capturing an image of substantially the same viewing angle as the viewing angle of the wide image capturing unit is attached to the tele image capturing unit, and different signal processing according to a result of the wide conversion lens attachment detection are executed.