# US 10,375,292 B2 · Image pickup apparatus and method for generating image having depth information > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,375,292 B2 - Document ID: US10375292B2 - Status: Granted - Kind: B2 - Application number: 14/626,289 - Filed: 2015-02-19 - Published: 2015-09-17 - Granted: 2019-08-06 - Earliest claimed: 2014-03-13 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Kyong-tae Park - Web page: https://patentgrep.com/patent/US10375292B2 ## CPC - H04N 23/676: {Bracketing for image capture at varying focusing conditions} - G02B 3/0006: {Arrays (G02B3/02, G02B5/188 take precedence)} - G02B 7/08: adapted to co-operate with a remote control mechanism - G02B 7/38: measured at different points on the optical axis {, e.g. focussing on two or more planes and comparing image data} - G02B 27/0075: {with means for altering, e.g. increasing, the depth of field or depth of focus} - H04N 13/207: using a single two-dimensional [2D] image sensor - H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps - H04N 23/959: by adjusting depth of field during image capture, e.g. maximising or setting range based on scene characteristics - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS ## Abstract Provided are an image pickup apparatus and method capable of improving a resolution of an image having depth information. The image pickup apparatus may include: a main lens configured to refract incident light; an image sensor comprising a plurality of two-dimensional (2D)-arranged pixels configured to output an image signal according to the incident light; a micro lens array between the main lens and the image sensor and comprising a plurality of 2D-arranged micro lenses; and a controller configured to receive the image signal from the image sensor and to generate an image according to the received image signal, wherein the controller is configured to obtain a plurality of images having different depths of field by changing a distance between the main lens and the image sensor and to obtain at least one depth map from the at least one of the obtained plurality of images. ## Claims ### Claim 1 An image pickup apparatus comprising: a main lens configured to refract incident light; an image sensor comprising a plurality of two-dimensional (2D)-arranged pixels configured to sense the incident light and to output an image signal according to the sensed incident light; a micro lens array between the main lens and the image sensor and comprising a plurality of 2D-arranged micro lenses; and a controller configured to receive the image signal from the image sensor and to generate an image according to the received image signal, wherein the controller is configured to capture, with the image sensor, a plurality of images having different depths of field by changing a distance between the main lens and the image sensor and to obtain at least one depth map, respectively corresponding to at least one of the captured plurality of images, from the at least one of the captured plurality of images, wherein the controller is configured to capture, with the image sensor, a first image by initially setting the distance between the main lens and the image sensor to focus on a predetermined distance, and to obtain a depth map from the captured first image; wherein, in response to an object being selected using the depth map, the controller is configured to capture, with the image sensor, a second image in which the selected object is focused by adjusting, using the obtained depth map, the distance between the main lens and the image sensor to focus on the selected object, and wherein the second image is captured with the image sensor after the depth map is obtained from the captured first image. ### Claim 2 (depends on 1) The image pickup apparatus of claim 1, wherein the predetermined distance is a hyperfocal distance. ### Claim 3 (depends on 1) The image pickup apparatus of claim 1, wherein the object is selected by a user input. ### Claim 4 (depends on 1) The image pickup apparatus of claim 1, wherein each of the plurality of 2D-arranged micro lenses respectively corresponds to at least two pixels in the image sensor. ### Claim 5 (depends on 4) The image pickup apparatus of claim 4, wherein the controller is configured to obtain a depth map by using outputs of at least two pixels corresponding to a same micro lens, among the plurality of 2D-arranged micro lenses, and to generate an image by combining the outputs of the at least two pixels corresponding to the same micro lens. ### Claim 6 (depends on 1) The image pickup apparatus of claim 1, wherein the controller is configured to change the distance between the main lens and the image sensor according to a depth of focus (DOF) unit. ### Claim 7 (depends on 6) The image pickup apparatus of claim 6, wherein the controller is configured to respectively capture in-focus images for all objects located between an infinite distance and a closest focusing distance by, each time the distance between the main lens and the image sensor is changed according to the DOF unit, capturing an image through the image sensor and obtaining a depth map. ### Claim 8 (depends on 7) The image pickup apparatus of claim 7, wherein the controller is configured to determine an object region where a depth value is a minimum in each depth map generated each time the distance between the main lens and the image sensor is changed according to the DOF unit. ### Claim 9 (depends on 8) The image pickup apparatus of claim 8, wherein, in response to an object region being selected from the plurality of images, the controller is configured to select a depth map of which a depth value in the selected object region is a minimum and to output an image corresponding to the selected depth map. ### Claim 10 (depends on 6) The image pickup apparatus of claim 6, wherein the controller is configured to change the distance between the main lens and the image sensor according to the DOF unit between Dh and Dc, where the distance between the main lens and the image sensor when focusing on a hyperfocal distance is Dh, and the distance between the main lens and the image sensor when focusing on a closest focusing distance is Dc. ### Claim 11 (depends on 10) The image pickup apparatus of claim 10, wherein the controller is configured to sequentially change the distance between the main lens and the image sensor according to the DOF unit until the distance between the main lens and the image sensor becomes Dc after initially setting the distance between the main lens and the image sensor as Dh. ### Claim 12 (depends on 10) The image pickup apparatus of claim 10, wherein the controller is configured to capture an image and obtain a depth map by initially setting the distance between the main lens and the image sensor as Dh and to adjust the distance between the main lens and the image sensor according to the DOF unit by analyzing the depth map so that an image is captured only for a depth of field where an object exists, and wherein the controller is configured to refrain from capturing images corresponding to depths of focus where an object does not exist. ### Claim 13 (depends on 6) The image pickup apparatus of claim 6, wherein the DOF is determined as 2×(an aperture ratio of the main lens)×(a circle of confusion (CoC)), and a size of the CoC may be equal to one or two pitches of a micro lens among the plurality of 2D-arranged micro lenses. ### Claim 14 (depends on 6) The image pickup apparatus of claim 6, wherein the controller is configured to identify a background and candidate objects of interest in the first image based on the depth map of the first image picked up by focusing at a hyperfocal distance, to select objects of interest according to a predetermined condition from among the identified candidate objects of interest, and to perform photographing for a depth of field range, wherein each of the selected objects of interest exists based on a depth value of each of the selected objects of interest, and wherein the controller is configured to refrain from performing photographing for a depth of field range wherein the selected objects of interest do not exist. ### Claim 15 (depends on 14) The image pickup apparatus of claim 14, wherein the controller is configured to adjust a number of selected objects of interest according to a remaining capacity of each of a battery and a memory. ### Claim 16 An image pickup method of an image pickup apparatus comprising a main lens, an image sensor, and a micro lens array between the main lens and the image sensor and comprising a plurality of two-dimensional (2D)-arranged micro lenses, the image pickup method comprising: capturing, with the image sensor, a plurality of images having different depths of field by changing a distance between the main lens and the image sensor; and obtaining at least one depth map, respectively corresponding to at least one of the captured plurality of images, from the at least one of the captured plurality of images, wherein the obtaining the at least one depth map comprises obtaining a depth map from a first image; wherein the capturing the plurality of images comprises: capturing, with the image sensor, the first image by initially setting the distance between the main lens and the image sensor to focus on a predetermined distance, and in response to an object being selected using the depth map, capturing, with the image sensor, a second image in which the selected object is focused by adjusting, using the obtained depth map, the distance between the main lens and the image sensor to focus on the selected object; and wherein the second image is captured with the image sensor after the depth map is obtained from the captured first image. ### Claim 17 (depends on 16) The image pickup method of claim 16, wherein the predetermined distance is a hyperfocal distance. ### Claim 18 (depends on 16) The image pickup method of claim 16, wherein the object is selected by a user input. ### Claim 19 (depends on 16) The image pickup method of claim 16, wherein each of the plurality of 2D-arranged micro lenses respectively corresponds to at least two pixels in the image sensor. ### Claim 20 (depends on 19) The image pickup method of claim 19, further comprising: obtaining a depth map by using outputs of at least two pixels corresponding to a same micro lens, among the plurality of 2D-arranged micro lenses; and generating an image by combining the outputs of the at least two pixels corresponding to the same micro lens. ### Claim 21 (depends on 16) The image pickup method of claim 16, wherein the distance between the main lens and the image sensor is changed according to a depth of focus (DOF) as a step unit. ### Claim 22 (depends on 21) The image pickup method of claim 21, wherein the capturing the plurality of images comprises capturing in-focus images for all objects located from an infinite distance to a closest focusing distance by, each time the distance between the main lens and the image sensor is changed according to the DOF unit, capturing an image and obtaining a depth map. ### Claim 23 (depends on 22) The image pickup method of claim 22, further comprising determining an object region where a depth value is a minimum in each depth map generated each time the distance between the main lens and the image sensor is changed according to the DOF unit. ### Claim 24 (depends on 23) The image pickup method of claim 23, further comprising: selecting an object region from the plurality of images; selecting a depth map of which a depth value in the selected object region is a minimum; and outputting an image corresponding to the selected depth map. ### Claim 25 (depends on 21) The image pickup method of claim 21, wherein the distance between the main lens and the image sensor is changed according to the DOF unit between Dh and Dc, where the distance between the main lens and the image sensor when focusing on a hyperfocal distance is Dh, and the distance between the main lens and the image sensor when focusing on a closest focusing distance is Dc. ### Claim 26 (depends on 25) The image pickup method of claim 25, wherein the capturing the plurality of images comprises: initially setting the distance between the main lens and the image sensor as Dh; and sequentially changing the distance between the main lens and the image sensor according to the DOF unit until the distance between the main lens and the image sensor becomes Dc. ### Claim 27 (depends on 25) The image pickup method of claim 25, wherein the capturing the plurality of images comprises: initially setting the distance between the main lens and the image sensor as Dh; and changing the distance between the main lens and the image sensor according to the DOF unit by analyzing the depth map obtained in the initial setting of the distance so that an image is captured only for a depth of field where an object exists. ### Claim 28 (depends on 21) The image pickup method of claim 21, wherein the DOF is determined as 2×(an aperture ratio of the main lens)×(a circle of confusion (CoC)), and a size of the CoC may be equal to one or two pitches of a micro lens among the plurality of 2D-arranged micro lenses. ### Claim 29 (depends on 21) The image pickup method of claim 21, further comprising: picking up an initial image by focusing at a hyperfocal distance; identifying a background and candidate objects of interest in the initial image through a depth map of the picked up initial image; selecting objects of interest according to a predetermined condition from among the identified candidate objects of interest; and performing photographing for a depth of a field range wherein each of the selected objects of interest exists based on a depth value of each of the selected objects of interest. ### Claim 30 (depends on 29) The image pickup method of claim 29, further comprising: checking a remaining capacity of each of a battery and a memory; and adjusting a number of selected objects of interest according to the remaining capacity of each of the battery and the memory. ### Claim 31 A refocusing method of an image pickup apparatus comprising a main lens and an image sensor, the refocusing method comprising: capturing, with the image sensor, a plurality of images having different depths of field by changing a distance between the main lens and the image sensor; obtaining a depth map from each of the respective captured plurality of images; determining, for each respective depth map, an object region where a depth value in the depth map is a minimum; after determining the object region for each respective depth map, receiving a selection from a user of a selected object region from the captured plurality of images; selecting a depth map of which a depth value in the selected object region is a minimum; and outputting an image corresponding to the selected depth map. ### Claim 32 An image pickup apparatus comprising: a main lens configured to refract incident light; an image sensor comprising a plurality of two-dimensional (2D)-arranged pixels configured to sense the incident light and to output an image signal according to the sensed incident light; a micro lens array between the main lens and the image sensor and comprising a plurality of 2D-arranged micro lenses; and a controller configured to receive the image signal from the image sensor and to generate an image according to the received image signal, wherein the controller is configured to capture, with the image sensor, a first image, having a first depth of field, by initially setting a distance between the main lens and the image sensor to focus on a predetermined distance, and to obtain a depth map from the captured first image, wherein in response to an object being selected using the depth map, the controller is configured to capture, with the image sensor, a second image, having a second depth of field different from the first depth of field, in which the selected object is focused by adjusting, using the obtained depth map, the distance between the main lens and the image sensor to focus on the selected object, and wherein the second image is captured with the image sensor after the depth map is obtained from the captured first image. ### Claim 33 A refocusing method of an image pickup apparatus comprising a main lens and an image sensor, the refocusing method comprising: capturing, with the image sensor, a first image having a first depth of field with the image sensor; obtaining a depth map from the captured first image; and in response to an object being selected using the depth map, capturing, with the image sensor, a second image, having a second depth of field different from the first depth of field with the image sensor, in which the selected object is focused by adjusting, using the obtained depth map, a distance between the main lens and the image sensor to focus on the selected object, wherein the second image is captured with the image sensor after the depth map is obtained from the captured first image. ### Claim 34 (depends on 1) The image pickup apparatus of claim 1, wherein the controller is configured to select the object according to a history of images previously captured by a user. ### Claim 35 (depends on 16) The image pickup method of claim 16, wherein the object is selected according to a history of images previously captured by a user. ### Claim 36 (depends on 1) The image pickup apparatus of claim 1, wherein the second image is captured with the image sensor after adjusting the distance between the main lens and the image sensor. ### Claim 37 (depends on 16) The image pickup method of claim 16, wherein the second image is captured with the image sensor after adjusting the distance between the main lens and the image sensor.