# US 2021/0174479 A1 · Apparatus and method for dynamic multi-camera rectification using depth camera > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0174479 A1 - Document ID: US20210174479A1 - Status: Published application - Kind: A1 - Application number: 16/703,712 - Filed: 2019-12-04 - Published: 2021-06-10 - Earliest claimed: 2019-12-04 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Yingmao Li; Chenchi Luo; George Q. Chen; Youngjun Yoo - Web page: https://patentgrep.com/patent/US20210174479A1 ## CPC - G06T 5/80: Geometric correction - G06T 3/18: Image warping, e.g. rearranging pixels individually - G06T 5/40: using histogram techniques - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 7/55: from multiple images - G06T 7/593: from stereo images - G06T 7/74: {involving reference images or patches} - G06T 2207/10012: Stereo images - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform - G06T 2207/30196: Human being; Person - G06T 2207/30244: Camera pose - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method includes obtaining, using first and second image sensors of an electronic device, first and second images, respectively, of a scene. The method also includes obtaining, using an image depth sensor of the electronic device, a third image and a first depth map of the scene, the first depth map having a resolution lower than a resolution of the first and second images. The method further includes undistorting the first and second images using the third image and the first depth map. The method also includes rectifying the first and second images using the third image and the first depth map. The method further includes generating a disparity map using the first and second images that have been undistorted and rectified. In addition, the method includes generating a second depth map using the disparity map and the first depth map, where the second depth map has a resolution that is higher than the resolution of the first depth map. ## Claims ### Claim 1 A method comprising: obtaining, using first and second image sensors of an electronic device, first and second images, respectively, of a scene; obtaining, using an image depth sensor of the electronic device, a third image and a first depth map of the scene, the first depth map having a resolution lower than a resolution of the first and second images; undistorting the first and second images using the third image and the first depth map; rectifying the first and second images using the third image and the first depth map; generating a disparity map using the first and second images that have been undistorted and rectified; and generating a second depth map using the disparity map and the first depth map, the second depth map having a resolution that is higher than the resolution of the first depth map. ### Claim 2 (depends on 1) The method of claim 1, further comprising: generating a final image of the scene using the second depth map and at least one of the first and second images. ### Claim 3 (depends on 1) The method of claim 1, wherein undistorting the first and second images comprises: matching multiple feature points in common between the first and third images and between the second and third images; estimating a first pose between the first image sensor and the image depth sensor and a second pose between the second image sensor and the image depth sensor using the matched feature points; projecting the matched feature points onto an image plane using the first depth map; generating an undistortion map for each of the first and second images using the projected feature points; and undistorting each of the first and second images using the corresponding undistortion map. ### Claim 4 (depends on 1) The method of claim 1, wherein rectifying the first and second images comprises: matching multiple feature points in common between the first and second images; estimating a pose between the first image sensor and the second image sensor using the matched feature points; and warping at least one of the first and second images. ### Claim 5 (depends on 1) The method of claim 1, wherein generating the second depth map using the disparity map comprises: generating a depth histogram using the first depth map; generating a disparity histogram using the disparity map; generating a mapping function based on a comparison of the depth histogram and the disparity histogram; and generating the second depth map using the mapping function. ### Claim 6 (depends on 1) The method of claim 1, wherein the resolution of the second depth map is substantially equal to the resolution of the first and second images. ### Claim 7 (depends on 1) The method of claim 1, wherein the image depth sensor comprises a time-of-flight (ToF) camera. ### Claim 8 An electronic device comprising: first and second image sensors configured to generate first and second images of a scene, respectively; an image depth sensor configured to generate a third image and a first depth map of the scene, the first depth map having a resolution lower than a resolution of the first and second images; and at least one processing device configured to: undistort the first and second images using the third image and the first depth map; rectify the first and second images using the third image and the first depth map; generate a disparity map using the first and second images which have been undistorted and rectified; and generate a second depth map using the disparity map and the first depth map, the second depth map having a resolution that is higher than the resolution of the first depth map. ### Claim 9 (depends on 8) The electronic device of claim 8, wherein the at least one processing device is further configured to generate a final image of the scene using the second depth map and at least one of the first and second images. ### Claim 10 (depends on 8) The electronic device of claim 8, wherein, to undistort the first and second images, the at least one processing device is configured to: match multiple feature points in common between the first and third images and between the second and third images; estimate a first pose between the first image sensor and the image depth sensor and a second pose between the second image sensor and the image depth sensor using the matched feature points; project the matched feature points onto an image plane using the first depth map; generate an undistortion map for each of the first and second images using the projected feature points; and undistort each of the first and second images using the corresponding undistortion map. ### Claim 11 (depends on 8) The electronic device of claim 8, wherein, to rectify the first and second images, the at least one processing device is configured to: match multiple feature points in common between the first and second images; estimate a pose between the first image sensor and the second image sensor using the matched feature points; and warp at least one of the first and second images. ### Claim 12 (depends on 8) The electronic device of claim 8, wherein, to generate the second depth map using the disparity map, the at least one processing device is configured to: generate a depth histogram using the first depth map; generate a disparity histogram using the disparity map; generate a mapping function based on a comparison of the depth histogram and the disparity histogram; and generate the second depth map using the mapping function. ### Claim 13 (depends on 8) The electronic device of claim 8, wherein the resolution of the second depth map is substantially equal to the resolution of the first and second images. ### Claim 14 (depends on 8) The electronic device of claim 8, wherein the image depth sensor comprises a time-of-flight (ToF) camera. ### Claim 15 A non-transitory machine-readable medium containing instructions that when executed cause at least one processor of an electronic device to: obtain first and second images of a scene that are captured using first and second image sensors, respectively, of the electronic device; obtain a third image and a first depth map of the scene that are captured using an image depth sensor of the electronic device, the first depth map having a resolution lower than a resolution of the first and second images; undistort the first and second images using the third image and the first depth map; rectify the first and second images using the third image and the first depth map; generate a disparity map using the first and second images which have been undistorted and rectified; and generate a second depth map using the disparity map and the first depth map, the second depth map having a resolution that is higher than the resolution of the first depth map. ### Claim 16 (depends on 15) The non-transitory machine-readable medium of claim 15, wherein a final image of the scene is generated using the second depth map and at least one of the first and second images. ### Claim 17 (depends on 15) The non-transitory machine-readable medium of claim 15, wherein the first and second images are undistorted by: matching multiple feature points in common between the first and third images and between the second and third images; estimating a first pose between the first image sensor and the image depth sensor and a second pose between the second image sensor and the image depth sensor using the matched feature points; projecting the matched feature points onto an image plane using the first depth map; generating an undistortion map for each of the first and second images using the projected feature points; and undistorting each of the first and second images using the corresponding undistortion map. ### Claim 18 (depends on 15) The non-transitory machine-readable medium of claim 15, wherein the first and second images are rectified by: matching multiple feature points in common between the first and second images; estimating a pose between the first image sensor and the second image sensor using the matched feature points; and warping at least one of the first and second images. ### Claim 19 (depends on 15) The non-transitory machine-readable medium of claim 15, wherein the second depth map is generated using the disparity map by: generating a depth histogram using the first depth map; generating a disparity histogram using the disparity map; generating a mapping function based on a comparison of the depth histogram and the disparity histogram; and generating the second depth map using the mapping function. ### Claim 20 (depends on 15) The non-transitory machine-readable medium of claim 15, wherein the resolution of the second depth map is substantially equal to the resolution of the first and second images.