# US 2018/0025472 A1 · Target-Less Auto-Alignment of Image Sensors in a Multi-Camera System > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0025472 A1 - Document ID: US20180025472A1 - Status: Published application - Kind: A1 - Application number: 15/711,951 - Filed: 2017-09-21 - Published: 2018-01-25 - Earliest claimed: 2014-03-20 - Assignees: GoPro, Inc. - Applicants: GoPro, Inc. - Inventors: Yajie Sun; Timothy Macmillan; Elizabeth Torres Edwards; Sumit Chawla - Web page: https://patentgrep.com/patent/US20180025472A1 ## CPC - H04N 13/243: using three or more two-dimensional [2D] image sensors - G06F 18/22: Matching criteria, e.g. proximity measures - G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images - G06T 5/70: Denoising; Smoothing - G06T 7/13: Edge detection - G06T 7/337: {involving reference images or patches} - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G06T 7/85: {Stereo camera calibration} - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - H04N 13/246: Calibration of cameras - H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps - H04N 13/296: Synchronisation thereof; Control thereof - H04N 17/002: {for television cameras} - H04N 23/689: {Motion occurring during a rolling shutter mode} - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - H04N 25/48: Increasing resolution by shifting the sensor relative to the scene - H04N 25/76: Addressed sensors, e.g. MOS or CMOS sensors - H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A pair of cameras having an overlapping field of view is aligned based on images captured by image sensors of the pair of cameras. A pixel shift is identified between the images. Based on the identified pixel shift, a calibration is applied to one or both of the pair of cameras. To determine the pixel shift, the camera applies correlation methods including edge matching. Calibrating the pair of cameras may include adjusting a read window on an image sensor. The pixel shift can also be used to determine a time lag, which can be used to synchronize subsequent image captures. ## Claims ### Claim 1 A computer-implemented method for determining a pixel shift between an image pair captured by image sensors, the method comprising: accessing a first image and a second image of the image pair, the images comprising image data representative of an overlapping field of view between the image sensors; performing a smoothing operation on the first image and the second image to produce smoothed image data; determining edge magnitude data and edge phase data from the smoothed image data; identifying, by one or more processors, one or more edges in the image data based at least in part on the edge magnitude and the edge phase of the smoothed image data; determining a pixel shift between the image pair based at least in part on a distance between corresponding identified edges in the image pair. ### Claim 2 (depends on 1) The method of claim 1, wherein identifying the one or more edges in the image data comprises: identifying a first set of smoothed image data having determined edge magnitude greater than an edge magnitude threshold; identifying a second set of smoothed image data having determined edge phase within a threshold of a target edge phase value; and identifying the one or more edges from image data in both the first set of smoothed image data and the second set of smoothed image data. ### Claim 3 (depends on 2) The method of claim 2, wherein the identified one or more edges are substantially perpendicular to a rolling shutter direction of the image sensors, and the target edge phase value corresponds to the rolling shutter direction. ### Claim 4 (depends on 1) The method of claim 1, wherein determining a pixel shift between the image pair comprises: determining edge lengths for the identified one or more edges; and matching a first edge in the first image with a second edge in the second image based at least in part on the determined edge lengths. ### Claim 5 (depends on 1) The method of claim 1, wherein the image sensors have a rolling shutter direction, the method further comprising: before determining the edge magnitude data and edge phase data, compressing the first and second images perpendicular to the rolling shutter direction without compressing the first and second images along the rolling shutter direction. ### Claim 6 (depends on 1) The method of claim 1, wherein the first image and the second image are downsampled before the smoothing operation is performed. ### Claim 7 (depends on 1) The method of claim 1, further comprising cropping the first and second images by removing image data corresponding to outer borders of the first and second images prior to determining the edge magnitude data and the edge phase data. ### Claim 8 (depends on 1) The method of claim 1, further comprising calibrating the image sensors using the determined pixel shift. ### Claim 9 A non-transitory computer readable medium comprising instructions for determining a pixel shift between an image pair captured by image sensors, the instructions executable by a processor, the instructions comprising instructions for: accessing a first image and a second image of the image pair, the images comprising image data representative of an overlapping field of view between the image sensors; performing a smoothing operation on the first image and the second image to produce smoothed image data; determining edge magnitude data and edge phase data from the smoothed image data; identifying, by one or more processors, one or more edges in the image data based at least in part on the edge magnitude and the edge phase of the smoothed image data; determining a pixel shift between the image pair based at least in part on a distance between corresponding identified edges in the image pair. ### Claim 10 (depends on 9) The computer-readable medium of claim 9, wherein identifying the one or more edges in the image data comprises: identifying a first set of smoothed image data having determined edge magnitude greater than an edge magnitude threshold; identifying a second set of smoothed image data having determined edge phase within a threshold of a target edge phase value; and identifying the one or more edges from image data in both the first set of smoothed image data and the second set of smoothed image data. ### Claim 11 (depends on 10) The computer-readable medium of claim 10, wherein the identified one or more edges are substantially perpendicular to a rolling shutter direction of the image sensors, and the target edge phase value corresponds to the rolling shutter direction. ### Claim 12 (depends on 9) The computer-readable medium of claim 9, wherein determining a pixel shift between the image pair comprises: determining edge lengths for the identified one or more edges; and matching a first edge in the first image with a second edge in the second image based at least in part on the determined edge lengths. ### Claim 13 (depends on 9) The computer-readable medium of claim 9, wherein the image sensors have a rolling shutter direction, wherein the instructions further comprise instructions for: before determining the edge magnitude data and the edge phase data, compressing the first and second images perpendicular to the rolling shutter direction without compressing the first and second images along the rolling shutter direction. ### Claim 14 (depends on 9) The computer-readable medium of claim 9, wherein the instructions further comprise instructions for calibrating the image sensors using the determined pixel shift. ### Claim 15 A camera system comprising at least one processor, an image sensor, and a non-transitory computer readable medium comprising instructions for determining a pixel shift between an image pair captured by image sensors including the image sensor, the instructions executable by the at least one processor, the instructions comprising instructions for: accessing a first image and a second image of the image pair, the images comprising image data representative of an overlapping field of view between the image sensors; performing a smoothing operation on the first image and the second image to produce smoothed image data; determining edge magnitude data and edge phase data from the smoothed image data; identifying, by one or more processors, one or more edges in the image data based at least in part on the edge magnitude and the edge phase of the smoothed image data; determining a pixel shift between the image pair based at least in part on a distance between corresponding identified edges in the image pair. ### Claim 16 (depends on 15) The camera system of claim 15, wherein identifying the one or more edges in the image data comprises: identifying a first set of smoothed image data having determined edge magnitude greater than an edge magnitude threshold; identifying a second set of smoothed image data having determined edge phase within a threshold of a target edge phase value; and identifying the one or more edges from image data in both the first set of smoothed image data and the second set of smoothed image data. ### Claim 17 (depends on 16) The camera system of claim 16, wherein the identified one or more edges are substantially perpendicular to a rolling shutter direction of the image sensors, and the target edge phase value corresponds to the rolling shutter direction. ### Claim 18 (depends on 15) The camera system of claim 15, wherein determining a pixel shift between the image pair comprises: determining edge lengths for the identified one or more edges; and matching a first edge in the first image with a second edge in the second image based at least in part on the determined edge lengths. ### Claim 19 (depends on 15) The camera system of claim 15, wherein the image sensors have a rolling shutter direction, wherein the instructions further comprise instructions for: before determining the edge magnitude data and the edge phase data, compressing the first and second images perpendicular to the rolling shutter direction without compressing the first and second images along the rolling shutter direction. ### Claim 20 (depends on 15) The camera system of claim 15, wherein the instructions further comprise instructions for calibrating the image sensors using the determined pixel shift.