# US 2014/0192145 A1 · Estimation of panoramic camera orientation relative to a vehicle coordinate frame > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0192145 A1 - Document ID: US20140192145A1 - Status: Published application - Kind: A1 - Application number: 14/206,577 - Filed: 2014-03-12 - Published: 2014-07-10 - Earliest claimed: 2009-02-20 - Assignees: Google Inc. - Applicants: Google Inc. - Inventors: Dragomir Anguelov; Daniel Joseph Filip - Web page: https://patentgrep.com/patent/US20140192145A1 ## CPC - H04N 23/69: Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - H04N 5/2628: {Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation} - H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A system and method are presented for estimating the orientation of a panoramic camera mounted on a vehicle relative to the vehicle coordinate frame. An initial pose estimate of the vehicle is determined based on global positioning system data, inertial measurement unit data, and wheel odometry data of the vehicle. Image data from images captured by the camera is processed to obtain one or more tracks, each track including a sequence of matched feature points stemming from a same three-dimensional location. A correction parameter determined from the initial pose estimate and tracks can then be used to correct the orientations of the images captured by the camera. The correction parameter can be optimized by deriving a correction parameter for each of a multitude of distinct subsequences of one or more runs. Statistical analysis can be performed on the determined correction parameters to produce robust estimates. ## Claims ### Claim 1 A method for estimating orientation of cameras mounted on vehicles, comprising: determining, by one or more processors, an initial pose estimate of a vehicle having a camera mounted thereon, the initial pose estimate being a position and orientation of the vehicle relative to the world; processing, by the one or more processors, image data from images captured by the camera to obtain one or more tracks, each track including a sequence of matched feature points stemming from a single three-dimensional location; and determining, by the one or more processors, from the initial pose estimate and the tracks, a correction parameter to correct orientations of the images captured by the camera, wherein determining the correction parameter includes evaluating a set of vehicle poses, three-dimensional locations from the one or more tracks, and an orientation of the camera relative to one or more sensors of the vehicle. ### Claim 2 (depends on 1) The method of claim 1, wherein determining the correction parameter is done such that a reprojection error is minimized. ### Claim 3 (depends on 1) The method of claim 1, wherein the correction parameter is a camera rotation correction parameter that is determined for a particular portion of an uninterrupted data collection interval. ### Claim 4 (depends on 1) The method of claim 1, wherein the set of vehicle poses is obtained from location data, inertial measurement data, and wheel odometry data of the vehicle. ### Claim 5 (depends on 1) The method of claim 1, further comprising correcting orientations of the images by applying the correction parameter to the image data. ### Claim 6 (depends on 5) The method of claim 5, wherein the determining the correction parameter includes determining: an optimized pose of the vehicle; a location of points of the tracks in three dimensions; and a camera to vehicle pose rotation. ### Claim 7 (depends on 1) The method of claim 1, wherein: the determining includes determining the correction parameter for each of a multitude of subsequences of a selected run; and performing statistical analysis on the determined correction parameters for different subsequences to determine an optimized correction parameter. ### Claim 8 (depends on 7) The method of claim 7, wherein performing the statistical analysis includes determining a median value of the determined correction parameters to use as the optimized correction parameter. ### Claim 9 (depends on 8) The method of claim 8, wherein determining the median value omits determined correction parameters for subsequences of the selected run in which acceleration of the vehicle is above a predetermined value or outside of a predetermined range. ### Claim 10 (depends on 8) The method of claim 8, wherein determining the median value omits determined correction parameters for subsequences of the selected run in which the vehicle is moving outside of a predetermined velocity range. ### Claim 11 (depends on 7) The method of claim 7, wherein determining the correction parameter for each of a multitude of subsequences of a selected run includes performing a cost function to chooses subsequences of the selected run that conform to a predetermined rule. ### Claim 12 (depends on 1) The method of claim 1, wherein: the determining includes determining the correction parameter for each of a multitude of subsequences of a multitude of runs; and performing statistical analysis on the determined correction parameters across the multitude of runs to determine an optimized correction parameter. ### Claim 13 (depends on 12) The method of claim 12, wherein the performing statistical analysis includes: for each run of the multitude of runs, determining a first median of the correction parameters for each of a closest set of runs backward in time and determining a second median of the correction parameters for each of a closest set of runs forward in time; and selecting either the first median or the second median as the optimized correction parameter based on which of the first or second median is closest to the correction parameter determined for that run. ### Claim 14 A system, comprising: one or more processors configured to: determine an initial pose estimate of a vehicle having a camera mounted thereon, the initial pose estimate being a position and orientation of the vehicle relative to the world; process image data from images captured by the camera to obtain one or more tracks, each track including a sequence of matched feature points stemming from a single three-dimensional location; and determine, from the initial pose estimate and the tracks, a correction parameter to correct orientations of the images captured by the camera, wherein determining the correction parameter includes evaluating a set of vehicle poses, three-dimensional locations from the one or more tracks, and an orientation of the camera relative to one or more sensors of the vehicle. ### Claim 15 (depends on 15) The system of claim 15, wherein the one or more processors are further configured to correct orientations of the images by applying the correction parameter to the image data. ### Claim 16 (depends on 15) The system of claim 15, wherein the one or more processors are configured to determine the correction parameter by determining: an optimized pose of the vehicle; a location of points of the tracks in three dimensions; and a camera to vehicle pose rotation. ### Claim 17 (depends on 15) The system of claim 15, wherein: the one or more processors are configured to determine the correction parameter for each of a multitude of subsequences of a selected run; and the one or more processors are further configured to perform statistical analysis on the determined correction parameters for different subsequences to determine an optimized correction parameter. ### Claim 18 (depends on 15) The system of claim 15, wherein: the one or more processors are configured to determine the correction parameter for each of a multitude of subsequences of a multitude of runs; and the one or more processors are further configured to perform statistical analysis on the determined correction parameters across the multitude of runs to determine an optimized correction parameter. ### Claim 19 (depends on 18) The system of claim 18, wherein performing the statistical analysis by the one or more processors includes: for each run of the multitude of runs, determining a first median of the correction parameters for each of a closest set of runs backward in time and determining a second median of the correction parameters for each of a closest set of runs forward in time; and selecting either the first median or the second median as the optimized correction parameter based on which of the first or second median is closest to the correction parameter determined for that run. ### Claim 20 A non-transitory computer-readable storage medium on which computer readable instructions of a program are stored, the instructions, when executed by one or more processors, cause the one or more processors to perform a method for estimating orientation of cameras mounted on vehicles, the method comprising: determining an initial pose estimate of a vehicle having a camera mounted thereon, the initial pose estimate being a position and orientation of the vehicle relative to the world; processing image data from images captured by the camera to obtain one or more tracks, each track including a sequence of matched feature points stemming from a single three-dimensional location; and determining from the initial pose estimate and the tracks, a correction parameter to correct orientations of the images captured by the camera, wherein determining the correction parameter includes evaluating a set of vehicle poses, three-dimensional locations from the one or more tracks, and an orientation of the camera relative to one or more sensors of the vehicle.