# US 12,112,506 B2 · System for calibrating extrinsic parameters for a camera in an autonomous vehicle > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,506 B2 - Document ID: US12112506B2 - Status: Granted - Kind: B2 - Application number: 17/688,157 - Filed: 2022-03-07 - Published: 2023-09-07 - Granted: 2024-10-08 - Earliest claimed: 2022-03-07 - Assignees: GM Global Technology Operations LLC - Applicants: GM Global Technology Operations LLC - Inventors: Farui Peng; Hao Yu - Web page: https://patentgrep.com/patent/US12112506B2 ## CPC - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 7/292: Multi-camera tracking - G06T 7/73: using feature-based methods - G06T 2207/10016: Video; Image sequence - G06T 2207/30244: Camera pose - G06T 2207/30248: Vehicle exterior or interior - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A system for determining calibrated camera extrinsic parameters for an autonomous vehicle includes a camera mounted to the autonomous vehicle collecting image data including a plurality of image frames. The system also includes one or more automated driving controllers in electronic communication with the camera that executes instructions to determine a vehicle pose estimate based on position and movement of the autonomous vehicle by a localization algorithm. The one or more automated driving controllers determine the calibrated camera extrinsic parameters based on three dimensional coordinates for specific feature points of interests corresponding to two sequential image frames, the specific feature points of interests corresponding to the two sequential image frames, and the camera pose corresponding to the two sequential image frames. ## Claims ### Claim 1 A system for determining calibrated camera extrinsic parameters for an autonomous vehicle, the system comprising: a camera mounted to the autonomous vehicle collecting image data including a plurality of image frames; one or more automated driving controllers in electronic communication with the camera that executes instructions to: determine a vehicle pose estimate based on position and movement of the autonomous vehicle by a localization algorithm; determine a camera pose of the camera based on the vehicle pose estimate and initial camera extrinsic parameters; identify specific feature points of interest between two sequential image frames that are part of the plurality of image frames; execute a triangulation algorithm to determine three dimensional coordinates for the specific feature points of interests corresponding to the two sequential image frames; and execute a non-linear optimization algorithm to determine the calibrated camera extrinsic parameters based on the three dimensional coordinates for the specific feature points of interests corresponding to the two sequential image frames, the specific feature points of interests corresponding to the two sequential image frames, and the camera pose corresponding to the two sequential image frames. ### Claim 2 (depends on 1) The system of claim 1, wherein identifying the specific feature points of interest between the two sequential image frames further comprises: execute a feature detection algorithm to detect the specific feature points in the two sequential image frames; track feature points detected in an immediately preceding image frame with a current image frame; and select the specific feature points of interest between the two sequential image frames. ### Claim 3 (depends on 1) The system of claim 1, wherein the one or more automated driving controllers execute instructions to: determine a change in motion between the specific feature points of interest between the two sequential image frames. ### Claim 4 (depends on 3) The system of claim 3, wherein the one or more automated driving controllers execute instructions to: compare the change in motion between the specific feature points of interest between the two sequential image frames with a threshold amount of change in motion, wherein the threshold amount of change in motion change in motion is a predefined tuning parameter that selects two image frames that are part of a sliding window; and determine the change in motion between the specific feature points of interest between the two sequential image frames is equal to or greater than the threshold amount of change in motion. ### Claim 5 (depends on 1) The system of claim 1, wherein executing the non-linear optimization algorithm comprises: minimize reprojection error with respect to the calibrated camera extrinsic parameters and the three dimensional coordinates of the specific feature points of interests while solving a cost function. ### Claim 6 (depends on 5) The system of claim 5, wherein the one or more automated driving controllers execute instructions to: save results of solving the non-linear optimization algorithm to memory; clear a sliding window containing the sequential image frames; redetermine the calibrated camera extrinsic parameters; and after obtaining a threshold number of calibrated camera extrinsic parameters, perform one or more filtering and averaging operations. ### Claim 7 (depends on 1) The system of claim 1, wherein the camera pose is determined by transforming the initial camera extrinsic parameters from a vehicle body coordinate system to a world coordinate system. ### Claim 8 (depends on 1) The system of claim 1, wherein the initial camera extrinsic parameters are saved in memory of the one or more automated driving controllers. ### Claim 9 (depends on 1) The system of claim 1, wherein the initial camera extrinsic parameters represent an initial approximation of actual extrinsic parameters of the camera. ### Claim 10 (depends on 1) The system of claim 1, wherein the initial camera extrinsic parameters are expressed based on a vehicle body coordinate system and include an initial translation vector and an initial rotation matrix. ### Claim 11 (depends on 1) The system of claim 1, wherein the specific feature points of interests corresponding to the two sequential image frames are expressed based on a two-dimensional image plane coordinate system. ### Claim 12 (depends on 1) The system of claim 1, wherein the three dimensional coordinates of the specific feature points of interest are expressed based on a world coordinate system. ### Claim 13 (depends on 1) The system of claim 1, wherein the plurality of image frames are provided to the one or more automated driving controllers based on a sliding window buffer technique. ### Claim 14 (depends on 1) The system of claim 1, wherein the vehicle pose estimate includes a vehicle translation vector and a vehicle rotation matrix that are both expressed based on a world coordinate system. ### Claim 15 A method for determining calibrated camera extrinsic parameters for an autonomous vehicle, the method comprising: determining, by one or more automated driving controllers, a vehicle pose estimate based on position and movement of the autonomous vehicle by a localization algorithm; determining a camera pose of a camera mounted to the autonomous vehicle based on the vehicle pose estimate and initial camera extrinsic parameters, wherein the camera collects image data including a plurality of image frames; identifying specific feature points of interest between two sequential image frames that are part of the plurality of image frames; executing a triangulation algorithm to determine three dimensional coordinates for the specific feature points of interests corresponding to the two sequential image frames; and executing a non-linear optimization algorithm to determine the calibrated camera extrinsic parameters based on the three dimensional coordinates for the specific feature points of interests corresponding to the two sequential image frames, the specific feature points of interests corresponding to the two sequential image frames, and the camera pose corresponding to the two sequential image frames. ### Claim 16 (depends on 15) The method of claim 15, further comprising: executing a feature detection algorithm to detect the specific feature points in the two sequential image frames; tracking the feature points detected in an immediately preceding image frame with a current image frame; and selecting the specific feature points of interest between the two sequential image frames. ### Claim 17 (depends on 15) The method of claim 15, further comprising: determining a change in motion between the specific feature points of interest between the two sequential image frames. ### Claim 18 (depends on 17) The method of claim 17, further comprising: comparing the change in motion between the specific feature points of interest between the two sequential image frames with a threshold amount of change in motion, wherein the threshold amount of change in motion is a predefined tuning parameter that selects two image frames that are part of a sliding window; and determining the change in motion between the specific feature points of interest between the two sequential image frames is equal to or greater than the threshold amount of change in motion. ### Claim 19 (depends on 18) The method of claim 18, wherein executing the non-linear optimization algorithm comprises: minimizing reprojection error with respect to the calibrated camera extrinsic parameters and the three dimensional coordinates of the specific feature points of interests while solving a cost function. ### Claim 20 (depends on 19) The method of claim 19, further comprising: saving results of the non-linear optimization algorithm to memory; clearing a sliding window containing the sequential image frames; redetermine the calibrated camera extrinsic parameters; and after obtaining a threshold number of calibrated camera extrinsic parameters, performing one or more filtering and averaging operations.