# US 12,112,508 B2 · Calibrating system for colorizing point-clouds > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,508 B2 - Document ID: US12112508B2 - Status: Granted - Kind: B2 - Application number: 18/449,934 - Filed: 2023-08-15 - Published: 2023-11-30 - Granted: 2024-10-08 - Earliest claimed: 2021-04-27 - Assignees: Faro Technologies, Inc. - Applicants: FARO Technologies, Inc. - Inventors: Jafar Amiri Parian; Martin Ossig; Hani Kaabi - Web page: https://patentgrep.com/patent/US12112508B2 ## CPC - G06T 7/90: Determination of colour characteristics - G01S 7/497: Means for monitoring or calibrating - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G01S 17/89: for mapping or imaging - G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06V 10/757: {Matching configurations of points or features} - H04N 17/002: {for television cameras} - H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20164: Salient point detection; Corner detection - G06T 2219/2012: Colour editing, changing, or manipulating; Use of colour codes - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A system includes a three-dimensional (3D) scanner that captures a 3D point cloud corresponding to one or more objects in a surrounding environment. The system further includes a camera that captures a control image by capturing a plurality of images of the surrounding environment, and an auxiliary camera configured to capture an ultrawide-angle image of the surrounding environment. One or more processors of the system colorize the 3D point cloud using the ultrawide-angle image by mapping the ultrawide-angle image to the 3D point cloud. The system performs a limited system calibration before colorizing each 3D point cloud, and a periodic full system calibration before/after a plurality of 3D point clouds are colorized. ## Claims ### Claim 1 A system comprising: a three-dimensional (3D) scanner that captures a 3D point cloud that comprises a plurality of 3D coordinates corresponding to one or more objects scanned in a surrounding environment; a camera that captures a control image by capturing a plurality of images of the surrounding environment, wherein images from the plurality of images are stitched to form the control image; an auxiliary camera configured to capture an ultrawide-angle image of the surrounding environment; and one or more processors configured to: colorize the 3D point cloud using the ultrawide-angle image by mapping the ultrawide-angle image to the 3D point cloud; perform a full system calibration to calibrate the 3D scanner, the camera, and the auxiliary camera using the 3D point cloud, the control image, and a calibration image, which is another ultrawide-angle image from the auxiliary camera, wherein performing the full system calibration comprises: extracting a first plurality of features from the control image using a feature-extraction algorithm; extracting a second plurality of features from the calibration image using the feature-extraction algorithm; determining a set of matching features from the first plurality of features and the second plurality of features by using a feature-matching algorithm; building a control point test-field by using the set of matching features and the 3D point cloud; and determining a set of system calibration parameters on-the-fly using bundle adjustment and camera self-calibration; and perform a limited system calibration before colorizing the 3D point cloud, wherein the limited system calibration comprises updating a subset of system calibration parameters from the set of system calibration parameters. ### Claim 2 (depends on 1) The system of claim 1, wherein the limited system calibration comprises: extracting a third plurality of features from the control image using the feature-extraction algorithm; determining another set of matching features to the extracted features at the first plurality of features by using a hybrid feature matching algorithm; building another control point test-field by using the another set of matching features and the 3D point cloud; and determining updated values for at least one of the system calibration parameters. ### Claim 3 (depends on 2) The system of claim 2, wherein the another set of matching features is determined based at least in part from the set of matching features. ### Claim 4 (depends on 1) The system of claim 1, wherein extracting the second plurality of features from the calibration image comprises: transforming the calibration image to a spherical image; and extracting the second plurality of features from the spherical image. ### Claim 5 (depends on 1) The system of claim 1, wherein the full system calibration is performed after the 3D point cloud is colorized. ### Claim 6 (depends on 1) The system of claim 1, wherein the auxiliary camera includes two lenses at predetermined offsets relative to each other. ### Claim 7 (depends on 1) The system of claim 1, wherein the set of system calibration parameters includes a first plurality of camera calibration parameters, a second plurality of dual-camera calibration parameters, and a third plurality of multi-device orientation parameters. ### Claim 8 (depends on 1) The system of claim 1, wherein the auxiliary camera is mounted on the 3D scanner at a predetermined position relative to the 3D scanner. ### Claim 9 (depends on 1) The system of claim 1, wherein the control point test-field is dynamically built by selecting a subset of points of the set of matching features and the 3D point cloud. ### Claim 10 (depends on 1) The system of claim 1, wherein each of the parameters of the set of system calibration parameters is determined simultaneously. ### Claim 11 (depends on 1) The system of claim 1, wherein each of the parameters of the set of system calibration parameters is determined from a set of images captured by the 3D scanner, the camera, or the auxiliary camera. ### Claim 12 (depends on 11) The system of claim 11, wherein the set of images comprises the plurality of 3D coordinates corresponding to one or more objects scanned in a surrounding environment, the plurality of images of the surrounding environment, and the ultrawide-angle image of the surrounding environment. ### Claim 13 A method comprising: periodically performing a full system calibration of a measurement device that comprises a 3D scanner, a camera, and an auxiliary camera, wherein the full system calibration is performed using a first 3D point cloud, a control image, and a calibration image, wherein performing the full system calibration comprises: extracting a first plurality of features from the control image using a feature-extraction algorithm, wherein the control image is captured by the camera by capturing and stitching a plurality of images; extracting a second plurality of features from the calibration image using the feature-extraction algorithm, the calibration image being an ultrawide-angle image from the auxiliary camera; determining a set of matching features from the first plurality of features and the second plurality of features by using a feature-matching algorithm; building a control point test-field by using the set of matching features and the first 3D point cloud; and determining a set of system calibration parameters on-the-fly using bundle adjustment and camera self-calibration; and performing a limited system calibration of the measurement device in response to capturing a second 3D point cloud that is to be colorized, wherein the limited system calibration comprises updating a subset of system calibration parameters from the set of system calibration parameters using the second 3D point cloud and a second ultrawide-angle image from the auxiliary camera. ### Claim 14 (depends on 13) The method of claim 13, wherein the limited system calibration comprises: extracting a third plurality of features from the control image using the feature-extraction algorithm; determining another set of matching features to the extracted features at the first plurality of features by using a hybrid feature matching algorithm; building another control point test-field by using the another set of matching features and the second 3D point cloud; and determining updated values for at least one of the system calibration parameters. ### Claim 15 (depends on 13) The method of claim 13, wherein extracting the second plurality of features from the calibration image comprises: transforming the ultrawide-angle image to a spherical image; and extracting the second plurality of features from the spherical image. ### Claim 16 (depends on 13) The method of claim 13, wherein two successive full system calibrations are performed after a predetermined interval. ### Claim 17 (depends on 13) The method of claim 13, wherein the auxiliary camera includes two lenses at predetermined offsets relative to each other. ### Claim 18 (depends on 13) The method of claim 13, wherein successive full system calibrations are performed after a predetermined number of 3D point clouds are colorized. ### Claim 19 A computer program product comprising one or more memory devices with computer executable instructions stored thereon, the computer executable instructions when executed by one or more processors cause the one or more processors to perform a method comprising: periodically performing a full system calibration of a measurement device that comprises a 3D scanner, a camera, and an auxiliary camera, wherein the full system calibration is performed using a first 3D point cloud, a control image, and a calibration image, wherein performing the full system calibration comprises: extracting a first plurality of features from the control image using a feature-extraction algorithm, wherein the control image is captured by the camera by capturing and stitching a plurality of images; extracting a second plurality of features from the calibration image using the feature-extraction algorithm, the calibration image being an ultrawide-angle image from the auxiliary camera; determining a set of matching features from the first plurality of features and the second plurality of features by using a feature-matching algorithm; building a control point test-field by using the set of matching features and the first 3D point cloud; and determining a set of system calibration parameters on-the-fly using bundle adjustment and camera self-calibration; and performing a limited system calibration of the measurement device in response to capturing a second 3D point cloud that is to be colorized, wherein the limited system calibration comprises updating a subset of system calibration parameters from the set of system calibration parameters using the second 3D point cloud and a second ultrawide-angle image from the auxiliary camera. ### Claim 20 (depends on 19) The computer program product of claim 19, wherein the limited system calibration comprises: extracting a third plurality of features from the control image using the feature-extraction algorithm; determining another set of matching features to the extracted features at the first plurality of features by using a hybrid feature matching algorithm; building another control point test-field by using the another set of matching features and the second 3D point cloud; and determining updated values for at least one of the system calibration parameters.