# US 2023/0264702 A1 · Integrated fiducial marker for simultaneously calibrating sensors of different types > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0264702 A1 - Document ID: US20230264702A1 - Status: Published application - Kind: A1 - Application number: 18/141,864 - Filed: 2023-05-01 - Published: 2023-08-24 - Earliest claimed: 2020-04-14 - Assignees: PlusAI, Inc. - Applicants: PlusAI, Inc. - Inventors: Huan Yang CHANG - Web page: https://patentgrep.com/patent/US20230264702A1 ## CPC - B60W 50/06: Improving the dynamic response of the control system, e.g. improving the speed of regulation or avoiding hunting or overshoot - G01S 17/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46) - G01S 7/4091: {during normal radar operation} - G01S 13/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S13/46) - G01S 13/86: Combinations of radar systems with non-radar systems, e.g. sonar, direction finder - G01S 13/865: {Combination of radar systems with lidar systems} - G01S 13/867: {Combination of radar systems with cameras} - G01S 13/931: of land vehicles - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G01S 17/931: of land vehicles - G01S 19/03: Cooperating elements; Interaction or communication between different cooperating elements or between cooperating elements and receivers - G01S 19/14: specially adapted for specific applications - G01S 19/35: Constructional details or hardware or software details of the signal processing chain - G01S 19/51: Relative positioning - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G01S 7/40: Means for monitoring or calibrating - G01S 7/497: Means for monitoring or calibrating - G01S 2013/9316: {combined with communication equipment with other vehicles or with base stations} - G01S 2013/9322: {using additional data, e.g. driver condition, road state or weather data} - G06T 2207/30204: Marker - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract The present teaching relates to different configurations for facilitating calibration of multiple sensors of different types. A plurality of fiducial markers are arranged in space for simultaneously calibrating multiple sensors of different types. Each of the plurality of fiducial markers has a feature point thereon and is provided to enable the multiple sensors to calibrate by detecting the features points and estimating their corresponding 3D coordinates with respect to respective coordinate systems of the multiple sensors. ## Claims ### Claim 1 A computer-implemented method comprising: determining, by a computing system, a location of a marker based on a GPS signal associated with the marker; calibrating, by the computing system, a visual based sensor based on a visual identification of a feature point on the marker and the location of the marker; and calibrating, by the computing system, a depth based sensor based on a depth measurement of the feature point on the marker. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising: calibrating, by the computing system, a radar based sensor based on a distance measurement of the feature point on the marker and the location of the marker. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising: identifying, by the computing system, the feature point in a first coordinate system used by the visual based sensor; identifying, by the computing system, the feature point in a second coordinate system used by the depth based sensor; and determining, by the computing system, a transformation matrix for converting first points in the first coordinate system to second points in the second coordinate system based on the feature point. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the calibrating the visual based sensor comprises: identifying, by the computing system, an intersection of different color patches on the marker, wherein the intersection corresponds with a location of the feature point on the marker. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the calibrating the depth based sensor comprises: determining, by the computing system, scanned depth values of the marker; and determining, by the computing system, a local minimum depth value or a peak depth value from the scanned depth values of the marker, wherein the local minimum depth value or the peak depth value corresponds with a location of the feature point on the marker. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, further comprising: determining, by the computing system, the marker is within a distance threshold of the visual based sensor and the depth based sensor, wherein the distance threshold is increased based on light conditions, and wherein the distance threshold is decreased based on dark conditions; and initiating, by the computing system, calibration of the visual based sensor and the depth based sensor. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, further comprising: receiving, by the computing system, a command from a user to initiate calibration of the visual based sensor and the depth based sensor; and initiating, by the computing system, the calibration of the visual based sensor and the depth based sensor based on the command. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the visual based sensor and the depth based sensor are calibrated simultaneously. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, further comprising: determining, by the computing system, a first orientation of the marker; determining, by the computing system, a discrepancy between the first orientation of the marker and a second orientation of a vehicle; and generating, by the computing system, a control signal to position the vehicle based on the discrepancy. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the visual based sensor is a stereo camera pair, wherein the depth based sensor is a LiDAR sensor, and wherein the stereo camera pair and the LiDAR sensor are installed on a sensor rack. ### Claim 11 A system comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the system to perform operations comprising: determining a location of a marker based on a GPS signal associated with the marker; calibrating a visual based sensor based on a visual identification of a feature point on the marker and the location of the marker; and calibrating a depth based sensor based on a depth measurement of the feature point on the marker. ### Claim 12 (depends on 11) The system of claim 11, the operations further comprising: calibrating a radar based sensor based on a distance measurement of the feature point on the marker and the location of the marker. ### Claim 13 (depends on 11) The system of claim 11, the operations further comprising: identifying the feature point in a first coordinate system used by the visual based sensor; identifying the feature point in a second coordinate system used by the depth based sensor; and determining a transformation matrix for converting first points in the first coordinate system to second points in the second coordinate system based on the feature point. ### Claim 14 (depends on 11) The system of claim 11, wherein the calibrating the visual based sensor comprises: identifying an intersection of different color patches on the marker, wherein the intersection corresponds with a location of the feature point on the marker. ### Claim 15 (depends on 11) The system of claim 11, wherein the calibrating the depth based sensor comprises: determining scanned depth values of the marker; and determining a local minimum depth value or a peak depth value from the scanned depth values of the marker, wherein the local minimum depth value or the peak depth value corresponds with a location of the feature point on the marker. ### Claim 16 A non-transitory computer-readable storage medium including instructions that, when executed by at least on processor of a computing system, cause the computing system to perform: determining a location of a marker based on a GPS signal associated with the marker; calibrating a visual based sensor based on a visual identification of a feature point on the marker and the location of the marker; and calibrating a depth based sensor based on a depth measurement of the feature point on the marker. ### Claim 17 (depends on 16) The non-transitory computer-readable storage medium of claim 16, the operations further comprising: calibrating a radar based sensor based on a distance measurement of the feature point on the marker and the location of the marker. ### Claim 18 (depends on 16) The non-transitory computer-readable storage medium of claim 16, the operations further comprising: identifying the feature point in a first coordinate system used by the visual based sensor; identifying the feature point in a second coordinate system used by the depth based sensor; and determining a transformation matrix for converting first points in the first coordinate system to second points in the second coordinate system based on the feature point. ### Claim 19 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein the calibrating the visual based sensor comprises: identifying an intersection of different color patches on the marker, wherein the intersection corresponds with a location of the feature point on the marker. ### Claim 20 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein the calibrating the depth based sensor comprises: determining scanned depth values of the marker; and determining a local minimum depth value or a peak depth value from the scanned depth values of the marker, wherein the local minimum depth value or the peak depth value corresponds with a location of the feature point on the marker.