# US 12,000,923 B2 · Sensor fusion with alternating correspondence analysis of sensor data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,000,923 B2 - Document ID: US12000923B2 - Status: Granted - Kind: B2 - Application number: 17/264,476 - Filed: 2019-07-31 - Published: 2021-10-14 - Granted: 2024-06-04 - Earliest claimed: 2018-08-01 - Assignees: Carl Zeiss AG - Applicants: Carl Zeiss AG - Inventors: Daniel Spies; Matthias Karl - Web page: https://patentgrep.com/patent/US12000923B2 ## CPC - G01S 13/862: {Combination of radar systems with sonar systems} - G01S 13/865: {Combination of radar systems with lidar systems} - G01S 13/867: {Combination of radar systems with cameras} - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G06F 18/2113: by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation - G06F 18/251: {of input or preprocessed data} - G06N 20/00: Machine learning - G06V 20/20: in augmented reality scenes - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Sensor data are received from a multiplicity of sensors, wherein the multiplicity of sensors image a common scene using a multiplicity of measurement modalities. For each sensor of the multiplicity of sensors, at least one corresponding feature is determined in the respective sensor data and corresponding performance specification data are also obtained. In addition, mutual correspondence analysis between the features of the sensor data is carried out taking into account the corresponding performance specification data. Sensor fusion is also carried out on the basis of the correspondence analysis. ## Claims ### Claim 1 A computer-implemented sensor fusion method, wherein the method comprises: receiving sensor data from a multiplicity of sensors, wherein the multiplicity of sensors image a common scene using a multiplicity of measurement modalities, wherein the multiplicity of sensors comprise at least a first sensor providing first sensor data and a second sensor providing second sensor data, for each sensor of the multiplicity of sensors: determining at least one corresponding feature in the respective sensor data, for each sensor of the multiplicity of sensors: obtaining corresponding performance specification data, carrying out a mutual correspondence analysis at least between first features of the first sensor data and second features of the second sensor data taking into account the corresponding performance specification data of the first sensor and the second sensor, the mutual correspondence analysis determining whether, for one or more of the first features that are selected in accordance with the performance specification data, a corresponding one of the second features is present or whether a corresponding one of the second features is absent, and carrying out the sensor fusion on the basis of a result of the correspondence analysis. ### Claim 2 (depends on 1) The method as claimed in claim 1, wherein the performance specification data are indicative of at least one limitation of the respective measurement modality for imaging features with particular properties. ### Claim 3 (depends on 2) The method as claimed in claim 2, wherein the at least one limitation comprises a lower threshold value for a lateral extent of objects in the scene. ### Claim 4 (depends on 2) The method as claimed in claim 2, wherein the at least one limitation comprises a geometrical orientation of objects in the scene on the basis of a pose of the corresponding sensor with respect to the object. ### Claim 5 (depends on 2) The method as claimed in any one of claim 2, wherein the at least one limitation comprises a reflectivity of a surface of an object in the scene. ### Claim 6 (depends on 1) The method as claimed in claim 1, wherein the at least one feature is respectively determined on the basis of an individual measurement of the sensor data. ### Claim 7 A computer-implemented sensor fusion method, wherein the method comprises: receiving sensor data from a multiplicity of sensors, wherein the multiplicity of sensors image a common scene using a multiplicity of measurement modalities, for each sensor of the multiplicity of sensors: determining at least one corresponding feature in the respective sensor data, for each sensor of the multiplicity of sensors: obtaining corresponding performance specification data, carrying out mutual correspondence analysis between the features of the sensor data taking into account the corresponding performance specification data, carrying out the sensor fusion on the basis of the correspondence analysis if the correspondence analysis indicates, as the result, a limited correspondence which is plausible within the scope of the corresponding performance specification data: using a scene reconstruction algorithm. ### Claim 8 (depends on 7) The method as claimed in claim 7, wherein the scene reconstruction algorithm corrects the limited correspondence on the basis of predefined reconstruction rules for expected features of the scene. ### Claim 9 (depends on 8) The method as claimed in claim 8, wherein the predefined reconstruction rules are trained on the basis of machine learning. ### Claim 10 (depends on 8) The method as claimed in claim 8, wherein the correspondence rules are varied in a plurality of iterations of the scene reconstruction algorithm until an abort criterion has been satisfied, wherein the abort criterion comprises a plausibility of a result of the scene reconstruction algorithm on the basis of the sensor data. ### Claim 11 (depends on 7) The method as claimed in any claim 7, furthermore comprising: adapting the performance specification data on the basis of a result of the scene reconstruction algorithm. ### Claim 12 (depends on 7) The method as claimed in claim 7, wherein the scene reconstruction algorithm uses a-priori assumptions relating to the scene. ### Claim 13 (depends on 1) The method as claimed in claim 1, wherein the at least one corresponding feature is determined for a respective sensor of the multiplicity of sensors on the basis of respective calibration data, wherein the calibration data describe imaging properties of the respective sensor. ### Claim 14 (depends on 1) The method as claimed in claim 1, furthermore comprising: controlling the movement of a robot in the scene on the basis of sensor fusion. ### Claim 15 An apparatus having at least one processor and a memory, wherein the at least one processor is configured to load a program code from the memory and to execute it and to carry out the following steps on the basis of the execution of the program code: receiving sensor data from a multiplicity of sensors, wherein the multiplicity of sensors image a common scene using a multiplicity of measurement modalities, wherein the multiplicity of sensors comprise at least a first sensor providing first sensor data and a second sensor providing second sensor data, for each sensor of the multiplicity of sensors: determining at least one corresponding feature in the respective sensor data, for each sensor of the multiplicity of sensors: obtaining corresponding performance specification data, carrying out a mutual correspondence analysis between at least first features of the first sensor data and second features of the second sensor data taking into account the corresponding performance specification data of the first sensor and the second sensor, the mutual correspondence analysis determining whether, for one or more of the first features that are selected in accordance with the performance specification data, a corresponding one of the second features is present or whether a corresponding one of the second features is absent, and carrying out the sensor fusion on the basis of a result of the correspondence analysis. ### Claim 16 A computer-implemented sensor fusion method, wherein the method comprises: receiving sensor data from a multiplicity of sensors, wherein the multiplicity of sensors image a common scene using a multiplicity of measurement modalities, for each sensor of the multiplicity of sensors: determining at least one corresponding feature in the respective sensor data, for each sensor of the multiplicity of sensors: obtaining corresponding performance specification data, wherein the performance specification data are indicative of at least one limitation of the respective measurement modality for imaging features with particular properties, wherein the at least one limitation comprises at least one of a lower threshold value for a lateral extent of objects in the scene, a geometrical orientation of objects in the scene on the basis of a post of the corresponding sensor with respect to the object, or a reflectivity of a surface of an object in the scene, carrying out mutual correspondence analysis between the features of the sensor data taking into account the corresponding performance specification data, and carrying out the sensor fusion on the basis of the correspondence analysis.