# US 2021/0323164 A1 · Automatic sensor conflict resolution for sensor fusion system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0323164 A1
- Document ID: US20210323164A1
- Status: Published application
- Kind: A1
- Application number: 16/851,946
- Filed: 2020-04-17
- Published: 2021-10-21
- Earliest claimed: 2020-04-17
- Assignees: ABB Schweiz AG
- Applicants: ABB Schweiz AG
- Inventors: Biao Zhang; Jianjun Wang; Yixin Liu; Saumya Sharma; Jorge Vidal-Ribas; Jordi Artigas; Ramon Casanelles
- Web page: https://patentgrep.com/patent/US20210323164A1

## CPC

- B25J 9/1694: {characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion}
- B25J 9/1653: {parameters identification, estimation, stiffness, accuracy, error analysis}
- B25J 9/1674: {characterised by safety, monitoring, diagnostic}
- B25J 9/1692: {Calibration of manipulator}
- B25J 13/089: {Determining the position of the robot with reference to its environment}
- G05B 23/0221: {Preprocessing measurements, e.g. data collection rate adjustment; Standardization of measurements; Time series or signal analysis, e.g. frequency analysis or wavelets; Trustworthiness of measurements; Indexes therefor; Measurements using easily measured parameters to estimate parameters difficult to measure; Virtual sensor creation; De-noising; Sensor fusion; Unconventional preprocessing inherently present in specific fault detection methods like PCA-based methods}
- G06F 18/25: Fusion techniques
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06V 10/80: Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10)
- G05B 2219/25428: Field device
- G05B 2219/37331: Sensor fusion using extended kalman filter
- G05B 2219/40412: Sensor knowledge command fusion network, data and feature and action and constraint
- G06V 2201/06: Recognition of objects for industrial automation
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A system and method that automatically resolves conflicts among sensor information in a sensor fusion robot system. Such methods can accommodate converging ambiguous and divergent sensor information in a manner that can allow continued, and relatively accurate, robotic operations. The processes can include handling sensor conflict via sensor prioritization, including, but not limited, prioritization based on the particular stage or segment of the assembly operation when the conflict occurs, overriding sensor data that exceeds a threshold value, and/or prioritization based on evaluations of recent sensor performance, predictions, system configuration, and/or historical information. The processes can include responding to sensor conflicts through comparisons of the accuracy of workpiece location predictions from different sensors during different assembly stages in connection with arriving at a determination of which sensor(s) is providing accurate and reliable predictions.

## Claims

### Claim 1

A method comprising:
predicting, using information from a plurality of sensors, a location of a workpiece;
detecting a conflict between information from at least some of the plurality of sensors;
displacing a robot to a first location, the first location being determined using the predicted location of the workpiece;
detecting an actual location of the workpiece; and
identifying a sensor from the plurality of sensors that provided the information that resulted in the closest prediction of the actual location of the workpiece.

### Claim 2 (depends on 1)

The method of claim 1, further including:
predicting, based on information from the identified sensor, another location of the workpiece; and
displacing the robot to a second location, the second location being determined using the predicted other location.

### Claim 3 (depends on 1)

The method of claim 1, further including:
applying, for each sensor of the plurality of sensors, information from the sensor to determine another predicted location for the workpiece;
displacing the robot to a second location, the second location being determined based on the information from the identified sensor;
detecting a location of the workpiece at the other location;
determining, for each of the plurality of sensors, an accuracy of the other predicted location for the workpiece based on the detected location of the workpiece;
selecting, based on the determined accuracy, one or more sensors from the plurality of sensors to use with the identified sensor in determining another location for the robot.

### Claim 4 (depends on 1)

The method of claim 1, wherein predicting the location of the workpiece comprises using information from different sensors of the plurality of sensors to derive a plurality of predictions of the location of the workpiece.

### Claim 5 (depends on 4)

The method of claim 4, wherein identifying the sensor includes determining an accuracy for each of the plurality of predicted locations of the workpiece in view of the actual location.

### Claim 6 (depends on 1)

The method of claim 1, further including tracking a movement of the workpiece by the plurality of sensors, and wherein the information from the plurality of sensors comprises information obtained from tracking the workpiece. A method comprising:
tracking, by a plurality of sensors, movement of a workpiece along a robotic assembly operation;
applying, for each sensor of the plurality of sensors, information obtained by tracking movement of the workpiece to determine a predicted workpiece location;
assigning to one or more of the predicted workpiece locations a weighted value;
determining, using the predicted workpiece location for each of the plurality of sensors and the weighted value, an adjusted predicted workpiece location; and
displacing a robot to a location that is determined, at least in part, using the adjusted predicted workpiece location.

### Claim 8 (depends on 7)

The method of claim 7, wherein the step of assigning the weighted value is predicated on detecting a conflict between the predicted workpiece location determined for two or more sensors of the plurality of sensors.

### Claim 9 (depends on 8)

The method of claim 8, further including the step of determining if a threshold number of assembly cycles have been complete, and wherein the step of assigning the weighted value is predicated on satisfaction of completion of the threshold number of assembly cycles.

### Claim 10 (depends on 9)

The method of claim 9, further including:
detecting an actual location of the workpiece;
determining an accuracy of the predicted workpiece location for each sensor based on the actual location of the workpiece; and
adjusting, based on the determined accuracy, the weighted value of the predicted workpiece location for one or more sensor of the plurality of sensors.

### Claim 11 (depends on 10)

The method of claim 10, further including the step of compiling a record of the determined accuracies, and wherein adjusting the weighted value is predicated on the accuracies being determined for a threshold number of assembly operations.

### Claim 12 (depends on 8)

The method of claim 8, further including the step of determining, for one or more of the plurality of sensors, a value of the weighted value, and wherein the value is at least partially based on a current assembly stage of an assembly cycle.

### Claim 13 (depends on 12)

The method of claim 12, wherein the step of determining the value comprises assigning a different value to the weighted value for different assembly stages of the assembly cycle.

### Claim 14 (depends on 13)

The method of claim 13, wherein the step of determining the value comprises assigning the value based on a proximity of one or more sensors of the plurality of sensors to the workpiece during one or more of the assembly stages.

### Claim 15 (depends on 8)

The method of claim 8, further including the step of determining, for one or more of the plurality of sensors, a value of the weighted value, wherein the value is based on a determined historical accuracy of one or more of the plurality of sensors.

### Claim 16 (depends on 7)

The method of claim 7, further including the step of assigning a predetermined value to the weighted value.

### Claim 17 (depends on 16)

The method of claim 16, wherein the step of assigning the predetermined value includes determining the value based on an outcome of calibrating one or more of the plurality of sensors.

### Claim 18

A method comprising:
detecting if both a natural tracking feature and an artificial tracking feature are in a field of view of a vision device of a vision guidance system;
determining, if both the natural tracking feature and the artificial tracking feature are detected in the field of view, a reliability of the detection of the natural tracking feature; and
referencing the artificial tracking feature for tracking the natural tracking feature based on a determination of the reliability of the natural tracking feature.

### Claim 19 (depends on 18)

The method of claim 18, further including the steps of:
determining if only one of the natural tracking feature or the artificial tracking feature is in the field of view;
determining, if only one of the natural tracking feature or the artificial tracking feature is in the field of view, a reliability for the natural tracking feature or the artificial tracking feature that is in the field of view; and
expanding, based on the determined reliability of the natural tracking feature or the artificial tracking feature that is in the field of view, the field of view until both the natural tracking feature and the artificial tracking feature are in the field of view.

### Claim 20 (depends on 19)

The method of claim 19, further including:
expanding, if neither the natural tracking feature nor the artificial tracking feature is in the field of view, the field of view of the vision device until both of the natural tracking feature and the artificial tracking feature are in the expanded field of view; and
tracking, upon both the natural tracking feature and the artificial tracking feature being in the expanded field of view, at least one of the natural tracking feature and the artificial feature.
