US 10,318,917 B1 · Multiple sensor data fusion system

Abstract

An inventory location such as a shelf may be used to stow different types of items. Interactions may take place, such as the pick or place of one or more items from the inventory location. Image data may be acquired from cameras viewing the shelf and weight data may be acquired from weight sensors coupled to the shelf. Hypotheses may be determined that indicate possible interactions with the inventory location, such as pick or place of an item with regard to the inventory location, and the probability that those interactions are correct. The hypotheses and their associated probabilities may be aggregated. From the aggregated hypotheses, a hypothesis with a highest confidence value may be deemed a solution.

A system comprising:

a shelf to hold a first type of item in a first partitioned area and a second type of item in a second partitioned area;
a plurality of weight sensors, each of the plurality of weight sensors positioned at the shelf at a different point;
a camera with a field-of-view that includes at least a portion of the shelf;
and
a computing device comprising:
a memory, storing computer-executable instructions;
and
a hardware processor to execute the computer-executable instructions to:
determine that an interaction has been detected involving at least one of the first type of item and the second type of item on the shelf;
responsive to determining that the interaction has been detected, identify that weight data and image data describing the interaction are available, wherein the weight data is acquired by the plurality of weight sensors and the image data is acquired by the camera;
receive the weight data acquired by the plurality of weight sensors;
receive the image data acquired by the camera, wherein the weight data and the image data are acquired contemporaneously with one another;
access item data indicative of item identifiers associated with respective ones of the first partitioned area and the second partitioned area;
generate, from the image data, motion data indicative of an occurrence of motion at the shelf;
determine, from the image data, a start frame acquired before the motion and an end frame acquired after the motion;
generate, from the start frame and the end frame, data indicative of a change in appearance of the shelf;
determine, based on the image data, a first set of hypotheses for the interaction, each hypothesis of the first set including one or more item identifiers with probability values indicative of a probability that the one or more item identifiers of one or more items were involved in the interaction at the shelf;
determine, based on the weight data, a second set of hypotheses for the interaction, each hypothesis of the second set including:
one or more item identifiers,
a predicted quantity of items for each of the one or more item identifiers, and
probability values indicative of a probability that the hypothesis is true;
combine the first set of hypotheses and the second set of hypotheses using Bayes' rule to generate a third set of hypotheses for the interaction, each hypothesis of the third set including:
one or more item identifiers indicative of one or more items;
and
predicted quantities of the one or more items with probability values indicative of a probability the hypothesis is true;
determine, from the third set of hypotheses for the interaction, a first hypothesis having a first highest probability and a second hypothesis having a second highest probability;
determine a first confidence value for the first hypothesis based on a difference between the first highest probability and the second highest probability of the third set of hypotheses;
compare the first confidence value and a first confidence value threshold;
determine the first confidence value is below the first confidence value threshold;
determine, responsive to the data indicative of a change in appearance of the shelf and based on the start frame and the end frame, a fourth set of hypotheses for the interaction, each hypothesis of the fourth set including:
one or more predicted quantities of the one or more items with probability values indicative of a probability that the hypothesis is true;
combine the third set of hypotheses for the interaction and the fourth set of hypotheses for the interaction using Bayes' rule to generate a fifth set of hypotheses for the interaction, each hypothesis of the fifth set including:
one or more item identifiers,
the predicted quantity of items for each of the one or more item identifiers, and
probability values indicative of a probability that the one or more item identifiers and the predicted quantity of items for each of the one or more item identifiers is true;
select one hypothesis of the fifth set of hypotheses as a solution that describes the interaction;
and
update inventory quantities for the at least one of the first type of item and the second type of item using the solution.

35 more elements

Multiple sensor data fusion system

Life of this patent

Multiple sensor data fusion system

Life of this patent

Abstract

Claims21