Machine learning based decision making for robotic item handling

A robotic item unloader comprising:

• a vision system comprising:
• a first sensor device positioned at a first location on the robotic item unloader;
• and
• a second sensor device positioned at a second location on the robotic item unloader;
• a robotic arm comprising an end effector configured to operate in a first operating mode;
• a platform comprising a conveyor configured to operate in a second operating mode;
• and
• a processing unit comprising one or more processors, the processing unit communicatively coupled to at least one of the vision system, the platform, and the robotic arm, the processing unit configured to:
• obtain first point cloud data related to a first three-dimensional image captured by the first sensor device;
• obtain second point cloud data related to a second three-dimensional image captured by the second sensor device;
• transform the first point cloud data and the second point cloud data to generate a combined point cloud data;
• construct a machine learning model by using the combined point cloud data as an input to a convolution neural network;
• output, by the machine learning model, a decision classification indicative of a first probability associated with the first operating mode and a second probability associated with the second operating mode, wherein:
• (i) the first operating mode is associated with picking an item by grasping the item using an end effector of a robotic arm of the robotic item unloader, and
• (ii) the second operating mode is associated with sweeping a pile of items from an item docking station with a platform of the robotic item unloader;
• select one of the first operating mode and the second operating mode of the robotic item unloader based on the decision classification outputted by the machine learning model, wherein:
• (i) the first operating mode is selected in response to the first probability being higher than the second probability, and
• (ii) the second operating mode is selected in response to the second probability being higher than the first probability;
• control the robotic item unloader by:
• operating the robotic arm of the robotic item unloader according to the selection of the one of the first operating mode and the second operating mode;
• evaluate the selection of the first operating mode and the second operating mode, based on a pre-defined heuristic associated with past operations of the robotic item unloader;
• adjust a first weight associated with the decision classification and a second weight associated with the pre-defined heuristic used for the evaluation of the selection, based on a performance associated with the output of the machine learning model over a period of time, wherein during initial stages of learning by the machine learning model, the first weight is less than the second weight and after a substantial period of time utilized to train the machine learning model, the second weight is less than the first weight;
• and
• control the robotic item unloader based on the evaluation of the selection.

A non-transitory computer readable medium that stores thereon computer-executable instructions that in response to execution by a processor, perform operations comprising:

• obtaining first point cloud data related to a first three-dimensional image captured by a first sensor device of a robotic item handler;
• obtaining second point cloud data related to a second three-dimensional image captured by a second sensor device of the robotic item handler;
• transforming the first point cloud data and the second point cloud data to generate a combined point cloud data;
• constructing a machine learning model by using the combined point cloud data as an input to a convolution neural network;
• outputting, by the machine learning model, a decision classification indicative of a first probability associated with a first operating mode of the robotic item handler and a second probability associated with a second operating mode, wherein:
• (i) the first operating mode is associated with picking an item by grasping the item using an end effector of a robotic arm of the robotic item handler, and
• (ii) the second operating mode is associated with sweeping a pile of items from an item docking station with a platform of the robotic item handler;
• generating a first command to operate the robotic item handler based on the decision classification, wherein the first command is to:
• (i) operate the robotic item handler according to the first operating mode in response to the first probability being higher than the second probability;
• and
• (ii) operate the robotic item handler according to the second operating mode in response to the second probability being higher than the first probability;
• evaluating a selection of the operating mode of the robotic item handler based on a predefined heuristic associated with past operations of the robotic item handler;
• adjusting a first weight associated with the decision classification and a second weight associated with the pre-defined heuristic used for the evaluation of the selection, based on a performance associated with the output of the machine learning model over a period of time, wherein during initial stages of learning by the machine learning model, the first weight is less than the second weight and after a substantial period of time utilized to train the machine learning model, the second weight is less than the first weight;
• and
• generating a second command to operate the robotic item handler based on the evaluation of the selection.