Kitting machine

A method to operate a kitting system comprising an actuator configured to move a surface or receptacle to a position associated with item retrieval, a first controller configured to control operation of the actuator, and a second controller associated with a robotic retrieval device, the method comprising:

• using information obtained from one or more cameras or sensors in connection with determining one or more items to retrieve and pack into a box or other container, wherein determining the one or more items includes determining a location from which an item of the one or more items is to be retrieved, and determining one or more attributes associated with the item, and the one or more cameras or sensors are located within an environment in which a robotic retrieval device operates to retrieve the one or more items and pack the one or more items into a box or other container;
• obtaining a plan to retrieve the item and to pack the item into the box or other container, wherein:
• the plan is based at least in part on the one or more items to be retrieved, the location from which the item is to be retrieved, and the one or more attributes associated with the item;
• the plan includes a coordination of controlling the robotic retrieval device via the second controller and the actuator via the first controller so that the actuator moves the item to the position associated with item retrieval and the robotic retrieval device moves to retrieve the item from the position associated with item retrieval;
• and
• the obtaining the plan includes:
• determining a plan to retrieve and collectively pack a plurality of items into the box or other container, the plurality of items comprising the item that is retrieved from the position associated with item retrieval;
• using the actuator to position the item into the position associated with item retrieval;
• and
• using the robotic retrieval device to retrieve the item from the position associated with item retrieval and place the item in a destination location, wherein retrieval and placement of the item is based at least in part on the plan, including:
• using at least the robotic retrieval device to attempt to perform an individual task according to a first predefined strategy;
• in response to determining that the performing the individual task according to the first predefined strategy was attempted, determining whether the individual task was completed successfully according to the attempt to implement the first predefined strategy;
• in response to a determination by a processor that the individual task was not completed using the first predefined strategy;
• determining that a plurality of alternative strategies are available to perform the individual task;
• selecting a first alternative strategy to perform the individual task from the plurality of alternative strategies based on a respective confidence score associated with the first alternative strategy;
• and
• iteratively using at least the robotic retrieval device to perform the individual task according to the plurality of alternative strategies for the individual task until the earlier of (i) the individual task is completed, or (ii) a prescribed number of attempts have been performed, wherein the plurality of alternative strategies are iteratively used based on their respective confidence scores;
• and
• in response to a determination by the processor that the prescribed number of attempts have been performed, invoking a teleoperation for human intervention in which a user remotely controls the robotic retrieval device to perform the individual task,
• wherein:
• the actuator is used to position the item in the position associated with item retrieval in a manner that is synchronized at least in part with operation of the robotic retrieval device.

A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions to operate a kitting system comprising an actuator configured to move a surface or receptacle to a position associated with item retrieval, a first controller configured to control operation of the actuator, and a second controller associated with a robotic retrieval device, the computer program product comprising computer instructions for:

• using information obtained from one or more cameras or sensors in connection with determining one or more items to retrieve and pack into the box or other container, wherein determining the one or more items includes determining a location from which an item of the one or more items is to be retrieved, and determining one or more attributes associated with the item, and the one or more cameras or sensors are located within an environment in which the robotic retrieval device operates to retrieve the one or more items and pack the one or more items into a box or other container;
• obtaining a plan to retrieve the item and to pack the item into the box or other container, wherein:
• the plan is based at least in part on the one or more items to be retrieved, the location from which the item is to be retrieved, and the one or more attributes associated with the item;
• the plan includes a coordination of controlling the robotic retrieval device via the second controller and the actuator via the first controller so that the actuator moves the item to the position associated with item retrieval and the robotic retrieval device moves to retrieve the item from the position associated with item retrieval;
• and
• the obtaining the plan includes:
• determining a plan to retrieve and collectively pack a plurality of items into the box or other container, the plurality of items comprising the item that is retrieved from the position associated with item retrieval;
• using the actuator to position the item into the position associated with item retrieval;
• and
• using the robotic retrieval device to retrieve the item from the position associated with item retrieval and place the item in a destination location, wherein retrieval and placement of the item is based at least in part on the plan, including:
• using at least the robotic retrieval device to attempt to perform an individual task according to a first predefined strategy;
• in response to determining that the performing the individual task according to the first predefined strategy was attempted, determining whether the individual task was completed successfully according to the attempt to implement the first predefined strategy;
• in response to a determination by a processor that the individual task was not completed using the first defined strategy;
• determining that a plurality of alternative strategies are available to perform the individual task;
• selecting a first alternative strategy to perform the individual task from the plurality of alternative strategies based on a respective confidence score associated with the first alternative strategy;
• and
• iteratively using at least the robotic retrieval device to perform the individual task according to one or more other predefined strategies for the individual task until the earlier of (i) the individual task is completed, or (ii) a prescribed number of attempts have been performed, wherein the plurality of alternative strategies are iteratively used based on their respective confidence scores;
• and
• in response to a determination by the processor that the prescribed number of attempts have been performed, invoking a teleoperation for human intervention in which a user remotely controls the robotic retrieval device to perform the individual task;
• wherein:
• the actuator is used to position the item in the position associated with item retrieval in a manner that is synchronized at least in part with operation of the robotic retrieval device.