Multimodal identification system and method for robotic item manipulators

An apparatus comprising:

• a memory;
• and
• a hardware processor communicatively coupled to the memory, the hardware processor configured to:
• determine one or more of a weight of an item, a packaging type of the item, a packaging material of the item, a barcode of the item, a rigidity of the item, or a physical response of the item to being lifted;
• determine a visual appearance of the item;
• determine, using the visual appearance and the shape or size, a preliminary identity of the item before the item is lifted out of a container, wherein the preliminary identity is used to determine which item to lift out of the container;
• determine a shape or a size of the item;
• compare, using a machine learning model, the visual appearance, the shape or size, and one or more of the weight, packaging type, packaging material, barcode, rigidity, or the physical response of the item to a manifest for the container, wherein the manifest identifies a plurality of items in the container;
• and
• determine, using the machine learning model, an identity of the item based on comparing the visual appearance, the shape or size, and one or more of the weight, packaging type, packaging material, barcode, rigidity, or the physical response of the item to the manifest, wherein the identity is different than the preliminary identity.

A method comprising:

• determining one or more of a weight of an item, a packaging type of the item, a packaging material of the item, a barcode of the item, a rigidity of the item, or a physical response of the item to being lifted;
• determining a visual appearance of the item;
• determining, using the visual appearance and the shape or size, a preliminary identity of the item before the item is lifted out of a container, wherein the preliminary identity is used to determine which item to lift out of the container;
• determining a shape or size of the item;
• comparing, using a machine learning model or a combinatorial optimization algorithm, the visual appearance, the shape or size, and one or more of the weight, packaging type, packaging material, barcode, rigidity, or the physical response of the item to a manifest for the container, wherein the manifest identifies a plurality of items in the container;
• determining, using the machine learning model or the combinatorial optimization algorithm, an identity of the item based on comparing the visual appearance, the shape or size, and one or more of the weight, packaging type, packaging material, barcode, rigidity, or the physical response of the item to the manifest;
• and
• communicating an instruction to a robotic arm to sort the item based on the determined identity.