Object material type identification using multiple types of sensors

A method, comprising:

• obtaining a machine learning model, wherein the machine learning model was trained using training data that was obtained by:
• obtaining a set of visual sensor data of a set of objects from one or more vision sensors;
• obtaining a set of non-visual sensor data of the set of objects from one or more non-vision sensors;
• using the set of non-visual sensor data to determine respective non-visual material characteristics corresponding to the set of objects;
• and
• annotating the set of visual sensor data with labels of the respective non-visual material characteristics corresponding to the set of objects at locations in which the set of objects appears within the set of visual sensor data, wherein the machine learning model is trained based at least in part on the annotated set of visual sensor data to cause the machine learning model to learn correlations between visual characteristics of the set of objects and the respective non-visual material characteristics corresponding to the set of objects;
• obtaining, from a vision sensor located in a sorting facility, a vision sensor signal corresponding to an object within the sorting facility;
• running inference with the machine learning model by providing the vision sensor signal but not a non-vision sensor signal corresponding to the object to the machine learning model to predict a material characteristic type associated with the object;
• and
• causing a sorting device in the sorting facility to perform a sorting operation on the object based at least in part on the material characteristic type associated with the object.

A computer program product, the computer program product being embodied in a non-transitory computer-readable storage medium and comprising computer instructions for:

• obtaining a machine learning model, wherein the machine learning model was trained using training data that was obtained by:
• obtaining a set of visual sensor data of a set of objects from one or more vision sensors;
• obtaining a set of non-visual sensor data of the set of objects from one or more non-vision sensors;
• using the set of non-visual sensor data to determine respective non-visual material characteristics corresponding to the set of objects;
• and
• annotating the set of visual sensor data with labels of the respective non-visual material characteristics corresponding to the set of objects at locations in which the set of objects appears within the set of visual sensor data, wherein the machine learning model is trained based at least in part on the annotated set of visual sensor data to cause the machine learning model to learn correlations between visual characteristics of the set of objects and the respective non-visual material characteristics corresponding to the set of objects;
• obtaining, from a vision sensor located in a sorting facility, a vision sensor signal corresponding to an object within the sorting facility;
• running inference with the machine learning model by providing the vision sensor signal but not a non-vision sensor signal corresponding to the object to the machine learning model to predict a material characteristic type associated with the object;
• and
• causing a sorting device in the sorting facility to perform a sorting operation on the object based at least in part on the material characteristic type associated with the object.