Method for generating training data used to learn machine learning model, system, and non-transitory computer-readable storage medium storing computer program

A system for generating training data used to learn a machine learning model for recognizing a position and attitude of a first object of a plurality of objects handled by a robot, the plurality of objects being loaded in bulk on a base, the system comprising:

• a camera capturing an image of the plurality of objects;
• a memory configured to store a program;
• and
• a processor configured to execute the program so as to perform generating the training data, wherein
• the processor is further configured to perform:
• (a) processing of executing prior learning of the machine learning model, using simulation data of the plurality of objects loaded in bulk on the base;
• (b) processing of capturing a first image of the plurality of objects from a first direction of image capture, using the camera, the first image including a first two-dimensional image and a first depth image, the first depth image being acquired in a diffused reflection light condition in which the camera captures diffuse reflection light that is formed by reflecting, along the first direction of image capture, illumination light from a illumination light source at the plurality of objects;
• (c) processing of identifying the first object based on the first image and recognizing a first position and attitude of the first object from the first image, using the machine learning model already learned through the prior learning, the first object being overlapped by another object of the plurality of objects by an overlapping amount, the overlapping amount being 20% or less of an area of the first object being overlapped by the another object;
• (d) processing of performing a correctness determination about the first position and attitude;
• (e) processing of capturing a second image of the first object from a second direction of image capture that is different from the first direction of image capture, using the camera, the second image including a second two-dimensional image and a second depth image, the second depth image being acquired in a specular reflection light condition in which the camera captures specular reflection light that is formed by reflecting, along the second direction of image capture, the illumination light from the illumination light source at the plurality of objects, wherein, when it is determined that the first position and attitude is correct, then converting the first position and attitude according to a change from the first direction of image capture to the second direction of image capture and thus calculating a second position and attitude, and assigning the second position and attitude to the second image and thus generating training data;
• and
• (f) processing of changing an actual position and attitude of the plurality of objects and repeating the processing (b) to (e).

A non-transitory computer-readable storage medium storing a computer program causing a processor to execute processing of generating training data used to learn a machine learning model for recognizing a position and attitude a first object of a plurality of objects handled by a robot, the plurality of objects being loaded in bulk on a base, the computer program causing the processor to execute:

• (a) processing of executing prior learning of the machine learning model, using simulation data of the plurality of objects loaded in bulk on the base;
• (b) processing of capturing a first image of the plurality of objects from a first direction of image capture, using a camera, the first image including a first two-dimensional image and a first depth image, the first depth image being acquired in a diffused reflection light condition in which the camera captures diffuse reflection light that is formed by reflecting, along the first direction of image capture, illumination light from a illumination light source at the plurality of objects;
• (c) processing of identifying the first object based on the first image and recognizing a first position and attitude of the first object from the first image, using the machine learning model already learned through the prior learning, the first object being overlapped by another object of the plurality of objects by an overlapping amount, the overlapping amount being 20% or less of an area of the first object being overlapped by the another object;
• (d) processing of performing a correctness determination about the first position and attitude;
• (e) processing of capturing a second image of the first object from a second direction of image capture that is different from the first direction of image capture, using the camera, the second image including a second two-dimensional image and a second depth image, the second depth image being acquired in a specular reflection light condition in which the camera captures specular reflection light that is formed by reflecting, along the second direction of image capture, the illumination light from the illumination light source at the plurality of objects, wherein, when it is determined that the first position and attitude is correct, then converting the first position and attitude according to a change from the first direction of image capture to the second direction of image capture and thus calculating a second position and attitude, and assigning the second position and attitude to the second image and thus generating training data;
• and
• (f) processing of changing an actual position and attitude of the plurality of objects and repeating the processing (b) to (e).