Method, system, and computer program for recognizing position and posture of object

A system for recognizing the position and a posture of an object, the system comprising:

• a camera that captures an image of the object;
• a depth sensor that measures depths of the object;
• and
• a recognition section that recognizes the position and posture of the object, wherein the recognition section
• (a) generates an object image by capturing an image of the object with the camera,
• (b) determines estimated depths of the object by using a depth estimator that receives the object image as input and outputs the estimated depths in the object image,
• (c) determines whether the estimated depths are useable by using an identifier that receives at least one of the object image and an object image feature quantity extracted from the object image as input and outputs a state in which the estimated depths are usable or unusable,
• (d) when the estimated depths are usable, recognizes the position and posture of the object by using the estimated depths and the object image,
• (e) when the estimated depths are unusable:
• determines measured depths of the object with the depth sensor;
• and
• recognizes the position and posture of the object by using the measured depths and the object image,
• a first training step of training the depth estimator and a second training step of training the identifier before the steps (a) to (e), and
• the first and second training steps each include collecting training-purpose data while causing a robot to pick up the object in accordance with the position and posture of the object.

A non-transitory storage medium storing a computer program that causes a processor to recognize a position and a posture of an object, the computer program causing the processor to

• (a) generate an object image by capturing an image of the object with a camera,
• (b) determine estimated depths of the object by using a depth estimator that receives the object image as input and outputs the estimated depths in the object image,
• (c) determine whether the estimated depths are useable by using an identifier that receives at least one of the object image and an object image feature quantity extracted from the object image as input and outputs a state in which the estimated depths are usable or unusable,
• (d) when the estimated depths are usable, recognize the position and posture of the object by using the estimated depths and the object image,
• (e) when the estimated depths are unusable:
• determine measured depths of the object with a depth sensor;
• and
• recognize the position and posture of the object by using the measured depths and the object image,
• a first training step of training the depth estimator and a second training step of training the identifier before the steps (a) to (e), and
• the first and second training steps each include collecting training-purpose data while causing a robot to pick up the object in accordance with the position and posture of the object.