Vision system for a robotic machine

A vision system configured to determine the pose in a local coordinate system of an object gripped by a robotic machine including an arm and an end effector at a distal end thereof configured to grip and manipulate the object, the vision system including:

• a) a first sensor apparatus that in use is mounted to the end effector and having a first location and a first field of view to image a first face of the object when gripped by the end effector;
• b) a second sensor apparatus that in use is mounted to the end effector and having a second location and a second field of view to image at least a portion of a second face of the object orthogonal to the first face when gripped by the end effector;
• and
• c) at least one processor and a memory, the memory comprising instructions to cause the processor to perform a method for determining a pose of an object in a local coordinate system of a robotic machine including:
• i. capturing, using the first sensor apparatus, at least one image of the first face of the object and using the second sensor apparatus, capturing at least one image of at least a portion of the second face of the object, wherein the at least a portion of the second face includes a first edge where the first face meets the second face;
• ii. generating a point cloud representation of at least part of the object using image data obtained from the captured images of the first and second faces;
• iii. fitting a first plane to the first face of the object and fitting a second plane to the second face of the object using the point cloud representation;
• iv. determining a pose of the first plane and a pose of the second plane using a pose of the first and second sensor apparatus in the local coordinate system;
• v. retrieving a shape model of at least the first face of the object comprising dimensions of the at least first face in a scale of the local coordinate system;
• vi. locating the shape model in the local coordinate system using at least in part the at least one image of the first face;
• and,
• vii. determining the pose of the object in the local coordinate system using dimensions of the object, the pose of the first and second plane and the location of the shape model in the local coordinate system.

A robotic machine including:

• a) a base;
• b) an articulated arm extending from the base;
• c) a robotic end effector attached to a distal end of the articulated arm and comprising a first arm, a second arm substantially parallel to the first arm, and a bridging portion to define a Y shaped receiving portion;
• d) a robotic object gripper configured to receive and to grip an object within the Y shaped receiving portion, and
• e) a vision system configured to determine the pose, in a local coordinate system of the robotic machine, of an object gripped by the robotic end effector at a distal end thereof the vision system including:
• a first sensor apparatus mounted to the bridging portion of the robotic end effector and having a first location and a first field of view to image a first face of the object when gripped by the robotic end effector;
• a second sensor apparatus mounted to the robotic end effector and having a second location and a second field of view to image at least a portion of a second face of the object orthogonal to the first face when gripped by the robotic end effector;
• at least one processor and a memory located in the base, the memory comprising instructions to cause the processor to perform a method for determining the pose of an object in the local coordinate system of the robotic machine including:
• i. capturing, using the first sensor apparatus, at least one image of the first face of the object and using the second sensor apparatus, capturing at least one image of at least a portion of the second face of the object, wherein the at least a portion of the second face includes a first edge where the first face meets the second face;
• ii. generating a point cloud representation of at least part of the object using image data obtained from the captured images of the first and second faces;
• iii. fitting a first plane to the first face of the object and fitting a second plane to the second face of the object using the point cloud representation;
• iv. determining a pose of the first plane and a pose of the second plane using a pose of the first and second sensor apparatus in the local coordinate system;
• v. retrieving a shape model of at least the first face of the object comprising dimensions of the at least first face in a scale of the local coordinate system;
• vi. locating the shape model in the local coordinate system using at least in part the at least one image of the first face;
• and,
• vii. determining the pose of the object in the local coordinate system using dimensions of the object, the pose of the first and second plane and the location of the shape model in the local coordinate system
• wherein the second sensor apparatus is mounted in either the first arm or the second arm of the robotic and effector such that the field of view of the second sensor apparatus is substantially orthogonal to the field of view of the first sensor apparatus.