Robot interaction with objects based on semantic information associated with embedding spaces

A system comprising one or more processors and memory operably coupled with the one or more processors, wherein the memory stores instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations:

• obtaining, from one or more sensors of a robot, sensor data that includes data indicative of an object observed in an environment in which the robot operates;
• processing the sensor data as input utilizing a first trained machine learning model to generate a first embedded feature vector, wherein the first embedded feature vector maps the data indicative of the object to an embedding space;
• identifying, for the first embedded feature vector, at least one nearest neighbor in the embedding space;
• identifying one or more semantic labels based at least in part on the at least one nearest neighbor, wherein the one or more semantic labels are associated with a pose;
• selecting a given grasp option from one or more enumerated grasp options previously associated with the pose;
• and
• operating the robot to interact with the object based on the pose and using the given grasp option.

At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations:

• obtaining, from one or more sensors of a robot, sensor data that includes data indicative of an object observed in an environment in which the robot operates;
• processing the sensor data as input utilizing a first trained machine learning model to generate a first embedded feature vector, wherein the first embedded feature vector maps the data indicative of the object to an embedding space;
• identifying, for the first embedded feature vector, at least one nearest neighbor in the embedding space;
• identifying one or more semantic labels based at least in part on the at least one nearest neighbor, wherein the one or more semantic labels are associated with a pose;
• selecting a given grasp option from one or more enumerated grasp options previously associated with the pose;
• and
• operating the robot to interact with the object based on the pose and using the given grasp option.