System and method for unknown object manipulation from pure synthetic stereo data

A non-transitory computer-readable medium having program code recorded thereon for training a neural network to perform 3D object manipulation, the program code being executed by a processor and comprising:

• program code to extract features from each image of a synthetic stereo pair of images;
• program code to generate a low-resolution disparity image based on the features extracted from each image of the synthetic stereo pair of images;
• program code to generate a feature map based on the low-resolution disparity image and features extracted from a single one of the synthetic stereo pair of images using the neural network;
• and
• program code to manipulate an unknown object perceived from the feature map according to a perception prediction from a prediction head.

A system for training a neural network to perform 3D object manipulation, the system comprising:

• a stereo feature extraction module to extract features from each image of a synthetic stereo pair of images;
• a disparity image generation module to generate a low-resolution disparity image based on the features extracted from each image of the synthetic stereo pair of images;
• a feature map generation module to generate a feature map based on the low-resolution disparity image and features extracted from a single one of the synthetic stereo pair of images using the neural network;
• and
• a 3D object manipulation module to manipulate an unknown object perceived from the feature map according to a perception prediction from a prediction head.