Virtual teach and repeat mobile manipulation system

An apparatus for controlling a robotic device, comprising:

• at least one processor;
• and
• at least one memory coupled with the at least one processor and storing instructions operable, when executed by the at least one processor, to cause the apparatus to:
• map a plurality of task image pixel descriptors associated with a first group of pixels in a task image of a task environment to a plurality of teaching image pixel descriptors associated with a second group of pixels in a teaching image of a teaching environment based on positioning the robotic device within the task environment;
• determine a relative transform between the task image and the teaching image based on mapping the plurality of task image pixel descriptors, the relative transform indicating a change in one or more of points of three-dimensional (3D) space between the task image and the teaching image;
• update one or more parameters of a set of parameterized behaviors associated with the teaching image based on determining the relative transform;
• and
• control the robotic device to perform a task associated with the set of parameterized behaviors based on updating the one or more parameters.

A non-transitory computer-readable medium having program code recorded thereon for controlling a robotic device, the program code executed by at least one processor and comprising:

• program code to map a plurality of task image pixel descriptors associated with a first group of pixels in a task image of a task environment to a plurality of teaching image pixel descriptors associated with a second group of pixels in a teaching image of a teaching environment based on positioning the robotic device within the task environment;
• program code to determine a relative transform between the task image and the teaching image based on mapping the plurality of task image pixel descriptors, the relative transform indicating a change in one or more of points of three-dimensional (3D) space between the task image and the teaching image;
• program code to update one or more parameters of a set of parameterized behaviors associated with the teaching image based on determining the relative transform;
• and
• program code to control the robotic device to perform a task associated with the set of parameterized behaviors based on updating the one or more parameters.