Techniques for closed-loop code generation for robot control
AssigneeNVIDIA Corporation
InventorsValts BLUKIS, Alexander ZOOK, Stanley BIRCHFIELD, Balakumar SUNDARALINGAM, Jonathan TREMBLAY
Life of this patent
Estimating the term…
Abstract
One embodiment of a method for processing data includes receiving an image; segmenting, using a first trained machine learning model, the image to generate a segmentation mask; generating one or more descriptions of one or more objects using a second machine learning model and based on the segmentation mask; generating program code using a third trained machine learning model and based on the one or more descriptions, the image, and a task; and causing a robot to move based on the program code.
Claims20
A computer-implemented method for robot control, the method comprising:
- causing a robot to move within an environment based on first program code;
- performing one or more operations to verify a state of the environment based on an assertion included in the first program code;
- and
- in response to not verifying the state of the environment:
- generating second program code using a first trained machine learning model and based on (i) the state of the environment and (ii) a task, and
- causing the robot to move based on the second program code.
One or more non-transitory computer-readable media storing instructions that, when executed by at least one processor, cause the at least one processor to perform the steps of:
- causing a robot to move within an environment based on first program code;
- performing one or more operations to verify a state of the environment based on an assertion included in the first program code;
- and
- in response to not verifying the state of the environment:
- generating second program code using a first trained machine learning model and based on (i) the state of the environment and (ii) a task, and
- causing the robot to move based on the second program code.
A system, comprising:
- a memory storing instructions;
- and
- one or more processors, that when executing the instructions, are configured to perform the steps of:
- causing a robot to move within an environment based on first program code,
- performing one or more operations to verify a state of the environment based on an assertion included in the first program code, and
- in response to not verifying the state of the environment:
- generating second program code using a first trained machine learning model and based on (i) the state of the environment and (ii) a task;
- and
- causing the robot to move based on the second program code.
Dependencies are inferred from the claim text.