Techniques for vision-based robot control
AssigneeNVIDIA Corporation
InventorsXiangyun MENG, Xuning YANG, Fabio TOZETO RAMOS, Dieter FOX
Life of this patent
Estimating the term…
Abstract
Techniques for controlling a robot include receiving sensor data and one or more goal specifications, processing the sensor data, a robot size, and the one or more goal specifications using one or more trained encoders to generate a plurality context tokens, processing the plurality of context tokens using one or more trained decoders to generate a robot plan, and controlling a robot based on the robot plan.
Claims20
A computer-implemented method for controlling a robot, the method comprising:
- receiving sensor data and one or more goal specifications;
- processing the sensor data, a robot size, and the one or more goal specifications using one or more trained encoders to generate a plurality context tokens;
- processing the plurality of context tokens using one or more trained decoders to generate a robot plan;
- and
- controlling a robot based on the robot plan.
One or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
- receiving sensor data and one or more goal specifications;
- processing the sensor data, a robot size, and the one or more goal specifications using one or more trained encoders to generate a plurality context tokens;
- processing the plurality of context tokens using one or more trained decoders to generate a robot plan;
- and
- controlling a robot based on the robot plan.
A system comprising:
- one or more memories storing instructions, and
- one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to:
- receive sensor data and one or more goal specifications,
- process the sensor data, a robot size, and the one or more goal specifications using one or more trained encoders to generate a plurality context tokens,
- process the plurality of context tokens using one or more trained decoders to generate a robot plan, and
- control a robot based on the robot plan.
Dependencies are inferred from the claim text.