Safety redundancy and training for embodied agent planning

Abstract

Implementations are provided for safety redundancy and training for embodied agent planning. In various implementations, a safety agent process may obtain visual representation(s) of a robot interacting with an environment. The robot may be controlled at least in part based on commands generated by a robotic planner process that is executed by one or more of the processors. The safety agent process may assemble, as a safety agent input prompt, data indicative of the one or more visual representations. The safety agent process may cause the safety agent input prompt to be processed using a vision language model (VLM) to generate VLM output. Responsive to the VLM output including alternative command(s) for the robot, the safety agent process may generate a control signal for the robot based on the one or more alternative commands.

Dependencies are inferred from the claim text.