Simulating multiple robots in virtual environments

A system comprising one or more processors and memory storing instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to:

• simulate a three-dimensional virtual environment that includes an interactive object, wherein the three-dimensional virtual environment includes a plurality of robot avatars that are controlled by a corresponding plurality of robot controllers that are external from the three-dimensional virtual environment, wherein a given robot controller of the plurality of robot controllers is operably coupled with the one or more processors;
• provide, to each robot controller of the plurality of robot controllers, sensor data that is generated from a perspective of the respective robot avatar of the plurality of robot avatars that is controlled by the given robot controller, wherein the sensor data provided to the given robot controller is injected into a sensor data channel between one or more real-world sensors of a physical robot and one or more processors of the given robot controller that is integral with the physical robot;
• receive, from each robot controller of the plurality of robot controllers, joint commands that cause actuation of one or more joints of the respective robot avatar of the plurality of robot avatars that is controlled by the given robot controller;
• and
• actuate one or more joints of each robot avatar of the plurality of robot avatars pursuant to corresponding joint commands, wherein the actuating causes two or more of the plurality of robot avatars to act upon the interactive object in the three-dimensional virtual environment, wherein the actuating comprises operating a robot avatar that is controlled by the given robot controller at a frequency that corresponds to a real-world frequency of the given robot controller.

A method implemented using one or more processors, comprising:

• simulating a three-dimensional virtual environment that includes an interactive object, wherein the three-dimensional virtual environment includes a plurality of robot avatars that are controlled by a corresponding plurality of robot controllers that are external from the three-dimensional virtual environment, wherein a given robot controller of the plurality of robot controllers is operably coupled with the one or more processors;
• providing, to each robot controller of the plurality of robot controllers, sensor data that is generated from a perspective of the respective robot avatar of the plurality of robot avatars that is controlled by the given robot controller, wherein joint commands received from the given robot controller are intercepted from a joint command channel between one or more processors of the given robot controller and one or more joints of a physical robot;
• receiving, from each robot controller of the plurality of robot controllers, joint commands that cause actuation of one or more joints of the respective robot avatar of the plurality of robot avatars that is controlled by the given robot controller;
• and
• actuating one or more joints of each robot avatar of the plurality of robot avatars pursuant to corresponding joint commands, wherein the actuating causes two or more of the plurality of robot avatars to act upon the interactive object in the three-dimensional virtual environment, wherein the actuating comprises operating a robot avatar that is controlled by the given robot controller at a frequency that corresponds to a real-world frequency of the given robot controller.

A system comprising one or more processors and memory storing instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to:

• simulate a three-dimensional virtual environment that includes an interactive object, wherein the three-dimensional virtual environment includes a plurality of robot avatars that are controlled by a corresponding plurality of robot controllers that are external from the three-dimensional virtual environment, wherein a given robot controller of the plurality of robot controllers is operably coupled with the one or more processors;
• provide, to each robot controller of the plurality of robot controllers, sensor data that is generated from a perspective of the respective robot avatar of the plurality of robot avatars that is controlled by the given robot controller, wherein joint commands received from the given robot controller are intercepted from a joint command channel between one or more processors of the given robot controller and one or more joints of a physical robot;
• receive, from each robot controller of the plurality of robot controllers, joint commands that cause actuation of one or more joints of the respective robot avatar of the plurality of robot avatars that is controlled by the given robot controller;
• and
• actuate one or more joints of each robot avatar of the plurality of robot avatars pursuant to corresponding joint commands, wherein the actuating causes two or more of the plurality of robot avatars to act upon the interactive object in the three-dimensional virtual environment, wherein the actuating comprises operating a robot avatar that is controlled by the given robot controller at a frequency that corresponds to a real-world frequency of the given robot controller.