Testing robotic software systems using perturbations in simulation environments

A non-transitory computer-readable storage medium coupled to one or more processors and having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations for testing a robotic system, operations comprising:

• providing a robotic model representative of the robotic system that is to operate in a real-world environment;
• determining baseline data by executing a simulation using the robotic model and a simulation world, the simulation world representing the real-world environment,
• wherein the robotic model and the simulation world comprises one or more objects,
• wherein the baseline data indicates a pattern of movement of an object of the one or more objects and an envelope of an acceptable tolerance of the pattern,
• wherein the baseline data includes a set of active nodes of a plurality of nodes of the robotic model and the simulation world, and
• wherein the set of active nodes comprising nodes that are active during the simulation;
• generating, by a perturbation system, perturbations to be included in a set of test cases,
• wherein each test case of the set of test cases comprising at least one perturbation of the perturbations to at least one object of the one or more objects of the robotic model and the simulation world relative to a baseline of the at least one object determined from the baseline data,
• wherein a first set of perturbations of the perturbations is generated by a machine learning model trained on real-world training data that represents actual variations of characteristics of the one or more objects, and
• wherein the at least one perturbation comprising a latency perturbation that represents a disruption in communication between nodes of the plurality of nodes of the robotic model and the simulation world;
• determining test result data for each test case of the set of test cases by, for each test case, modifying the at least one object in view of the at least one perturbation to provide a modified object based on the test result data and executing the simulation with the modified object,
• wherein executing the simulation with the modified object comprises applying the at least one perturbation to the set of active nodes,
• wherein the test result data for a test case of the set of test cases indicates one of a pass, a soft failure, or a hard failure related to the object,
• wherein the soft failure indicates that:
• a simulated pattern of movement of the object follows the pattern of the movement of the object indicated by the baseline data, and
• the simulated pattern of the movement of the object is outside the envelope, and
• wherein the hard failure indicates that the simulated pattern of the movement of the object deviates from the pattern of the movement of the object indicated by the baseline data;
• and
• displaying a graph representative of the at least one object based on the baseline data and the test result data for the test case of the set of test cases, the graph depicting variance of the object during the simulation using the test case based on one of the pass, the soft failure, or the hard failure.

A system, comprising:

• a computing device;
• and
• a computer-readable storage device coupled to the computing device and having instructions stored thereon which, when executed by the computing device, cause the computing device to perform operations for testing a robotic system, operations comprising:
• providing a robotic model representative of the robotic system that is to operate in a real-world environment;
• determining baseline data by executing a simulation using the robotic model and a simulation world, the simulation world representing the real-world environment,
• wherein the robotic model and the simulation world comprises one or more objects,
• wherein the baseline data indicates a pattern of movement of an object of the one or more objects and an envelope of an acceptable tolerance of the pattern,
• wherein the baseline data includes a set of active nodes of a plurality of nodes of the robotic model and the simulation world, and
• wherein the set of active nodes comprising nodes that are active during the simulation;
• generating, by a perturbation system, perturbations to be included in a set of test cases,
• wherein each test case of the set of test cases comprising at least one perturbation of the perturbations to at least one object of the one or more objects of the robotic model and the simulation world relative to a baseline of the at least one object determined from the baseline data,
• wherein a first set of perturbations of the perturbations is generated by a machine learning model trained on real-world training data that represents actual variations of characteristics of the one or more objects, and
• wherein the at least one perturbation comprising a latency perturbation that represents a disruption in communication between nodes of the plurality of nodes of the robotic model and the simulation world;
• determining test result data for each test case of the set of test cases by, for each test case, modifying the at least one object in view of the at least one perturbation to provide a modified object based on the test result data and executing the simulation with the modified object,
• wherein executing the simulation with the modified object comprises applying the at least one perturbation to the set of active nodes,
• wherein the test result data for a test case of the set of test cases indicates one of a pass, a soft failure, or a hard failure related to the object,
• wherein the soft failure indicates that:
• a simulated pattern of movement of the object follows the pattern of the movement of the object indicated by the baseline data, and
• the simulated pattern of the movement of the object is outside the envelope, and
• wherein the hard failure indicates that the simulated pattern of the movement of the object deviates from the pattern of the movement of the object indicated by the baseline data;
• and
• displaying a graph representative of the at least one object based on the baseline data and the test result data for the test case of the set of test cases, the graph depicting variance of the object during the simulation using the test case based on one of the pass, the soft failure, or the hard failure.