Methods and systems for testing robotic systems in an integrated physical and simulated environment

A method comprising:

• receiving one or more signals from one or more sensors located in a first physical test environment, the one or more signals indicative of a physical state of one or more elements of the first physical test environment, the one or more elements including at least one of a robot under control, a human, a mobile robot, or any combination thereof;
• determining a state of a model of the first physical test environment based at least in part on the one or more signals received from the one or more sensors located in the first physical test environment;
• determining a state of a model of a virtual test environment based on a desired set of test conditions;
• combining the model of the first physical test environment and the model of the virtual test environment to generate a combined model of the first physical test environment and the virtual test environment;
• determining a set of sensor signals indicative of a state of the combined model of the first physical test environment and the virtual test environment;
• communicating the set of sensor signals to a robot under control or a virtual robot under control;
• controlling the robot under control or the virtual robot under control based on the set of sensor signals;
• communicating image signals indicative of a state of at least one element of the virtual test environment to one or more image projection devices disposed in the first physical test environment;
• and
• projecting an image indicative of the state of the at least one element of the virtual test environment onto the first physical test environment, wherein the at least one element of the virtual test environment is any of a virtual robot under control, a virtual human, a virtual, stationary object, and a virtual, mobile object, wherein the one or more signals from the one or more sensors located in the first physical test environment are indicative of the image indicative of the state of the at least one element of the virtual test environment projected onto the first physical test environment.

A robotic system testing environment, comprising:

• a first physical test environment including one or more sensors and one or more elements, the one or more elements including at least one of a robot under control, a human, a stationary object, a mobile object, or any combination thereof;
• a virtual test environment including at least one of a virtual robot under control, a virtual human, a virtual, stationary object, a virtual, mobile object, one or more virtual sensors, or any combination thereof;
• one or more image projection devices located in the first physical test environment;
• and
• a non-transitory, computer-readable medium storing instructions that when executed by one or more processors cause the one or more processors to:
• receive one or more signals from the one or more sensors located in the first physical test environment, the one or more signals indicative of a physical state of one or more elements of the first physical test environment;
• determine a state of a model of the first physical test environment based at least in part on the one or more signals received from the one or more sensors located in the first physical test environment;
• determine a state of a model of the virtual test environment based on a desired set of test conditions;
• combine the model of the first physical test environment and the model of the virtual test environment to generate a combined model of the first physical test environment and the virtual test environment;
• determine a set of sensor signals indicative of a state of the combined model of the first physical test environment and the virtual test environment;
• communicate the set of sensor signals to a robot under control or a virtual robot under control;
• control the robot under control or the virtual robot under control based on the set of sensor signals;
• and
• communicate image signals indicative of a state of at least one of the virtual robot under control, the virtual human, the virtual, stationary object, and the virtual, mobile object to the one or more image projection devices, wherein the one or more image projection devices project an image indicative of the state of the at least one of the virtual robot under control, the virtual human, the virtual, stationary object, and the virtual, mobile object onto the first physical test environment, and wherein the one or more signals from the one or more sensors located in the first physical test environment are indicative of the image indicative of the state of the at least one of the virtual robot under control, the virtual human, the virtual, stationary object, and the virtual, mobile object projected onto the first physical test environment.