Mixed reality assisted spatial programming of robotic systems

A system comprising:

• a mixed reality tool configured to select entities of a mixed reality environment for simulation as simulated entities, and select an object related to one or more interactive tasks for the robotic device;
• a skill engine for an automation system, the skill engine comprising an application program comprising a first module configured to determine a spatial location of the object including Cartesian coordinates and orientation coordinates of the object, and a second module configured to execute an application program to operate the robotic device using the spatial location;
• and
• a simulator configured to:
• simulate, in the mixed reality environment, execution of the one or more tasks by the robotic device on the object related to a skill set,
• receive system data from the mixed reality environment comprising virtual data related to characteristics of the simulated entities and real data from one or more real components of the mixed reality environment, and
• process the received system data for various simulations of the mixed reality environment in each of a set of modes, including:
• a real-real mode in which only the real components interface the application program,
• a virtual-virtual mode in which one or more simulated entities respond to the application program and simulate activity or operations tracked by the mixed reality tool to produce virtual data inputs to the skill engine, and
• a real-virtual mode in which the skill engine processes the system data for one or more of the real components simultaneously with data for one or more of the simulated entities;
• wherein the mixed reality tool is further configured to:
• modify the initial parameters in response to instructions gesture inputs received from a user;
• wherein the simulator is further configured to repeat the simulating of the execution of the one or more tasks using the modified parameters;
• and
• wherein the parameters are modified and the execution of the one or more tasks are repeated iteratively until a convergence of parameters is achieved for execution of the one or more tasks within spatial tolerance thresholds.