Simulating physical interactions for automated systems

A method, comprising:

• obtaining motion data representing related motion of a first object and a second object in a physical environment;
• analyzing the motion data to generate one or more representations of the related motion;
• providing, based at least on the one or more representations, a simulation of the related motion of the first object and the second object in a virtual environment;
• detecting interaction of a virtual model of a robot with the first object in the virtual environment;
• and
• modifying simulation of at least one of the first object or the second object in the virtual environment based at least on the interaction.

A system, comprising:

• one or more processors to:
• analyze motion data representing motion of a human hand interacting with an object in a physical environment to generate a first representation of the motion of the human hand and a second representation of corresponding motion of the object;
• provide a simulation including a virtual robot to take possession of a virtual object from a virtual hand in a virtual environment;
• cause, during the simulation, the virtual hand to move according to the first representation of motion and the virtual object to move according to the second representation of motion;
• and
• determine a successful transfer of possession of the virtual object from the virtual hand to the virtual robot in response to the virtual object being in simulated contact with an end effector of the virtual robot for at least a minimum period of time.