Simulating physical interactions for automated systems

A method comprising:

• obtaining motion data of at least one hand of a subject in a physical environment interacting with an object in the physical environment;
• performing, based at least partially on the motion data, a simulation of a robotic assembly interacting with the at least one hand of the subject and the object in a virtual environment;
• and
• modifying, based on one or more results of the simulation, one or more parameters of a machine learning model for producing one or more inferences corresponding to an operation of one or more portions of the robotic assembly in the physical environment;
• and
• causing, using the machine learning model with the one or more modified parameters, one or more operations to be performed by the robotic assembly in the physical environment.

A system comprising one or more processors to:

• obtain motion data of at least one hand of a subject in a physical environment interacting with an object in the physical environment;
• perform, based at least partially on the motion data, a simulation of a robotic assembly interacting with the at least one hand and the object in a virtual environment;
• modify, based on one or more results of the simulation, one or more parameters of a machine learning model to produce one or more inferences corresponding to an operation of one or more portions of the robotic assembly in the physical environment;
• and
• cause, using the machine learning model with the one or more modified parameters, one or more actions to be performed by the robotic assembly in the physical environment.