Learned computer control using pointing device and keyboard actions

A computer-implemented method for controlling a particular computer to execute a task, the method comprising:

• receiving a control input that represents at least a current state of the particular computer;
• processing the control input using a neural network to generate one or more control outputs that are used to control the particular computer to execute the task, wherein the one or more control outputs comprise an action type output that specifies one or more actions from a set of possible actions, the set of possible actions comprising game controller button press actions;
• and
• executing the one or more actions to control the particular computer.

An apparatus comprising a particular computer and one or more storage devices storing instructions, that, when executed by the particular computer, cause the particular computer to perform operations comprising:

• determining a control input that represents at least a current state of the particular computer;
• sending the control input to a neural network system implemented by one or more computers;
• receiving, from the neural network systemin, one or more control outputs, wherein the one or more control outputs comprise an action type output that specifies one or more actions to be performed by the particular computer, wherein the one or more actions are from a set of possible actions comprising one or more game controller button press actions;
• and
• executing the one or more actions to control the particular computer.

A mobile device comprising a touchscreen, one or more processors and one or more storage devices storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

• determining a control input that represents at least a current state of the mobile device;
• sending the control input to a neural network system implemented by one or more computers;
• receiving, from the neural network system, one or more control outputs, wherein the one or more control outputs comprise an action type output that specifies one or more actions to be performed by the one or more processors, wherein the one or more actions are from a set of possible actions comprising one or more touchscreen actions;
• and
• executing the one or more actions to control the mobile device.