Device and method for controlling a robot device

A controller configured to control a robot device, the controller configured to:

• receive sensor data for each of a plurality of sensor data types;
• process the sensor data of the plurality of sensor data types by an encoding and fusion model which includes:
• a sequence of encoding stages, wherein each encoding stage includes an encoding layer for each of the sensor data types generating features for the sensor data of the sensor data type,
• a plurality of fusion layers, each fusion layer combining features of the plurality sensor data types generated by a respective one of the encoding stages and generating an input for a respective subsequent encoding stage in the sequence of encoding stages, wherein a first fusion layer of the plurality of fusion layers generates an input for an encoding stage in the sequence of encoding stages and the encoding stage generates an input for a second fusion layer of the plurality of fusion layers, the encoding stage being subsequent to the first fusion layer and the second fusion layer being subsequent to the encoding stage, and
• an output stage generating an output from an output of a last encoding stage of the sequence of encoding stages;
• select an action to be performed by the robot device using the generated output;
• and
• control the robot device to perform the selected action.

A non-transitory computer-readable medium on which are stored instructions for controlling a robot device, the instructions, when executed by a computer, causing the computer to perform the following steps:

• receiving sensor data for each of a plurality of sensor data types;
• processing the sensor data of the plurality of sensor data types by an encoding and fusion model which includes:
• a sequence of encoding stages, wherein each encoding stage includes an encoding layer for each of the sensor data types generating features for the sensor data of the sensor data type,
• a plurality of fusion layers, each fusion layer combining features of the plurality sensor data types generated by a respective one of the encoding stages and generating an input for a respective subsequent encoding stage in the sequence of encoding stages, wherein a first fusion layer of the plurality of fusion layers generates an input for an encoding stage in the sequence of encoding stages and the encoding stage generates an input for a second fusion layer of the plurality of fusion layers, the encoding stage being subsequent to the first fusion layer and the second fusion layer being subsequent to the encoding stage, and
• an output stage generating an output from an output of a last encoding stage of the sequence of encoding stages;
• selecting an action to be performed by the robot device using the generated output;
• and
• controlling the robot device to perform the selected action.