System and method for controlling autonomous machinery by processing rich context sensor inputs

A computer-implemented method for controlling an autonomous machine, comprising:

• acquiring sensor data streamed via a plurality of sensors calibrated with respect to a common real world reference frame centered on the autonomous machine,
• processing the streamed sensor data by a plurality of perception modules to extract perception information from the sensor data in real time,
• fusing the extracted real time perception information from the plurality of perception modules by a context awareness module to create a blackboard image, wherein the blackboard image is a representation of an operating environment of the autonomous machine derived from fusion of the extracted perception information using a controlled semantic, which defines a context of the autonomous machine,
• whereby the streamed sensor data is transformed into a stream of blackboard images defining an evolution of context of the autonomous machine with time, and
• processing the stream of blackboard images by an action evaluation module using a control policy to output a control action to be executed by the autonomous machine, the control policy comprising a learned mapping of context to control action using training data in which contexts are represented by blackboard images created using the controlled semantic.