Pixel-based automated testing of a navigable simulated environment

A method for pixel-based automated testing of a navigable simulated environment generated by an application under test, the method comprising:

• at one or more processors of a computing system, during an inference-time phase:
• receiving, by an application testing program including a machine learning model trained to detect errors in the application under test, a screen image of the navigable simulated environment generated by the application under test;
• determining if a key object is present in the screen image;
• if the key object is determined to be present in the screen image, then generating investigation inputs to investigate the key object;
• and
• if the key object is not determined to be present in the screen image, then generating an environment exploration action to be provided to the application under test as simulated user input.

A computing system for pixel-based automated testing of a navigable simulated environment generated by an application under test, comprising:

• a computing device including one or more processors configured to execute instructions using portions of associated memory to implement an application testing program to detect errors in the application under test, the application testing program including:
• an object detection neural network that processes a screen image of the navigable simulated environment generated by the application under test to determine if a key object is present in the screen image;
• an object investigation neural network that, if a key object is determined to be present in the screen image, receives an image of the detected key object and predicts investigation inputs to investigate the key object;
• and
• an exploration neural network that, if a key object is not determined to be present in the screen image, receives the screen image as input and predicts environment exploration actions to explore an environment included in the screen image, wherein
• the predicted investigation inputs and/or the predicted exploration actions are received by a simulated input generation module that generates investigation inputs and/or exploration actions to be provided to the application under test as simulated user input.