System error detection

A computer-implemented method comprising:

• collecting, by an event collection apparatus and for each of a plurality of executions of an application on a device, an event log for the execution that identifies automated interactions with the application performed by an application testing agent on the device, each event log comprising, for each of multiple automated events that occurred on the device in response to one or more of the automated interactions, one or more first parameter types and, for each of the first parameter types, a corresponding first parameter value;
• storing, by the event collection apparatus and for each of the plurality of executions of the application on the device, the event log in one or more content storage devices;
• storing, by the event collection apparatus and for one or more of the plurality of executions of the application on the device and in the one or more content storage devices, event tracking data captured by a third party system, separate from an event verification apparatus, during the automated interaction with the application on the device by the application testing agent, the event tracking data comprising, for each of at least some of the multiple automated events that occurred on the device, one or more second parameter types and, for each of the second parameter types, a corresponding second parameter value;
• comparing, by the event verification apparatus and for some of the plurality of executions of the application, the first parameter types with the second parameter types to determine whether expected parameter types from the first parameter types are included in the second parameter types;
• comparing, by the event verification apparatus and for each of the expected parameter types that are included in the second parameter types, the corresponding first parameter values with the corresponding second parameter values for the same parameter type to determine whether the corresponding values are the same;
• and
• detecting, by the event verification apparatus, a software error in one or more of the application, the third party system, or the event collection apparatus in response to determining that i) at least one of the expected parameter types from first parameter types is not included in the second parameter types, ii) at least one of the corresponding values is not the same, or iii) both i and ii.

A non-transitory computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

• collecting, by an event collection apparatus and for each of a plurality of executions of an application on a device, an event log for the execution that identifies automated interactions with the application performed by an application testing agent on the device, each event log comprising, for each of multiple automated events that occurred on the device in response to one or more of the automated interactions, one or more first parameter types and, for each of the first parameter types, a corresponding first parameter value;
• storing, by the event collection apparatus and for each of the plurality of executions of the application on the device, the event log in one or more content storage devices;
• storing, by the event collection apparatus and for one or more of the plurality of executions of the application on the device and in the one or more content storage devices, event tracking data captured by a third party system, separate from an event verification apparatus, during the automated interaction with the application on the device by the application testing agent, the event tracking data comprising, for each of at least some of the multiple automated events that occurred on the device, one or more second parameter types and, for each of the second parameter types, a corresponding second parameter value;
• comparing, by the event verification apparatus and for some of the plurality of executions of the application, the first parameter types with the second parameter types to determine whether expected parameter types from the first parameter types are included in the second parameter types;
• comparing, by the event verification apparatus and for each of the expected parameter types that are included in the second parameter types, the corresponding first parameter values with the corresponding second parameter values for the same parameter type to determine whether the corresponding values are the same;
• and
• detecting, by the event verification apparatus, a software error in one or more of the application, the third party system, or the event collection apparatus in response to determining that i) at least one of the expected parameter types from first parameter types is not included in the second parameter types, ii) at least one of the corresponding values is not the same, or iii) both i and ii.