AUGEMENTED VIDEO ANALYTICS FOR TESTING INTERNET OF THINGS (IoT) DEVICES

A computer program product, comprising:

• a computer-readable storage device;
• and
• a computer-readable program code stored in the computer-readable storage device, the computer-readable program code containing instructions that are executed by a central processing unit (CPU) of a computer system to implement a method of testing a device in an Internet of Things, the method comprising the steps of:
• the computer system receiving first data items indicating visually perceptible physical output of the device during a testing of the device, the first data items being captured on a video by a video camera surveilling the device during the testing, and the video camera being coupled to the computer system;
• the computer system receiving second data items indicating non-visually perceptible physical output of the device during the testing;
• the computer system receiving an event log from a beacon continuously streamed by the device, the event log generated by an instrumented application being executed in the device;
• the computer system receiving indications of interactions of one or more humans with the device that occur during the testing, the interactions captured by (1) a wearable device operated by a human during the testing or (2) a sensor that is coupled to the device and that senses the interactions;
• based on the first data items indicating visually perceptible physical output, the second data items indicating non-visually perceptible physical output, the event log, and the indications of the interactions, the computer system overlaying the first data items, the second data items, entries included in the event log, and the indications of the interactions onto respective frames of the video that include the device, so that timings of the frames are correlated with timestamps of (1) the first data items, (2) the second data items, (3) the entries from the event log, and (4) the indications of the interactions;
• based on the video having the overlaid first data items, second data items, entries included in the event log, and the indications of the interactions, the computer system detecting performance, functionality, and usability issues and generating respective markers of the issues;
• the computer system indexing the video with the markers by overlaying the markers on a timeline of the video;
• and
• in response to a selection of one of the markers by a user which indicates an issue included in the issues, the computer system locating and displaying a frame of the video that depicts the issue.

A computer system comprising:

• a central processing unit (CPU);
• a memory coupled to the CPU;
• and
• a computer readable storage device coupled to the CPU, the storage device containing instructions that are executed by the CPU via the memory to implement a method of testing a device in an Internet of Things, the method comprising the steps of:
• the computer system receiving first data items indicating visually perceptible physical output of the device during a testing of the device, the first data items being captured on a video by a video camera surveilling the device during the testing, and the video camera being coupled to the computer system;
• the computer system receiving second data items indicating non-visually perceptible physical output of the device during the testing;
• the computer system receiving an event log from a beacon continuously streamed by the device, the event log generated by an instrumented application being executed in the device;
• the computer system receiving indications of interactions of one or more humans with the device that occur during the testing, the interactions captured by (1) a wearable device operated by a human during the testing or (2) a sensor that is coupled to the device and that senses the interactions;
• based on the first data items indicating visually perceptible physical output, the second data items indicating non-visually perceptible physical output, the event log, and the indications of the interactions, the computer system overlaying the first data items, the second data items, entries included in the event log, and the indications of the interactions onto respective frames of the video that include the device, so that timings of the frames are correlated with timestamps of (1) the first data items, (2) the second data items, (3) the entries from the event log, and (4) the indications of the interactions;
• based on the video having the overlaid first data items, second data items, entries included in the event log, and the indications of the interactions, the computer system detecting performance, functionality, and usability issues and generating respective markers of the issues;
• the computer system indexing the video with the markers by overlaying the markers on a timeline of the video;
• and
• in response to a selection of one of the markers by a user which indicates an issue included in the issues, the computer system locating and displaying a frame of the video that depicts the issue.