Machine learning-based diagnostic classifier

A system for evaluating a user, the system comprising:

• a microphone;
• a camera positioned to capture an image of the user and configured to output video data;
• a memory containing machine readable medium comprising machine executable code having stored thereon instructions for performing a method of evaluating the user;
• and
• a control system coupled to the memory comprising one or more processors, the control system configured to execute the machine executable code to cause the control system to:
• record, by the camera, a set of test video data during a time window;
• record, by the microphone, a set of test audio data during the time window;
• assign a plurality of pixels to a face of the user in the video data;
• determine, based on the plurality of pixels, whether the face of the user is within a frame captured by the camera;
• in response to determining that the face of the user is within the frame captured by the camera, output video features associated with the user by processing the plurality of pixels;
• identify sounds representing a voice of the user and output audio features associated with the user by processing the audio data;
• process, using a neural network, the audio and video features, wherein the neural network was previously trained with training data in an unsupervised manner, the training data comprising audio and video data recorded from a plurality of individuals;
• and
• output an indication of whether the user has at least one of a plurality of characteristics based on the processed audio and video features.