Gesture Vox

A computer system for converting sign language gestures into spoken audio output in real time, the computer system comprising:

• one or more processors;
• and
• one or more non-transitory computer-readable memories storing instructions that, when executed by the one or more processors, cause the computer system to perform a model-deployment preparation stage followed by a calibration-gated live inference stage, wherein the model-deployment preparation stage comprises:
• generating pre-processed training gesture image data from labeled sign-language gesture images by resizing the labeled sign-language gesture images to a uniform model-input size, normalizing pixel values of the labeled sign-language gesture images to a model-input range, applying noise-reduction processing to reduce background noise or non-gesture image information, and augmenting the labeled sign-language gesture images using at least one of rotation, flipping, or scaling;
• dividing the pre-processed training gesture image data into a training set, a validation set, and an independent test set;
• training a plurality of candidate convolutional neural-network models using the training set;
• evaluating the plurality of candidate convolutional neural-network models using the validation set;
• measuring a final accuracy of each of the plurality of candidate convolutional neural-network models using the independent test set;
• calculating precision, recall, and F1-score for each of the plurality of candidate convolutional neural-network models;
• selecting, after the precision, recall, and F1-score have been calculated, a candidate convolutional neural-network model having a highest final accuracy on the independent test set among the plurality of candidate convolutional neural-network models;
• and
• embedding the selected candidate convolutional neural-network model into a software application as a deployed convolutional neural network configured for real-time gesture-to-speech conversion;
• wherein the calibration-gated live inference stage comprises:
• guiding a user, through a user interface of the software application, through a setup sequence comprising camera calibration and an initial gesture-recognition test;
• only after completion of the setup sequence, receiving, from a camera input calibrated during the setup sequence, a live video feed comprising sequential image frames of the user performing sign language gestures;
• generating standardized live gesture image data from the sequential image frames by resizing the sequential image frames to the uniform model-input size, normalizing pixel values of the sequential image frames to the model-input range, and applying noise-reduction processing to reduce background noise or non-gesture image information before the standardized live gesture image data is provided to the deployed convolutional neural network;
• processing the standardized live gesture image data using the deployed convolutional neural network, wherein the deployed convolutional neural network comprises:
• an input layer configured to receive the standardized live gesture image data while preserving height, width, and color-channel information;
• one or more convolutional layers configured to apply convolutional filters to the standardized live gesture image data to generate feature maps representing gesture-related edges, textures, or image patterns;
• one or more rectified linear unit activation functions configured to introduce non-linearity into the feature maps;
• one or more pooling layers configured to reduce dimensionality of the feature maps;
• a flattening operation configured to convert the reduced-dimensionality feature maps into a one-dimensional vector;
• one or more fully connected layers configured to classify the one-dimensional vector into a plurality of predefined sign-language gesture classes;
• a softmax output layer configured to generate a probability value for each of the plurality of predefined sign-language gesture classes;
• selecting, as a predicted sign-language gesture, one of the plurality of predefined sign-language gesture classes having a highest probability value generated by the softmax output layer;
• and
• generating spoken audio output corresponding to the predicted sign-language gesture.