Realtime ai sign language recognition

A method, comprising:

• generating, for each image in a sequence of images, a feature vector by:
• obtaining features corresponding to at least one of body, hand, or signing information from the image in the sequence of images;
• and
• forming the feature vector based on the features;
• processing the feature vectors using a machine learning model to generate a sequence prediction;
• and
• decoding the sequence prediction to generate an output corresponding to the sequence of images.

A system, comprising:

• one or more processors;
• and
• a memory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
• generating, for each image in a sequence of images, a feature vector by:
• obtaining features corresponding to at least one of body, hand, or signing information from the image in the sequence of images;
• and
• forming the feature vector based on the features;
• processing the feature vectors using a machine learning model to generate a sequence prediction;
• and
• decoding the sequence prediction to generate an output corresponding to the sequence of images.

A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

• detecting pose information for an image within a sequence of images, wherein detecting the pose information comprises:
• applying a first pose network to detect a body pose configuration in the image;
• applying a second pose network to detect a hand pose configuration in the image;
• generating a feature vector including the body pose configuration and the hand pose configuration;
• and
• converting the feature vector into a flattened feature vector having a predefined size;
• generating a feature queue by collecting the flattened feature vector for the image of the sequence of images;
• and
• converting the feature queue into a target language output.