Automatic translation between sign language and spoken language

A non-transitory machine readable medium comprising instructions to cause one or more processors to perform operations to translate a real-time conversation between sign language and spoken language, the operations including:

• obtain a continuous sign video that captures signs generated during the real-time conversation;
• create a list of a plurality of random video segments of the continuous sign video, the plurality of video segments including random start and end times during a duration of the continuous sign video;
• identify, via execution of a trained single sign language recognition model that is provided the continuous sign video and the list of random video segments, a plurality of candidate signs, each of the plurality of candidate signs identified based on features identified in respective subsets of the plurality of video segments, wherein at least two of the subsets of the plurality of video segments include one or more of the same video segments;
• associate, via the execution of the trained single sign language recognition model, a respective gloss to each of the plurality of candidate signs;
• group the glosses of the plurality of candidate signs that are identified using one or more of the same video segments to form a plurality of gloss groups;
• rank the glosses within each of the plurality of gloss groups;
• select one of the plurality of candidate signs as performed signs in each of the plurality of gloss groups based on the rank of the glosses in each of the plurality of gloss groups;
• and
• generate, via a model, at least one of visual, text, or audio data based on the performed signs.

A method to convert sign language to spoken language, the method comprising:

• obtaining a continuous sign video that captures signs generated during a real-time conversation for which translation between sign language and spoken language is occurring;
• creating a list of a plurality of random video segments of the continuous sign video, the plurality of video segments including random start and end times during a duration of the continuous sign video;
• identifying, via execution of a trained single sign language recognition model that is provided the continuous sign video and the list of random video segments, a plurality of candidate signs, each of the plurality of candidate signs identified based on features identified in respective subsets of the plurality of video segments, wherein at least two of the subsets of the plurality of video segments include one or more of the same video segments;
• associating, via the execution of the trained single sign language recognition model, a respective gloss each of the plurality of candidate signs;
• generating, via the execution of the trained single sign language recognition model, a respective confidence score for each of the respective glosses, the confidence score indicating a probability identified by the trained single sign language recognition model that the respective glosses are an accurate translation of the plurality of candidate signs;
• determining which of the plurality of candidate signs are identified using one or more of the same video segments;
• and
• selecting performed signs from the plurality of candidate signs based on the respective confidence scores and which of the plurality of candidate signs are identified using one or more of the same video segments, the performed signs used for the translation between sign language and spoken language during the real-time conversation.