Back to results
US 12,518,653 B2Granted
Realtime AI sign language recognition
AssigneeSign-Speak Inc.
InventorNikolas Anthony Kelly
Life of this patent
Estimating the term…
Abstract
A real time sign language recognition method that allows Deaf and Hard of Hearing individuals to sign into any apparatus with a camera to extract target information (such as a translation in a target language) is proposed.
Claims20
A method, comprising:
- capturing a sequence of images;
- for an image within the sequence of images, detecting pose information by:
- applying a first pose network to detect a body pose configuration in the image;
- applying a second pose network to detect a hand pose configuration in the image;
- generating a feature vector including the body pose configuration and the hand pose configuration;
- and
- converting the feature vector into a flattened feature vector having a predefined size;
- generating a feature queue by collecting the flattened feature vector for the image of the sequence of images;
- and
- converting the feature queue into a target language output.
7 more elements
A method, comprising:
- detecting hand pose information from a sequence of image frames;
- converting the hand pose information into a flattened feature vector;
- normalizing the flattened feature vector into a resultant feature vector;
- applying a convolutional neural network (CNN) to split the resultant feature vector into a plurality of individual regions by highlighting sign transition periods;
- applying the CNN to output a respective flag corresponding to each individual region of the plurality of individual regions, wherein the respective flag indicates whether the corresponding individual region is an intrasign region, intersign region, or a nonsigning region, and wherein the intrasign and intersign regions correspond to signing regions of the plurality of individual regions;
- processing the signing regions of the plurality of individual regions into a sign language string based on language information in one or more databases;
- and
- translating the sign language string into a target language output.
5 more elements
A device, comprising:
- a camera configured to capture a sequence of image frames;
- a computing device coupled to the camera, the computing device comprising:
- a processor;
- and
- a memory, wherein the memory contains instructions stored thereon that when executed by the processor cause the processor to:
- detect hand pose information from the sequence of image frames;
- convert the hand pose information into a flattened feature vector;
- normalize the flattened feature vector into a resultant feature vector;
- apply a convolutional neural network (CNN) to split the resultant feature vector into a plurality of individual regions by highlighting sign transition periods;
- apply the CNN to output a respective flag corresponding to each individual region of the plurality of individual regions, wherein the respective flag indicates whether the corresponding individual region is an intrasign region, intersign region, or a nonsigning region, and wherein the intrasign and intersign regions correspond to signing regions of the plurality of individual regions;
- process the signing regions of the plurality of individual regions into a sign language string based on language information in one or more databases;
- and
- translate the sign language string into a target language output.
10 more elements
Dependencies are inferred from the claim text.