Method and system for hand pose recognition, device and storage medium

A system for hand pose recognition, comprising:

• a data set module configured to capture a RGB image of a hand from a RGB camera and to capture a depth image of the hand from an active depth camera, so as to obtain a hand pose data set according to the RGB image and the depth image;
• an image preprocessing module configured to process the hand pose data set to obtain a 3D joint position, and to take the 3D joint position as a data set for training a software model;
• a feature extraction module configured to extract the RGB image by a feature extractor based on a depth neural network to obtain a feature map of a hand pose, a feature map extraction module comprising an inverted residual unit configured to:
• add an inverted residual module composed of an input layer, a bottleneck layer and an extension layer into the feature extractor;
• add an ACON activation function at an end of separable convolution before the inverted residual module;
• and learn degrees of linearity and nonlinearity of an activation function area obtained at a specific network location by setting a hyperparameter;
• and
• a feature recognition module configured to process the feature map according to an attention mechanism to obtain a global feature map of the hand pose, and to obtain a recognition result of the hand pose using the global feature map, the feature map recognition module comprising a result recognition unit configured to:
• obtain an estimated position heat map of a three-dimensional hand pose according to a processing result of the feature map;
• and adjust an average-pooled shape of the feature map so as to obtain position information of a palm heel node, and then determine a type of the hand to be a left hand or a right hand;
• wherein the feature map recognition module comprises an attention mechanism unit configured to:
• introduce a coordinate attention mechanism module into the feature extractor, and process the feature map according to the coordinate attention mechanism module to improve resolution and information completeness of the feature map;
• and
• introduce a multispectral attention mechanism module into the feature extractor, and estimate effectiveness of network feature extraction according to the multispectral attention mechanism module in the hand pose.