Pinch gesture detection and recognition method, device and system

An electronic device comprising:

• a memory, a processor and a camera device, wherein the memory includes a pinch gesture detection and recognition program, when the pinch gesture detection and recognition program is executed by the processor, the following operations are implemented:
• acquiring, in real time, image data of each frame in a video to be detected;
• performing a hand location detection on the image data based on a pre-trained hand detection model, to determine a hand position of the image data;
• performing a skeleton point recognition at the hand position based on the pre-trained skeleton point recognition model, to determine a preset number of skeleton points at the hand position;
• and
• determining whether the hand is in a pinch gesture or not according to information of a distance between the skeleton points of preset fingers,
• wherein the determining the preset number of skeleton points at the hand position comprises:
• determining two-dimensional coordinate information of the preset number of skeleton points at the hand position based on the skeleton point recognition model;
• and
• acquiring three-dimensional position coordinate information corresponding to the two-dimensional coordinate information in real time according to the two-dimensional coordinate information and a stereoscopic triangulation imaging principle;
• wherein the three-dimensional position coordinate information is used for determining the information of the distance between the skeleton points of the preset fingers, and/or
• wherein when the pinch gesture detection and recognition program is executed by the processor, the following further operations are implemented;
• acquiring pinch gesture information of the image data of a preset number of continuous frames;
• and
• determining the hand in the video to be detected is in the pinch gesture in a case where the hands of the image data of the preset number of frames are all in pinch gestures.

A pinch gesture detection and recognition system, comprises a memory storing instructions and a processor in communication with the memory, wherein the processor is configured to execute the instructions to:

• acquire, in real time, image data of each frame in a video to be detected;
• perform a hand location detection on the image data based on a pre-trained hand detection model, to determine a hand position of the image data;
• perform a skeleton point recognition at the hand position based on the pre-trained skeleton point recognition model, to determine a preset number of skeleton points at the hand position;
• and
• determine whether a hand corresponding to the image data is in a pinch gesture or not according to information of a distance between the skeleton points of preset fingers,
• wherein the processor, when determine the preset number of skeleton points at the hand position, is further configured to:
• determine two-dimensional coordinate information of the preset number of skeleton points at the hand position based on the skeleton point recognition model;
• and
• acquire three-dimensional position coordinate information corresponding to the two-dimensional coordinate information in real time according to the two-dimensional coordinate information and a stereoscopic triangulation imaging principle;
• wherein the three-dimensional position coordinate information is used for determining the information of the distance between the skeleton points of the preset fingers, and/or
• wherein the processor is further configured to:
• acquire pinch gesture information of the image data of a preset number of continuous frames;
• and
• determine the hand in the video to be detected is in the pinch gesture in a case where the hands of the image data of the preset number of frames are all in pinch gestures.