Multi-view human action recognition method based on hypergraph learning

A multi-view human action recognition method based on hypergraph learning, comprising acquiring video data from P views, wherein the method further comprises the following steps:

• step 1:
• pre-processing the video data, comprising:
• segmenting, for each of the P views, the video data into N frames, and extracting joint information of a human body from each frame;
• step 2:
• constructing spatial hypergraphs based on joint information, comprising:
• constructing N spatial hypergraphs, each corresponding to one frame of the video data, by using joints of the human body as vertices,
• dividing the human body into a plurality of body parts including a torso, a left hand, a right hand, a left leg, and a right leg, and
• for each body part, constructing a hyperedge that connects joints of the body part from different views at a same frame, such that each spatial hypergraph represents spatial correlations of joints across the P views;
• step 3:
• constructing temporal hypergraphs based on the joint information, comprising:
• constructing P temporal hypergraphs, each corresponding to one of the P views, by using the joints of the human body as vertices, and
• for each body part, constructing a hyperedge that connects joints of the body part from different frames within a same view, such that each temporal hypergraph represents temporal correlations of joints across the N frames;
• step 4:
• performing feature learning of the spatial hypergraphs and the temporal hypergraphs using hypergraph neural networks, comprising:
• performing feature learning of the spatial hypergraphs using a spatial hypergraph neural network to obtain spatial hypergraph features, and
• performing feature learning of the temporal hypergraphs using a temporal hypergraph neural network to obtain temporal hypergraph features;
• and
• step 5:
• extracting higher order information represented by the spatial hypergraphs and the temporal hypergraphs, and performing action recognition of human actions, comprising:
• fusing the spatial hypergraph features and the temporal hypergraph features to obtain fused features, and
• performing the action recognition based on the fused features.