METHOD FOR COUNTING SUCKlING PIGLETS BASED ON SELF-ATTENTION SPATIOTEMPORAL FEATURE FUSION

A method for counting suckling piglets based on self-attention spatiotemporal feature fusion, comprising the following steps:

• S1, collecting a video containing a sow and a plurality of piglets;
• S2, establishing a CenterNet network and a FlowNet network;
• detecting a video clip of the sow in a side-lying posture in the video by using the CenterNet network to extract a key frame sequence;
• and detecting an extended zone of a side-lying sow bounding box as a region of interest, and using the FlowNet network to acquire an optical flow image sequence corresponding to the key frame sequence;
• S3, extracting spatial features and temporal features respectively from an RGB image in the key frame sequence and the optical flow sequence by using a two-stream CNN;
• transforming the extracted spatial features and temporal features into a spatial feature vector and a temporal feature vector by using a spatial transformation vector and a temporal transformation vector;
• and summing the spatial feature vector and the temporal feature vector to obtain a fused feature vector;
• S4, inputting the fused feature vector into a fusion transformer to output a vector as a query of a temporal transformer and a spatial transformer;
• inputting the temporal feature vector and the spatial feature vector respectively into the temporal transformer and the spatial transformer;
• summing and normalizing the outputted vector respectively with the temporal feature vector and the spatial feature vector;
• then inputting the vector into a feed-forward network;
• summing and normalizing the output of the feed-forward network respectively with the temporal feature vector and the spatial feature vector again to output a column vector and a row vector respectively;
• and further establishing a self-attention matrix;
• S5, concatenating the spatial features and the temporal features outputted from the two-stream CNN to obtain fused spatiotemporal features, and performing an element-wise product for the fused spatiotemporal features and the self-attention matrix to obtain a self-attention spatiotemporal feature map;
• and
• S6, inputting the self-attention spatiotemporal feature map into are regression branch of a suckling piglet center point to output location estimation of the suckling piglets, and inputting the self-attention spatiotemporal feature map into a regression branch of the number of suckling piglets to output quantity estimation of the suckling piglets.