Flight pushback state monitoring method based on multi-modal data fusion

A flight pushback state monitoring method based on multi-modal data fusion, comprising the following steps of:

• step 1:
• constructing a control intention recognition rule, and extracting a control intention from a control instruction text;
• step 2:
• constructing a flight intention recognition model, and extracting the flight intention from a surface monitoring video;
• step 3:
• designing an intention alignment rule to align control intentions and flight intentions at different times;
• and
• step 4:
• judging intention matching, and fusing the control intention and the flight intention aligned to judge whether the control intention and the flight intention aligned are inconsistent;
• and when the control intention and the flight intention aligned are inconsistent, triggering an alarm;
• in step 2, the constructing the flight intention recognition model comprises:
• monitoring a surface stand by using a high resolution camera, collecting stand monitoring video data, dividing images by region, forming monitoring images for single stand, and labeling the monitoring images to construct a test data set;
• constructing an image classification model based on a convolutional neural network, wherein the image classification model is capable of recognizing three flight state types:
• the first type is that the flight is on the stand, that is, in an unmoving state;
• the second type is that a part of the flight is on the stand, that is, in an uncertain state;
• and the third type is that the flight is not on the stand, that is, in a moving state;
• and training the image classification model by using the test data set.