System and methods with a multimodal interface for command and control (inputs) and monitoring (outputs) of unmanned systems

A multi-modal system for controlling an unmanned vehicle comprising:

• a human-to-machine interface including a plurality of selectable input modes, including:
• (i) at least one of a voice input, a hand-held controller, and a touch screen input, and
• (ii) at least one selectable human feedback device;
• an interface process responsive to data indicative of instructions to the unmanned vehicle input on selected ones of the input modes and operable to map the inputs to a data model that includes data indicating the instructions, the unmanned vehicle, and the selected input mode corresponding to the input data indicative of the instructions;
• and
• a ground control effects process operable to process the data model including a process to extract from the data model input context data including the data indicative of the instructions to the unmanned vehicle and data indicative of the selected input mode corresponding to the input data indicative of the instructions and to translate the instructions to control command data executable by the unmanned vehicle;
• wherein the process to translate further includes a process to determine acontextual data indicative of the instructions and to determine from the acontextual data the control command data executable by the unmanned vehicle.

A networked system for controlling an unmanned vehicle comprising nodes in networked communication, the nodes including:

• at least a first user node having a first human-to-machine interface including a plurality of selectable input modes, including at least one of a native input mode and at least one of a non-native input mode, and at least one selectable human feedback device, the first user node further including an interface process responsive to data indicative of instructions to the unmanned vehicle input on the selected ones of the input modes and operable to map the inputs to a data model that includes data indicating the instructions, the unmanned vehicle and the selected input mode;
• at least a second user node having a human-to-machine interface including at least one input mode and at least one feedback device, the second user node further including a second user node interface process responsive to data indicative of instructions to the unmanned vehicle input on the at least one input mode according to the data model;
• a system management node including a process operable to extract contextual data from the data model, the contextual data including the data indicative of instructions and the at least one input mode, to determine from the contextual data acontextual command data and to translate the acontextual command data to control data executable by the unmanned vehicle, the system management portal node further including a message router operable to route the control data to the unmanned vehicle,
• wherein the unmanned vehicle executes the instructions input at the first and second user node according to the data model.

A method for controlling an unmanned vehicle having an associated native input modality, the method comprising:

• selecting, via a multi-modal human interface, at least one non-native input modality from a plurality of human input modalities;
• in response to an input at the multi-modal human interface, establishing networked communication between the multi-modal human interface and the unmanned vehicle;
• receiving control commands for the unmanned vehicle at the multi-modal human interface via the at least one non-native input modality;
• mapping the received control commands to data indicative of the received control commands according to a data input model;
• from the data input model, translating the data indicative of the received commands to data indicative of commands executable by the unmanned vehicle, including the steps of:
• extracting from the data model context data indicating the received control commands and the non-native input modality,
• determining acontextual command data from the extracted context data, and
• translating the acontextual command data to corresponding data indicative of commands executable by the unmanned vehicle;
• and
• transmitting the data indicative of commands executable by the unmanned vehicle on a channel accessible to the unmanned vehicle.