Generating symbolic domain models from multimodal data

A system to generate a formal planning domain description, which comprises:

• a computer system comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to:
• receive a first text-based description of a domain in an AI environment, where said domain includes at least one action and at least one attribute associated therewith, said description being written in natural language;
• responsive to receiving said first text-based description of said domain, extract a first set of domain actions and a first set of associated action attributes;
• receive at least one audio-visual element depicting an aspect of said domain;
• responsive to receiving said at least one audio-visual element, generate a second text-based description of said domain and extract therefrom a second set of domain actions and a second set of associated action attributes;
• construct at least one finite state machine (FSM) corresponding to each of said extracted actions and said associated action attributes;
• convert said at least one FSM into a symbolic model that describes said domain, said symbolic model being written in a formal planning language.

A computer program product to generate a formal planning domain description, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to:

• receive, using said computer, a first text-based description of a domain in an AI environment, where said domain includes at least one action and at least one attribute associated therewith, said description being written in natural language;
• responsive to receiving said first text-based description of said domain, extract, using said computer, a first set of domain actions and a first set of associated action attributes;
• receive, using said computer, at least one audio-visual element depicting an aspect of said domain;
• responsive to receiving said at least one audio-visual element, generate, using said computer, a second text-based description of said domain and extract, using said computer, therefrom a second set of domain actions and a second set of associated action attributes;
• construct, using said computer, at least one finite state machine (FSM) corresponding to each of said extracted actions and said associated action attributes;
• convert, using said computer, said at least one FSM into a symbolic model that describes said domain, said symbolic model being written in a formal planning language.