Systems and methods for generating markup-language based expressions from multi-modal and unimodal inputs

A system comprising:

• a processor;
• and
• a computer-readable storage medium having instructions stored which, when executed by the processor, cause the processor to perform operations comprising:
• generating an input lattice representing input data having a first modality input and a second modality input;
• using a finite-state device to perform a function using the input lattice, the finite-state device having symbols based on markup-language semantics, wherein performing the function comprises mapping a first markup-language expression to a second markup-language expression using a transducer relating the first-markup-language expression to the second-markup-language expression based on a level of coincidence between the first modality input and the second modality input;
• generating a result finite-state device representing a result of the function, wherein the result finite-state device relates a set of input symbols to a set of output symbols;
• concatenating output symbols in the set of output symbols of the result finite-state device, to yield a markup-language expression;
• and
• outputting the markup-language expression.

A computer-readable storage device having instructions stored which, when executed by a computing device, cause the computing device to perform operations comprising:

• generating an input lattice representing input data having a first modality input and a second modality input;
• using a finite-state device to perform a function using the input lattice, the finite-state device having symbols based on markup-language semantics, wherein performing the function comprises mapping a first markup-language expression to a second markup-language expression using a transducer relating the first-markup-language expression to the second-markup-language expression based on a level of coincidence between the first modality input and the second modality input;
• generating a result finite-state device representing a result of the function, wherein the result finite-state device relates a set of input symbols to a set of output symbols;
• concatenating output symbols in the set of output symbols of the result finite-state device, to yield a markup-language expression;
• and
• outputting the markup-language expression.