System and method of selecting capabilities of artificial intelligence productivity tool-enablable software application via a multimodal intent hierarchy for responsive application to a multimodal user-query input

A method of executing computer readable code instructions of an on the box (OTB) artificial intelligence (AI) productivity tool at an information handling system to respond to a multimodal user-query input comprising:

• storing, in a capabilities database memory, multimodal capability intent values from textual and non-textual natural language descriptions of capabilities of AI productivity tool-enablable software applications executable on the information handling system in a capabilities decision tree with each capability stored under a plurality of branches as a capability node grouped under a branch of the capabilities decision tree according to logical topics in hierarchical parent-child relationships, wherein metadata for each capability node identifies a child capability node or parent capability node of the capability node;
• executing machine-readable program code instructions, via a hardware processor, of a textual embedding algorithm and a nontextual embedding algorithm to generate a multimodal query input intent value from a multimodal user query input received via text and an image requesting a response by one of the plurality of AI productivity tool-enableable software applications;
• executing machine-readable program code instructions, via the hardware processor, to perform a cosine semantic similarity search comparing the multimodal capability intent values of a plurality of parent capability nodes in a first level of the capabilities decision tree to the multimodal query intent value to determine a select branch for further semantic similarity comparison;
• identifying a best match childless end-of-branch capability node in the select branch of the capabilities decision tree having a highest cosine semantic similarity search score with the multimodal query input intent value;
• and
• executing machine-readable program code instructions for a best match capability of the best match childless end-of-branch capability node with a first AI productivity tool-enableable software application in response to the multimodal user query input.

An information handling system executing computer readable code instructions for an on the box (OTB) artificial intelligence (AI) productivity tool comprising:

• a capabilities database memory to store multimodal capability intent values from textual and non-textual descriptions of capabilities of AI productivity tool-enablable software applications executable on the information handling system in a capabilities decision tree with each capability stored under a plurality of branches as a capability node grouped under a branch of the capabilities decision tree according to logical topics in hierarchical parent-child relationships;
• a hardware processor executing machine-readable program code instructions to generate a multimodal query input intent value from a multimodal user query input in at least a first portion by execution of code instructions for a text embedding algorithm performed on text or speech converted to text within the multimodal user-query input, where the multimodal user query input requests a response by one of the plurality of AI productivity tool-enableable software applications;
• the hardware processor to further generate the multimodal query input intent value from the multimodal user query input in at least a second portion by execution of code instructions for a visual transformer non-textual embedding algorithm performed on an image within the multimodal user-query input;
• the hardware processor to perform a cosine semantic similarity search comparing the multimodal capability intent values of a plurality of parent capability nodes in a first level of the capabilities decision tree to the multimodal query intent value to determine a select branch for further semantic similarity comparison;
• the hardware processor to identify a best match childless end-of-branch capability node in the select branch of the capabilities decision tree having a highest cosine semantic similarity search score with the multimodal query input intent value;
• and
• the hardware processor executing machine-readable program code instructions for a best match capability of the best match childless end-of-branch capability node with a first AI productivity tool-enableable software application in response to the multimodal user query input.