Systems and methods of multimodal interaction-based e-commerce

A computer-implemented method for performing multimodal fusion of computer interactions of a current user to improve productivity and interaction response time of the computer interactions for e-commerce, comprising:

• monitoring, by a computer comprising a processor and a memory, a focus prompt of the current user;
• detecting, by the computer, a response of the current user corresponding to a displayed item;
• collecting, by the computer, one or more attributes of the displayed item at different levels and track a focus and an emotional state of the current user;
• calculating, by the computer, an action-based similarity to generate a recommendation score of action similarity;
• applying, by the computer, collaborative filtering on a user-user similarity;
• identifying, by the computer, a most similar user to the current user based on the applied collaborative filtering to generate a user-user similarity;
• combining, by the computer, the recommendation score with the user-user similarity and superimpose a response classifier to generate a combined adaptive weighting scheme;
• identifying, by the computer, one or more top suggested actions the current user is expected to perform on the computer based on the combined adaptive weighting scheme to improve productivity and interaction response time of the current user with the computer;
• and
• recommending, by the computer, an item or service on the computer to the current user based on the one or more top suggested actions.

A non-transitory computer-readable medium embodied with software for performing multimodal fusion of computer interactions of a current user to improve productivity and interaction response time of the computer interactions for e-commerce, the software when executed is configured to:

• monitor, by a computer comprising a processor and a memory, a focus prompt of the current user;
• detect a response of the current user corresponding to a displayed item;
• collect one or more attributes of the displayed item at different levels and track a focus and an emotional state of the current user;
• calculate an action-based similarity to generate a recommendation score of action similarity;
• apply collaborative filtering on a user-user similarity;
• identify a most similar user to the current user based on the applied collaborative filtering to generate a user-user similarity;
• combine the recommendation score with the user-user similarity and superimpose a response classifier to generate a combined adaptive weighting scheme;
• identify one or more top suggested actions the current user is expected to perform on the computer based on the combined adaptive weighting scheme to improve productivity and interaction response time of the current user with the computer;
• and
• recommend an item or service on the computer to the current user based on the one or more top suggested actions.