Multi-modal interaction for people with disability

A system comprising:

• one or more processors;
• and
• memory storing thereon instructions that, as a result of being executed by the one or more processors, cause the system to perform operations comprising:
• detecting one or more interactions performed by a user, wherein the one or more interactions are associated with a particular application, and wherein the one or more interactions are performed using a first input mode;
• processing the one or more interactions using an intent-discovery machine-learning model to extract an intent associated with the one or more interactions, wherein the intent identifies one or more intended inputs for the application and an expected output to be triggered in response to the one or more intended inputs;
• determining a context based on the intent, wherein the context identifies a previous session of interactions that store previously used input modes, user ability data, and interaction progress associated with the previous session, and wherein the previous session is associated with the particular application;
• and
• resuming the previous session of interactions in accordance with the context and the intent, wherein resuming the previous session of interactions includes using a second input mode to perform subsequent interactions, and wherein the second input mode is different from the first input mode.

A non-transitory, computer-readable storage medium storing thereon executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to perform operations comprising:

• detecting one or more interactions performed by a user, wherein the one or more interactions are associated with a particular application, and wherein the one or more interactions are performed using a first input mode;
• processing the one or more interactions using an intent-discovery machine-learning model to extract an intent associated with the one or more interactions, wherein the intent identifies one or more intended inputs for the application and an expected output to be triggered in response to the one or more intended inputs;
• determining a context based on the intent, wherein the context identifies a previous session of interactions that store previously used input modes, user ability data, and interaction progress associated with the previous session, and wherein the previous session is associated with the particular application;
• and
• resuming the previous session of interactions in accordance with the context and the intent, wherein resuming the previous session of interactions includes using a second input mode to perform subsequent interactions, and wherein the second input mode is different from the first input mode.