Method for framework-independent user input processing in human-machine interfaces

A method for hands-free user input in a human-machine interface (HMI), the method comprises following steps:

• receiving a first command from a user via input means;
• processing the received first command into a corresponding machine-understandable text command;
• transmitting the text command to an engine of the HMI for analysis;
• identifying an intended interaction of the user by analysing a context of the text command;
• identifying one or more actionable UI areas;
• matching the identified intended interaction of the user to a corresponding actionable user interface area of the identified one or more user interface areas;
• generating a simulated input event at the location of the actionable user interface area matched with the identified intended interaction of the user;
• and
• injecting the simulated input event via an operating-system input mechanism such that the simulated input event is delivered to the HMI/UI framework for execution on the HMI display.

A method for input-based interaction with user interface (UI) elements of an application output on the human-machine interface (HMI), the method comprises:

• receiving a first command from a user;
• identifying from at least one UI element source one or more candidate actionable UI elements corresponding to the received first command;
• highlighting, by rendering an overlay on top of the graphics buffer of the HMI, a most probable candidate actionable UI element;
• receiving a first evaluation input from the user comprising an acceptance or a rejection of the highlighted candidate actionable UI element;
• and
• upon receiving an acceptance, executing the interaction by injecting a simulated input event via a virtual input device of the operating system at coordinates corresponding to the highlighted candidate actionable UI element.