Systems and methods for extensions to alternative control of touch-based devices

An apparatus for multi-modal control of a touch-based device, the method comprising:

• circuitry configured to receive multi-modal control input data from one or more of voice commands, a game controller, a handheld remote, and physical gestures detected by a sensor, wherein the multi-modal control input data is provided by inputs of a user;
• circuitry configured to convert, based on a translation mapping between touch-based data and multi-modal based control data, the multi-modal control input data into corresponding translated input data which model signals generated from receipt of physical inputs recognizable by underlying applications of the touch-based device;
• and
• circuitry configured to provide the corresponding translated input data to the underlying applications of the touch-based device for control thereof, wherein the translated input data is utilized to control the underlying applications executed on the touch-based device which recognize the corresponding translated input data as the signals from the physical inputs even though no physical inputs were received by the touch-based device for the signals, and wherein the translation mapping between the touch-based data and the multi-modal based control data is different while menus of the underlying applications are displayed than for other display operations of the underlying applications.

A non-transitory computer-readable storage medium having computer readable code stored thereon for programming a computer, server, appliance, or device, each of which comprising a processor which, in response to the computer readable code, performs steps of

• receiving multi-modal control input data from one or more of voice commands, a game controller, a handheld remote, and physical gestures detected by a sensor, wherein the multi-modal control input data is provided by inputs of a user;
• converting, based on a translation mapping between touch-based data and multi-modal based control data, the multi-modal control input data into corresponding translated input data which model signals generated from receipt of physical inputs recognizable by underlying applications of the touch-based device;
• and
• providing the corresponding translated input data to the underlying applications of the touch-based device for control thereof, wherein the translated input data is utilized to control the underlying applications executed on the touch-based device which recognize the corresponding translated input data as the signals from the physical inputs even though no physical inputs were received by the touch-based device for the signals, and wherein the translation mapping between the touch-based data and the multi-modal based control data is different while menus of the underlying applications are displayed than for other display operations of the underlying applications.