Extend the game controller functionality with virtual buttons using hand tracking

A method for verifying an input provided at a controller, comprising:

• detecting a finger gesture provided by a user on a surface of the controller, the finger gesture used to define the input for an interactive application selected for interaction by the user;
• receiving multi-modal data collected by tracking the finger gesture on the controller, the multi-modal data including modal data corresponding to different modes captured for the finger gesture using a plurality of sensors and components associated with the controller;
• assigning a weight to modal data for each mode included in the multi-modal data and using the weight assigned to each modal data in computing a cumulative weight of the multi-modal data captured for the finger gesture;
• generating an ensemble model using the multi-modal data received from the plurality of sensors and components, the ensemble model continuously trained using additional multi-modal data collected over time to generate different outputs, the ensemble model trained in accordance to training rules defined for different finger gestures using machine learning algorithm;
• and
• identifying an output from the ensemble model for the finger gesture detected on the surface of the controller, the output identified using the cumulative weight computed for the multi-modal data of the finger gesture, the output is interpreted to define the input for the interactive application based on the finger gesture detected at the controller,
• wherein operations of the method are performed by a processor of a server.