System and method for enhanced gesture-based interaction

A gesture-based interactive computerised system comprising:

• a three-dimensional imaging system configured to capture images within its frustum, the captured images being processed to detect and track over time at least three-dimensional information relating to at least one part of a user;
• a gesture recognition system configured to recognise gestures performed by the at least one part of the user;
• a user-operated control device configured to allow free movement of hand and fingers at the same time in the gesture recognition system the user-operated control device comprising:
• a housing;
• at least one sensing module including at least one control means mounted in the housing, each control means being capable of generating a control signal for the computerised system when operated;
• a feedback module mounted in the housing for providing contextual feedback to the user in accordance with an application operating on the computerised system;
• a communication module located within the housing and through which each control signal is transmitted to, and each feedback signal is received from, the computerised system;
• and a controller module mounted in the housing, the controller controlling the sensing module, the communication module and the feedback module in accordance with each generated control signal;
• and
• a computerised system associated with the three-dimensional imaging system and the gesture recognition system, the computerised system configured to process images captured by the three-dimensional imaging system and gestures recognised by the gesture recognition system and configured to use data from the user-operated control device in conjunction with the gestures recognised by the gesture recognition system to resolve ambiguities in three-dimensional information relating to at least one part of a user measured by the three-dimensional imaging system in order to control the gesture-based interactive computerised system.

A non-transitory computer readable medium storing a program causing a computer to execute a method of controlling a gesture-based interactive computerised system having a three-dimensional imaging system, a user-operated control device, a gesture recognition system, and a computerised system associated with the three-dimensional imaging system and the gesture recognition system, the user-operated control device being associated with at least one part of a user and being configured to allow free movement of hand and fingers at the same time in the gesture recognition system and comprising a housing;

• at least one sensing module including at least one control means mounted in the housing, each control means being capable of generating a control signal for the computerised system when operated;
• a feedback module mounted in the housing for providing contextual feedback to the user in accordance with an application operating on the computerised system;
• a communication module located within the housing and through which each control signal is transmitted to, and each feedback signal is received from, the computerised system;
• and a controller module mounted in the housing, the controller module controlling each sensing module, the communication module and the feedback module in accordance with each generated control signal, the three-dimensional imaging system including a three-dimensional camera having a frustum, the method comprising the steps of:
• a) capturing images using the three-dimensional imaging system;
• b) processing the captured images in the computerised system to recognise and track at least the three-dimensional information relating to the at least one part of the user;
• c) recognising gestures performed by the at least one part of the user using the gesture recognition system;
• d) generating control signals from the user-operated control device which include at least three-dimensional position information relating to the at least one part of the user with which the user-operated control device is associated;
• and
• e) using the three-dimensional position information provided by the user-operated control device to resolve ambiguities in the three-dimensional information relating to the at least one part of the user measured by the three-dimensional imaging system in order to control the gesture-based interactive computerised system.