System and process for controlling electronic components in a ubiquitous computing environment using multimodal integration

A computer-implemented process for controlling a user-selected electronic component within an environment using a pointing device, comprising using a computer to perform the following process actions:

• computing the similarity between an input sequence of sensor values output by the pointing device and recorded over a prescribed period of time and at least one stored prototype sequence, wherein each prototype sequence represents the sequence of said sensor values that are generated if the user performs a unique gesture representing a different control action for the selected electronic component using the pointing device;
• determining if the computed similarity between the input sequence and any prototype sequence exceeds a prescribed similarity threshold;
• and
• whenever it is determined that one of the computed similarities exceeds the similarity threshold, implementing a command represented by the gesture.

A system for controlling a user-selected electronic component within an environment using a pointing device, comprising:

• a general purpose computing device;
• a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,
• input orientation messages transmitted by the pointing device, said orientation messages comprising orientation sensor readings generated by orientation sensors of the pointing device,
• generate and store at least one prototype sequence in a training phase, each prototype sequence comprising prescribed one or ones of the orientation readings that are generated by the pointing device during the time a user moves the pointing device in a gesture representing a particular command for controlling the selected electronic component, wherein each prototype sequence is associated with a different gesture and a different command,
• record said prescribed one or ones of the orientation sensor readings from each orientation message inputted for a prescribed period of time to create an input sequence of sensor readings,
• for each stored prototype sequence, compute a similarity indicator between the input sequence and each prototype sequence under consideration, wherein the similarity indicator is a measure the similarity between sequences,
• identify the largest of the computed similarity indicators,
• determine if identified largest computed similarity indicator exceeds a prescribed similarity threshold,
• whenever the identified similarity indicator exceeds the prescribed similarity threshold, designate that the user has performed the gesture corresponding to the prototype sequence associated with that similarity indicator, and
• implement the command represented by the gesture that the user was designated to have performed.

A computer-readable medium having computer-executable instructions for controlling a user-selected electronic component within an environment using a pointing device, said computer-executable instructions comprising:

• inputting orientation messages transmitted by the pointing device, said orientation messages comprising orientation sensor readings generated by orientation sensors of the pointing device;
• generating and storing at least one prototype sequence in a training phase, each prototype sequence comprising a prescribed one or ones of the orientation readings that are generated by the pointing device during the time a user moves the pointing device in a gesture representing a particular command for controlling the selected electronic component, and wherein each prototype sequence is associated with a different gesture and represents a different command;
• recording said prescribed one or ones of the orientation sensor readings from each orientation message inputted for a prescribed period of time to create an input sequence of sensor readings;
• for each stored prototype sequence,
• computing a match score between correspondingly ordered time steps of the input sequence and each of a plurality of versions of the prototype sequence, wherein each time step refers to the recording of said prescribed one or ones of the orientation sensor readings from a single orientation message, and wherein said match score is a measure the similarity between time steps, and wherein each of the plurality of versions of the prototype sequence are formed by applying a different scaling factor and warping factor to the prototype sequence from a prescribed set of scaling factors for scaling the prototype sequence in amplitude and a prescribed set of warping factors for warping the prototype sequence in time,
• respectively averaging the match scores computed between correspondingly ordered time steps of the input sequence and each of a plurality of versions of the prototype sequence to produce an averaged match score for each,
• identifying the largest of the averaged match scores, and
• designating the largest of the averaged match scores as a similarity indicator for the prototype sequence under consideration;
• identifying the largest of the computed similarity indicators;
• determining if the identified largest similarity indicator exceed a prescribed similarity threshold;
• whenever the identified similarity indicator exceeds the prescribed similarity threshold, designating that the user has performed the gesture corresponding to the prototype sequence associated with the similarity indicator exceeding the prescribed similarity threshold;
• and
• implementing the command represented by the gesture that the user was designated to have performed.

A computer-implemented process for controlling a user-selected electronic component within an environment using a pointing device, comprising using a computer to perform the following process actions:

• inputting orientation messages transmitted by the pointing device, said orientation messages comprising orientation sensor readings generated by orientation sensors of the pointing device;
• recording a prescribed one or ones of the pointing device sensor outputs taken from an orientation message inputted whenever a user indicates performing a gesture representing a control action for the selected electronic component;
• for each gesture threshold definition assigned to the selected electronic component, determining whether the threshold if just one, or all the thresholds if more than one, of the gesture threshold definition under consideration are exceeded by the recorded sensor output associated with the same sensor output as the threshold;
• whenever it is determined that the threshold if just one, or all the thresholds if more than one, of one of the gesture threshold definitions are exceeded by the recorded sensor output associated with the same sensor output, designating that the user has performed the gesture associated with that gesture threshold definition;
• and
• implementing the command represented by the gesture that the user was designated to have performed.

A computer-readable medium having computer-executable instructions for controlling a user-selected electronic component within an environment using a pointing device, said computer-executable instructions comprising:

• (a) inputting orientation messages transmitted by the pointing device, said orientation messages comprising orientation sensor readings generated by orientation sensors of the pointing device;
• (b) waiting for an orientation message to be received;
• (c) whenever an orientation message is received, determining whether the switch state indicates that the pointing device's switch has been activated by the user to indicate that a control gesture is being performed;
• (d) whenever it is determined that the switch state does not indicate that the pointing devices switch is activated, repeat actions (b) and (c);
• (e) whenever it is determined that the switch state does indicate that the pointing devices switch is activated, recording a prescribed one or ones of the pointing device sensor outputs taken from the last received orientation message;
• (k) for each gesture threshold definition assigned to the selected electronic component, determining if the threshold if just one, or all the thresholds if more than one, of the gesture threshold definition under consideration are exceeded by the recorded sensor output associated with the same sensor output as the threshold;
• (l) whenever it is determined that the threshold if just one, or all the thresholds if more than one, of one of the gesture threshold definitions are exceeded by the recorded sensor output associated with the same sensor output, designating that the user has performed the gesture associated with that gesture threshold definition;
• and
• (m) implementing the command represented by the gesture that the user was designated to have performed.

A system for controlling electronic components within an environment using a pointing device, comprising:

• a pointing device comprising a transceiver and orientation sensors, wherein the outputs of the sensors are periodically packaged as orientation messages and transmitted using the transceiver;
• a base station comprising a transceiver which receives orientation messages transmitted by the pointing device;
• a pair of imaging devices each of which is located so as to capture images of the environment from different viewpoints;
• a computing device which is in communication with the base station and the imaging devices so as to receive orientation messages forwarded to it by the base station and images captured by the imaging devices, and which,
• computes the orientation and location of the pointer from the received orientation message and captured images,
• determines if the pointing device is pointing at an object in the environment which corresponds to, or is associated with, an electronic component that is controllable by the computing device using the orientation and location of the pointing device, and if so selects the electronic component,
• affects the selected electronic component in accordance with a command received from the pointing device.