Temporal fusion of multimodal data from multiple data acquisition systems to automatically recognize and classify an action

A method of using a multimodal sensing system to automatically recognize an action, the method comprising:

• operating a video camera to capture a sequence of digital images;
• operating at least one sensor device to capture other sensed data;
• and
• by a processor, executing programming instructions that cause the processing device to:
• receive, from the video camera, the sequence of digital images, wherein each of the digital images is associated with a time stamp,
• receive, from the at least one sensor device, the other sensed data, wherein each item of other sensed data is associated with a time stamp,
• extract a plurality of video features from the sequence of digital images so that each extracted image feature is associated with a time period,
• extract a plurality of other features from the other sensed data so that each extracted other feature is associated with the time period,
• use a multi-layer long-short-term memory network to fuse a group of the extracted video features having time stamps corresponding to the time period and a group of the extracted other features having time stamps corresponding to the time period to create a fused feature representation, wherein the multi-layer long-short-term memory network comprises:
• a first set of layers, wherein each layer in the first set is configured to independently operate on a single data modality, and wherein the first set of layers is configured to generate a set of per-modality features;
• and
• a second set of layers configured to receive the per-modality features from the first set of layers and generate the fused feature representation as a set of fused features across data modalities,
• analyze the fused feature representation to identify a class that applies to both the extracted video features and the extracted other features,
• access a data store of classes and actions to identify an action that is associated with the class, and
• save the identified action to a computer readable memory device.

A multimodal action classification system, comprising:

• a video camera;
• at least one sensor device that comprises one or more of the following:
• a microphone, a motion sensor, a positional sensor, or a temperature sensor;
• a processor;
• a data store of classes and actions;
• and
• a computer-readable memory device containing programming instructions that are configured to, when executed, cause the processor to:
• receive, from the video camera, a sequence of digital images, wherein each of the digital images is associated with a time stamp,
• receive, from the at least one sensor device, other sensed data, wherein each item of other sensed data is associated with a time stamp,
• extract a plurality of video features from the sequence of digital images so that each extracted image feature is associated with a time period,
• extract a plurality of other features from the other sensed data so that each extracted other feature is associated with the time period,
• use a long-short-term memory network having a first hidden layer and a second hidden layer to fuse a group of the extracted video features having time stamps corresponding to the time period and a group of the extracted other features having time stamps corresponding to the time period to create a fused feature representation so that:
• the first hidden layer independently identifies temporal trends in the extracted video features and the extracted motion features;
• and
• the second hidden layer uses the identified temporal trends to identify temporal correlations across modalities and performs the fusion,
• analyze the fused feature representation to identify a class that applies to both the extracted video features and the extracted other features,
• access the data store of classes and actions to identify an action that is associated with the class, and
• save the identified action to a computer readable memory device.

A method of using a multimodal sensing system to automatically recognize an action, the method comprising:

• operating a video camera to capture a sequence of digital images;
• operating at least one sensor device to capture other sensed data;
• and
• by a processor, executing programming instructions that cause the processing device to:
• receive, from the video camera, the sequence of digital images, wherein each of the digital images is associated with a time stamp,
• receive, from the at least one sensor device, the other sensed data, wherein each item of other sensed data is associated with a time stamp,
• extract a plurality of video features from the sequence of digital images so that each extracted image feature is associated with a time period,
• extract a plurality of other features from the other sensed data so that each extracted other feature is associated with the time period,
• using a long-short-term memory network having a first hidden layer and a second hidden layer to fuse a group of the extracted video features having time stamps corresponding to the time period and a group of the extracted other features having time stamps corresponding to the time period to create a fused feature representation so that:
• the first hidden layer independently identifies temporal trends in the extracted video features and the extracted motion features;
• and
• the second hidden layer uses the identified temporal trends to identify temporal correlations across modalities and performs the fusion,
• analyze the fused feature representation to identify a class that applies to both the extracted video features and the extracted other features,
• access a data store of classes and actions to identify an action that is associated with the class, and
• save the identified action to a computer readable memory device.