Method and system for processing a task with robustness to missing input information

A non-transitory computer-readable storage medium for storing computer-executable instructions which, when executed, cause a processing device to perform a method for processing a task, the method comprising:

• providing at least one numeric value to be used for processing a task,
• receiving information captured from more than one modality,
• for each of the more than one modality, following an initial training performed in accordance with a processing task to be performed, generating a feature map corresponding to the received information captured from the modality by applying at least one transformation corresponding to the modality on the information captured from the modality, wherein each of the feature maps are generated independently of each other;
• wherein the generating of each of the more than one corresponding feature map is performed by applying a given corresponding transformation on a given corresponding modality, wherein the more than one corresponding transformation is generated following the initial training performed in accordance with the processing task to be performed;
• in accordance with the processing task to be performed:
• selecting and combining the generated feature maps in accordance with at least one combining operation to generate at least one corresponding combined feature map by, for each generated feature map, reducing numeric values of the generated feature map, reducing numeric values of the generated feature map to a single numeric value of the at least one corresponding combined feature map;
• generating at least one final feature map by applying each of at least one final transformation on at least one of the at least one corresponding combined feature map;
• wherein the at least one final feature map is generated following the initial training performed in accordance with the task to be performed, and
• processing the generated at least one final feature map to provide a generated vector of at least one numeric value to be used for processing the task.

A non-transitory computer-readable storage medium for storing computer-executable instructions which, when executed, cause a processing device to perform a method for performing a task, the method comprising:

• providing at least one numeric value to be used for processing a task,
• receiving information captured from more than one modality,
• for each of the more than one modality, following an initial training performed in accordance with a processing task to be performed, generating a feature map corresponding to the received information captured from the modality by applying at least one transformation corresponding to the modality on the information captured from the modality, wherein each of the feature maps are generated independently of each other;
• wherein the generating of each of the more than one corresponding feature map is performed by applying a given corresponding transformation on a given corresponding modality, and wherein the more than one corresponding transformation is generated following the initial training performed in accordance with the processing task to be performed;
• in accordance with the processing task to be performed, selecting and combining the generated feature maps in accordance with at least one combining operation to generate at least one corresponding combined feature map by, for each generated feature map, reducing numeric values of the generated feature map, reducing numeric values of the generated feature map to a single numeric value of the at least one corresponding combined feature map;
• generating at least one final feature map by applying each of at least one final transformation on at least one of the at least one corresponding combined feature map;
• wherein the at least one final feature map is generated following the initial training performed in accordance with the task to be performed;
• and
• processing the generated at least one final feature map to provide a generated vector of at least one numeric value to be used for processing the task.

A processing device comprising:

• a central processing unit;
• a display device;
• a communication port for operatively connecting the processing device to a plurality of mobile processing devices, each carried by a user;
• a memory unit comprising an application for processing a task, the application comprising instructions for:
• providing at least one numeric value to be used for processing a task,
• receiving information captured from more than one modality,
• for each of the more than one modality, following an initial training performed in accordance with a processing task to be performed, generating a feature map corresponding to the received information captured from the modality by applying at least one transformation corresponding to the modality on the information captured from the modality, wherein each of the feature maps is generated independently of each other;
• wherein the generating of each of the more than one corresponding feature map is performed by applying a given corresponding transformation on a given corresponding modality, and wherein the more than one corresponding transformation is generated following the initial training performed in accordance with the processing task to be performed;
• in accordance with the processing task to be performed, selecting and combining the generated feature maps in accordance with at least one combining operation to generate at least one corresponding combined feature map by, for each generated feature map, reducing numeric values of the generated feature map, reducing numeric values of the generated feature map to a single numeric value of the at least one corresponding combined feature map;
• generating combined feature maps and generating at least one final feature map by applying each of at least one final transformation on at least one of the at least one corresponding combined feature map;
• wherein the at least one final feature map is generated following the initial training performed in accordance with the task to be performed;
• and
• processing the generated at least one final feature map to provide a generated a vector of at least one numeric value to be used for processing the task.

A method for processing a plurality of modalities, wherein the processing is robust to an absence of at least one modality, the method comprising:

• receiving information captured from a plurality of modalities;
• for each of the more than one modality, following an initial training performed in accordance with a processing task to be performed, generating a feature map corresponding to the received information captured from the modality by applying at least one transformation corresponding to the modality on the information captured from the modality, wherein each of the feature maps is generated independently of each other, each generated feature map comprising at least one corresponding numeric value, further wherein the generating of each of the more than one corresponding feature map is performed by applying a given corresponding transformation on a given corresponding modality, and each given corresponding transformation comprises a machine learning model composed of at least a plurality of levels of non-linear operations;
• processing the numeric values of the generated feature maps using at least one combining operation to generate at least one combined representation of the numeric values of the generated feature maps, wherein the at least one combining operation comprises, for each generated combined representation, a computation that reduces each corresponding numeric value of each of the generated feature maps to a single numeric value in the at least one combined representation of the numeric values obtained;
• and
• processing the at least one combined representation of the numeric values obtained using a machine learning model composed of at least one level of at least one of a nonlinear operation and a linear operation for performing the processing of the plurality of modalities.

A system for generating combined feature maps in accordance with a processing task to be performed, the system comprising:

• one or more processors configured to:
• receive information captured from more than one modality,
• for each of the more than one modality, following an initial training performed in accordance with a processing task to be performed, generate a feature map corresponding to the received information captured from the modality by applying at least one transformation corresponding to the modality on the information captured from the modality, wherein each of the feature maps are generated independently of each other;
• wherein the generating of each of the more than one corresponding feature map is performed by applying a given corresponding transformation on a given corresponding modality, and wherein the more than one corresponding transformation is generated following the initial training performed in accordance with the processing task to be performed;
• in accordance with the processing task to be performed, select and combine the corresponding more than one generated feature maps in accordance with at least one combining operation to generate at least one corresponding combined feature map by, for each generated feature map, reducing numeric values of the generated feature map, reducing numeric values of the generated feature map to a single numeric value of the at least one corresponding combined feature map.