Cross-modal manifold alignment across different data domains

A system for cross-modal manifold alignment of different data domains, the system comprising:

• a processor configured to:
• determine for a shared embedding space a first embedding function for data of a first domain and a second embedding function for data of a second domain using triplet loss, wherein triplets of the triplet loss include an anchor data point from the first domain and a positive and a negative data point from the second domain, wherein the data of the first domain includes RGB color data and RGB depth data;
• colorize plural RGB depth features of the RGB depth data by distributing the plural RGB depth features across plural RGB color features of the RGB color data;
• create a first mapping for the data of the first domain using the first embedding function in a shared embedding space;
• create a second mapping for the data of the second domain using the second embedding function in the shared embedding space;
• and
• generate a cross-modal manifold alignment for the data of the first domain and the data of the second domain,
• train a classifier model based on the cross-modal manifold alignment of the data of the first domain and the data of the second domain,
• wherein to generate the cross-modal manifold alignment the processor is configured to:
• superimpose one of the first mapping or the second mapping on the other of the first mapping or second mapping to generate a cross-modal manifold alignment, wherein to superimpose one of the first mapping or the second mapping on the other of the first mapping or second mapping, the processor is configured to at least rotate the first mapping and/or the second mapping in the shared embedding space by generating a rotation matrix based on a difference between a first term and a second term, the first term including a first ratio of a first difference between the first embedding function and an average of the first embedding function to a Frobenius matrix of the first difference, and the second term including a second ratio of a second difference between the second embedding function and an average of the second embedding function to a Frobenius matrix of the second difference.

A computer-implemented method comprising:

• creating a representation of a cross-modal manifold alignment, in a latent space, for robotics data spanning plural heterogenous domains including a first domain comprising domain data from a robotics language model and a second domain comprising domain data from a robotics imaging model;
• determining a data pair for the robotics data across the first domain and the second domain based on an analysis of the cross-modal manifold alignment, wherein the determining of the data pair comprises correlating words from the robotics language model of the first domain with robotics sensor image data including RGB-D image data of the second domain based on the analysis of the cross-modal manifold alignment;
• training, based on the determining of the data pair, a cross-modal alignment algorithm to identify a plurality of data pairs for the robotics data across at least the first domain and the second domain using the cross-modal manifold alignment;
• generating the representation cross-modal manifold alignment for the robotics data by superimposing a first manifold for the first domain with a second manifold for the second domain, which includes optimizing a translation, a scaling, and a rotation of the first manifold and the second manifold to generate the representation, the first manifold and the second manifold being obtained from a first manifold mapping file for the first domain and a second manifold mapping file for the second domain;
• and
• wherein optimizing a translation, a scaling, and a rotation of the first manifold and the second manifold includes generating a rotation matrix based on a difference between a first term and a second term of a shared embedding space, the first term including a first ratio of a first difference between a first embedding function and an average of the first embedding function to a Frobenius matrix of the first difference, and the second term including a second ratio of a second difference between a second embedding function and an average of the second embedding function to a Frobenius matrix of the second difference.