Spatio-temporal consistency embeddings from multiple observed modalities

A non-transitory, machine-readable medium storing instructions that when executed by one or more processors effectuate operations comprising:

• obtaining, with a computer system, data indicative of state of a dynamic mechanical system and an environment of the dynamic mechanical system, the data comprising two or more different channels of sensor data reporting properties sensed by respective types of sensors;
• segmenting each channel of sensor data into respective sets of sensor data records;
• forming, with the computer system, a set of training records that span the different channels by selecting, for each training record, a subset of sensor data records from each of the sets of sensor data records for each channel;
• training, with the computer system, a metric learning model to encode inputs corresponding to the different channels as vectors in an embedding space with self-supervised learning based on the set of training records, wherein:
• training comprises iteratively adjusting parameters of the metric learning model based on outputs of an objective function, and
• the objective function causes the parameters to be adjusted during training in directions that cause vectors in the embedding space to encode temporal consistency of properties sensed by the respective types of sensors in the different training records in the set of training records;
• and
• using, with the computer system, the trained metric learning model to output embedding vectors indicative of current system and environment state within the embedding space responsive to control of the dynamic mechanical system or another dynamic mechanical system.