Systems and methods for cross-modal signal inference using audio signals

A system for converting a primary one-dimensional signal into a secondary one-dimensional signal of another modality, the system comprising:

• a processor programmed to execute instructions stored in memory to:
• splice a primary signal into a plurality of consecutive frames;
• perform a first linear transformation to transform the frames into corresponding vectors;
• execute positional encoding on the vectors to encode relative positional information associated with each sample, wherein the relative positional information is associated with a sequential position of each sample within its respective frame;
• execute a multi-head self-attention mechanism configured to compare relative importance of the samples to each other within its respective frame to yield high-level representation vectors;
• perform a second linear transformation to transform the high-level representation vectors into corresponding secondary signal frames;
• and
• concatenating the secondary signal frames into a reconstructed one-dimensional secondary signal having a different modality than the primary signal.

A computer-controlled machine comprising:

• at least one microphone configured to generate an audio signal;
• a control system configured to predict an operational characteristic of the computer-controlled machine by translating the audio signal into a secondary one-dimensional signal representative of the operational characteristic, the control system configured to:
• splice the audio signal into a plurality of frames,
• execute a first linear transformation to transform the frames into corresponding vectors;
• execute positional encoding on the vectors to encode relative positional information associated with each sample;
• execute a transformer encoder on the vectors with the encoded positional information, wherein the transformer encoder has a multi-head self-attention mechanism configured to compare relative importance of the samples to each other within the respective vectors to yield high-level representation vectors;
• execute a second linear transformation to transform the high-level representation vectors into corresponding secondary signal frames;
• and
• concatenate the corresponding secondary signal frames into a reconstructed one-dimensional secondary signal having a different modality than the audio signal.