Semantic guided scene flow estimation

An apparatus for scene flow estimation, the apparatus comprising:

• a memory for storing multimodal data;
• and
• processing circuitry in communication with the memory, wherein the processing circuitry is configured to:
• receive the multimodal data having at least a first modality and a second modality, wherein the multimodal data represents a plurality of points in a scene;
• extract a first set of features from the first modality and extract a second set of features from the second modality;
• project the first set of features and the second set of features into a shared latent space to generate a first latent representation of the first set of features and a second latent representation of the second set of features;
• and
• estimate a flow of the plurality of points of the scene based on one or more relationships between the first set of features and the second set of features by using a model configured to represent the one or more relationships between the first set of features and the second set of features based on the first latent representation and the second latent representation, wherein the model is configured to integrate the first set of features with the second set of features to generate an integrated representation by concatenating the first set of features with the second set of features using cross-modal attention and using one or more attention weights comprising scalar values indicating a degree of influence the first set of features and the second set of features have on the integrated representation.

A non-transitory computer-readable medium storing instructions that, when applied by processing circuitry, causes the processing circuitry to:

• receive the multimodal data having at least a first modality and a second modality, wherein the multimodal data represents a plurality of points in a scene;
• extract a first set of features from the first modality and extract a second set of features from the second modality;
• project the first set of features and the second set of features into a shared latent space to generate a first latent representation of the first set of features and a second latent representation of the second set of features;
• and
• estimate a flow of the plurality of points of the scene based on the one or more relationships between the first set of features and the second set of features by using a model configured to represent the one or more relationships between the first set of features and the second set of features based on the first latent representation and the second latent representation, wherein the model is configured to integrate the first set of features with the second set of features to generate an integrated representation by concatenating the first set of features with the second set of features using cross-modal attention and using one or more attention weights comprising scalar values indicating a degree of influence the first set of features and the second set of features have on the integrated representation.