Methods for tokenization representation and learning of robotic perception data based on graph neural network

A method for token-based representation and learning of robotic perception data based on a graph neural network, comprising:

• S1, data obtaining:
• obtaining a plurality of types of perception data of a robot;
• S2, performing token-based representation according to types of the plurality of types of perception data;
• S3, token-based representation learning:
• S31, constructing an initial feature graph based on the plurality of types of perception data after the token-based representation;
• S32, learning a compact representation of a feature graph based on an autoencoder and reconstructing a graph structure, wherein the graph structure represents a relationship of edges between different nodes, the autoencoder includes an encoder and a decoder, the encoder employs a graph attention network, the encoder is configured to map node features to a latent space, and the decoder reconstructs an initial graph structure from the latent space;
• the learning a compact representation of a feature graph based on an autoencoder and reconstructing a graph structure including:
• random masking:
• selecting a portion of nodes for masking to obtain masked nodes, wherein a masking manner includes a random selection strategy, and for a feature vector corresponding to a masked node, the feature vector is replaced with a zero vector or a preset mask token;
• encoding:
• using the graph attention network as the encoder to encode nodes that are not masked;
• decoding:
• receiving, by the decoder, node representations generated by the encoder and mask information, and predicting features of the masked nodes;
• loss function definition:
• defining a mean squared error loss function to measure a difference between an output of the decoder and features of actual masked nodes, and introducing an actual physical constraint into the mean squared error loss function;
• backpropagation and iterative training:
• performing gradient descent on model parameters of the autoencoder by utilizing the mean squared error loss function to minimize a loss until the autoencoder converges or reaches a predetermined training epoch to obtain a trained autoencoder;
• and
• utilizing the trained autoencoder to learn node mappings of the plurality of types of perception data on the graph neural network, and establishing connections between nodes according to intrinsic correlations among the plurality of types of perception data to form an undirected graph;
• S33, after the autoencoder completes learning of the graph structure, fixing the graph structure;
• and
• S34, converting the plurality of types of perception data into node feature vectors, constructing a feature graph based on the graph structure, and performing numerical encoding on each of the node feature vectors by utilizing the graph neural network to obtain a representation of high-dimensional feature vectors of the plurality of types of perception data;
• the performing numerical encoding on each of the node feature vectors by utilizing the graph neural network to obtain a representation of high-dimensional feature vectors of the plurality of types of perception data including:
• initializing each node corresponding to each piece of perception data, and converting initial numerical values into an initial node feature vector using a multi-layer perceptron;
• utilizing a message passing mechanism of the graph neural network to allow the each node to exchange information with neighbor nodes of the each node, and updating a representation of the each node in each layer and aggregating information from the neighbor nodes into the representation of the each node to capture interrelationships between the plurality of types of perception data;
• and
• a plurality of iterative optimizations:
• after a plurality of iterations of the graph neural network, obtaining a new embedding vector by the each node, wherein the new embedding vector includes a node feature of the each node itself and the information from the neighbor nodes fused by the each node, and the new embedding vector is used as a learned representation of the high-dimensional feature vectors.