Machine learning systems and methods for attributed sequences

A training method of a machine learning system for embedding attributed sequence data comprising an attribute data part having a fixed number of attribute data elements and a sequence data part having a variable number of sequence data elements into a fixed-length feature representation, wherein the machine learning system comprises a multilayer feedforward neural network having an attribute data input layer and an attribute vector output layer which comprises a first predetermined number of units, operatively coupled to a long short-term memory (LSTM) network which comprises a second predetermined number of hidden units, the training method comprising:

• providing a dataset comprising a plurality of attributed sequences;
• for each attributed sequence in the dataset,
• training the multilayer feedforward neural network using the attribute data part of the attributed sequence via back-propagation with respect to a first objective function, and
• training the LSTM network using the sequence data part of the attributed sequence via back-propagation with respect to a second objective function,
• wherein training of the multilayer feedforward neural network is coupled with training the LSTM network such that, when trained, the machine learning system is configured to output a fixed-length feature representation of input attributed sequence data which encodes dependencies between different attribute data elements in the attribute data part, dependencies between different sequence data elements in the sequence data part, and dependencies between attribute data elements and sequence data elements within the attributed sequence data.

A training method of a machine learning system for embedding attributed sequence data comprising an attribute data part having a fixed number of attribute data elements and a sequence data part having a variable number of sequence data elements into a fixed-length feature representation, wherein the machine learning system comprises:

• an attribute network module comprising a multilayer feedforward neural network having an attribute data input layer and an attribute vector output layer which comprises a first predetermined number of units;
• a sequence network module comprising a long short-term memory (LSTM) network which comprises a second predetermined number of hidden units;
• and
• a fusion network comprising an input concatenation layer having a number of units equal to a sum of the first predetermined number and the second predetermined number, and a nonlinear function layer comprising a fully-connected feedforward neural network layer,
• wherein the training method comprises:
• providing a dataset comprising a plurality of pairs of attributed sequences, wherein each pair is labeled to indicate whether it comprises similar or dissimilar attributed sequence data;
• and
• for each pair of attributed sequences in the dataset,
• computing, using the multilayer feedforward neural network, a pair of attribute vectors, each having a number of elements equal to the first predetermined number, corresponding with attribute data parts of the attributed sequences;
• computing, using the LSTM network, a pair of sequence vectors, each having a number of elements equal to the second predetermined number, corresponding with sequence data parts of the attributed sequences;
• concatenating corresponding ones of the computed attribute vectors and the computed sequence vectors to generate a pair of fixed-length feature representations of the pair of attributed sequences;
• computing a nonlinear transformation function of the fixed-length feature representations to generate a pair of transformed feature representations;
• computing gradient information based upon a loss function defined in terms of a predetermined distance metric on the pair of transformed feature representations, wherein it is an objective to learn an embedding whereby the pairs of fixed-length feature representations of corresponding samples of attributed sequence data have a smaller distance under the distance metric when labeled as similar than when labeled as dissimilar;
• and
• back-propagating the gradient information through the multilayer feedforward neural network and the LSTM network, whereby parameters of the attribute network module and the sequence network module are updated towards achieving the objective.

A computer program product comprising:

• a non-transitory computer readable storage medium;
• and
• instructions stored on the non-transitory computer readable storage medium that provide a training method of a machine learning system for embedding attributed sequence data comprising an attribute data part having a fixed number of attribute data elements and a sequence data part having a variable number of sequence data elements into a fixed-length feature representation, wherein the machine learning system comprises a multilayer feedforward neural network having an attribute data input layer and an attribute vector output layer which comprises a first predetermined number of units, operatively coupled to a long short-term memory (LSTM) network which comprises a second predetermined number of hidden units, and
• wherein the instructions, upon execution by one or more processors, cause the one or more processors to:
• provide a dataset comprising a plurality of attributed sequences;
• and
• for each attributed sequence in the dataset,
• train the multilayer feedforward neural network using the attribute data part of the attributed sequence via back-propagation with respect to a first objective function;
• and
• train the LSTM network using the sequence data part of the attributed sequence via back-propagation with respect to a second objective function,
• wherein training of the multilayer feedforward neural network is coupled with training the LSTM network such that, when trained, the machine learning system is configured to output a fixed-length feature representation of input attributed sequence data which encodes dependencies between different attribute data elements in the attribute data part, dependencies between different sequence data elements in the sequence data part, and dependencies between attribute data elements and sequence data elements within the attributed sequence data.