Generation and Explanation of Transformer Computation Graph Using Graph Attention Model

A method implemented in a data processing system analyzing performance of a machine learning model, the method comprising:

• obtaining attention matrices from a first machine learning model, the first machine learning model having been pretrained, the first machine learning model including a plurality of self-attention layers, and the attention matrices being associated with the plurality of self-attention layers of the first machine learning model;
• analyzing the attention matrices to generate a computation graph based on the attention matrices, the computation graph providing a representation of behavior of the first machine learning model across the plurality of self-attention layers;
• and
• analyzing the computation graph using a second machine learning model, the second machine learning model being trained to receive the computation graph to output model behavior information, the model behavior information identifying which layers of model performed specific tasks associated with generating predictions by the first machine learning model.

A machine-readable medium on which are stored instructions that, when executed, cause a processor of a programmable device to perform operations of:

• obtaining attention matrices from a first machine learning model, the first machine learning model having been pretrained, the first machine learning model including a plurality of self-attention layers, and the attention matrices being associated with the plurality of self-attention layers of the first machine learning model;
• analyzing the attention matrices to generate a computation graph based on the attention matrices, the computation graph providing a representation of behavior of the first machine learning model across the plurality of self-attention layers;
• and
• analyzing the computation graph using a second machine learning model, the second machine learning model being trained to receive the computation graph to output model behavior information, the model behavior information identifying which layers of model performed specific tasks associated with generating predictions by the first machine learning model.