Structure-based transformers with localization and encoding for chart question answering

A non-transitory computer-readable storage medium including instructions stored thereon which, when executed by at least one processor, cause the at least one processor to:

• obtain a visual embedding for a graphical representation of data, the visual embedding representing a plurality of graphical elements of the graphical representation of data, each graphical element associated with position data and a feature map;
• obtain a query embedding for a query associated with the graphical representation of data, the query embedding representing a plurality of textual elements of the query with at least one textual element of the query substituted with an identifier for at least one graphical element of the plurality of graphical elements;
• generate a chart sequence from the visual embedding, each token of the chart sequence including the position data and the feature map representative of one graphical element of the plurality of graphical elements;
• generate a query sequence from the query embedding, each token of the query sequence including a textual element from the plurality of textual elements of the query and a corresponding position element indicating a position of the textual element in the query;
• generate an output sequence based on the chart sequence and the query sequence;
• and
• determine an answer to the query from the output sequence.

A computer-implemented method comprising:

• receiving, by a machine-learning backed service, a request to generate a response to a query associated with a graphical representation of data;
• processing a visual embedding for the graphical representation of data and a query embedding for the query associated with the graphical representation of data to determine a response to the query;
• and
• returning the response to the query.