Visual question answering using on-image annotations

A method implemented using one or more processors, comprising:

• obtaining a digital image;
• receiving, from a computing device operated by a user, a free-form natural language input;
• analyzing the free-form natural language input to identify data indicative of a question by the user about the digital image;
• applying the data indicative of the question and the digital image as input across a machine learning model to generate output indicative of a response to the question by the user;
• and
• providing, at the computing device operated by the user, audio or visual output based on the generated output;
• wherein the machine learning model includes an encoder portion and a decoder portion that are trained using a plurality of training examples, wherein each respective training example includes:
• a digital image that includes one or more on-image annotations that are used to focus attention of the encoder portion on a region of the digital image;
• and
• a question-answer pair associated with the digital image of the respective training example, wherein the question-answer pair includes a question and a corresponding answer;
• wherein the decoder portion decodes the answer of the at least one question-answer pair of the respective training example based on an encoding generated using the digital image, the one or more on-image annotations, and at least the question of the question-answer pair of the respective training example as input.