# US 2021/0240931 A1 · Visual question answering using on-image annotations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0240931 A1
- Document ID: US20210240931A1
- Status: Published application
- Kind: A1
- Application number: 17/049,642
- Filed: 2019-04-29
- Published: 2021-08-05
- Earliest claimed: 2018-04-30
- Assignees: Koninklijke Philips N.V.
- Applicants: Koninklijke Philips N.V.
- Inventors: Oladimeji Feyisetan FARRI; Sheikh Sadid AL HASAN; Yuan LING
- Web page: https://patentgrep.com/patent/US20210240931A1

## CPC

- G06F 40/279: Recognition of textual entities
- G16H 30/40: for processing medical images, e.g. editing
- G06F 3/16: Sound input; Sound output (speech processing G10L)
- G06F 40/30: Semantic analysis
- G06N 20/00: Machine learning
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G16H 80/00: ICT specially adapted for facilitating communication between medical practitioners or patients, e.g. for collaborative diagnosis, therapy or health monitoring
- A61B 6/032: {Transmission computed tomography [CT]}
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Techniques described herein relate to visual question answering (“VQA”) using trained machine learning models. In various embodiments, a VQA machine learning model may be trained using the follow operations: obtaining (302) a corpus of digital images, each respective digital image (232) including on-image annotation(s) (234) that identify pixel coordinate(s) on the respective digital image; obtaining (304) question-answer pair(s) associated with each of the digital images; generating (306) training examples, each including a respective digital image of the corpus, including the associated on-image annotations, and the associated question-answer pair(s); and for each respective training example of the plurality of training examples: applying (312) the respective training example as input across a machine learning model to generate a respective output; and training (314) the machine learning model based on comparison of the respective output with an answer of the question-answer pair(s) of the respective training example.

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 21

A method implemented using one or more processors, comprising:
obtaining a digital image;
receiving, from a computing device operated by a user, a free-form natural language input;
analyzing the free-form natural language input to identify data indicative of a question by the user about the digital image;
applying the data indicative of the question and the digital image as input across a machine learning model to generate output indicative of a response to the question by the user; and
providing, at the computing device operated by the user, audio or visual output based on the generated output;
wherein the machine learning model includes an encoder portion and a decoder portion that are trained using a plurality of training examples, wherein each respective training example includes:
a digital image that includes one or more on-image annotations that are used to focus attention of the encoder portion on a region of the digital image; and
a question-answer pair associated with the digital image of the respective training example, wherein the question-answer pair includes a question and a corresponding answer;
wherein the decoder portion decodes the answer of the at least one question-answer pair of the respective training example based on an encoding generated using the digital image, the one or more on-image annotations, and at least the question of the question-answer pair of the respective training example as input.

### Claim 22 (depends on 21)

The method of claim 21, wherein the encoder portion comprises a convolutional neural network.

### Claim 23 (depends on 21)

The method of claim 21, wherein the decoder portion comprises a recurrent neural network.

### Claim 24 (depends on 21)

The method of claim 21, wherein the encoder portion comprises one or more long short term memory networks.

### Claim 25 (depends on 21)

The method of claim 21, wherein the encoder portion comprises one or more gated recurrent units.

### Claim 26 (depends on 21)

The method of claim 21, wherein the corpus of digital images comprises medical images obtaining using one or more of magnetic resonance imaging, computed tomography scanning, and x-ray imaging, and the on-image annotations identify medically-significant features of the medical images.