Automated extraction of structured labels from medical text using deep convolutional networks and use thereof to train a computer vision model

A method for processing medical text and associated medical images comprising:

• training a natural language processor on a first corpus of curated free-text medical reports each of which having one or more structured labels assigned by an medical expert, the natural language processor trained to learn to read additional free-text medical reports and produce predicted structured labels for such additional free-text medical reports;
• applying the natural language processor to a second corpus of free-text medical reports without structured labels that are associated with medical images and wherein the natural language processor generates structured labels for the associated medical images;
• and
• training a computer vision model using the medical images and the structured labels generated by the natural language processor to assign a structured label to a further input medical image.

A machine learning system comprising, in combination,

• a computer memory storing a set of training data in the form of a multitude of training examples, each of which comprises a free-text medical report and one or more associated medical images, wherein a subset of the training examples contain ground truth structured labels assigned by a medical expert;
• and
• a computer system configured to operate on each of the multitude of training examples, and wherein the computer system is configured as
• a) a feature extractor receiving as input the one or more medical images and generating a vector of extracted image features from the one or more medical images;
• b) a diagnosis extraction network receiving as input a free-text medical report and the vector of extracted image features and generating a structured label;
• c) an image classifier trained on vectors of extracted features produced by the feature extractor from one or more medical images of the multitude of training examples and corresponding structured labels;
• wherein the image classifier is further configured to generate a structured label for a further input medical image from a feature vector generated by the feature extractor from the further input medical image.