# US 12,183,463 B2 · Generation of findings in radiology reports by machine learning based on impressions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,183,463 B2 - Document ID: US12183463B2 - Status: Granted - Kind: B2 - Application number: 17/650,896 - Filed: 2022-02-14 - Published: 2023-08-17 - Granted: 2024-12-31 - Earliest claimed: 2022-02-14 - Assignees: Siemens Healthineers AG - Applicants: Siemens Healthineers AG - Inventors: Oladimeji Farri; Ramya Vunikili - Web page: https://patentgrep.com/patent/US12183463B2 ## CPC - G16H 15/00: ICT specially adapted for medical reports, e.g. generation or transmission thereof - G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems - G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) - G06N 3/045: Combinations of networks - G16H 10/60: for patient-specific data, e.g. for electronic patient records - G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS - G16H 30/40: for processing medical images, e.g. editing - G16H 40/67: for remote operation - G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Machine training is used to learn to generate findings in radiology reports. Rather than merely learning to output findings from an input, the machine training uses loss based on impression derived from findings to machine train the model to generate the findings. Once trained, the machine-learned model generates findings but the findings are more accurate or complete due to having used impression loss in the training. ## Claims ### Claim 1 A method for generating a finding in radiology reports by a machine-learned system, the method comprising: obtaining a medical image of a patient; generating a first finding by a machine-learned model in response to input of the medical image to the machine-learned model, the machine-learned model trained, at least in part, from training data impressions based on training data findings, wherein the machine-learned model comprises a machine-learned first neural network configured to detect regions of interest and a machine-learned hierarchal recurrent neural network or a machine-learned bidirectional encoder representations from transformers network configured to generate the first finding from a projection of the detected regions of interest output by the machine-learned first neural network; and displaying the first finding. ### Claim 2 (depends on 1) The method of claim 1 wherein generating the first finding comprises generating text describing a first occurrence represented in the medical image, where the training data findings represented the first and a second occurrence and the training data impressions represented diagnostic conclusions based on the training data findings. ### Claim 3 (depends on 1) The method of claim 1 wherein generating comprises generating by the machine-learned first neural network comprising a machine-learned vision model configured to receive the medical image, and the first finding comprising text generated from an output of the machine-learned vision model. ### Claim 4 (depends on 3) The method of claim 3 wherein the machine-learned model was trained in a sequence where loss from the training data impressions is back propagated to an impressions model in a first training and then the vision model and the machine-learned hierarchal recurrent neural network or the machine-learned bidirectional encoder representations from transformers network are trained in a second training. ### Claim 5 (depends on 1) The method of claim 1 wherein generating comprises generating by the machine-learned model having been trained with machine learning by an impression model machine learning to generate output impressions from output findings of the model being trained for the machine-learned model, the training having used loss from the training data impressions relative to the output impressions. ### Claim 6 (depends on 5) The method of claim 5 wherein the machine-learned model was trained where values of learnable parameters of the model being trained for the machine-learned model were changed based on backpropagation from the loss of the training data impressions relative to the output impressions. ### Claim 7 (depends on 5) The method of claim 5 wherein the machine-learned model was trained where the impression model also received input of patient background information relative to each training data sample, the patient background information encoded with an attentional encoder. ### Claim 8 (depends on 1) The method of claim 1 wherein generating the first finding comprises generating the first finding as a paragraph of patient findings including the first finding. ### Claim 9 (depends on 1) The method of claim 1 wherein the machine-learned hierarchal recurrent neural network is used and comprises a sentence-level recurrent neural network outputting to a word-level recurrent neural network. ### Claim 10 (depends on 1) The method of claim 1 wherein displaying comprises integrating the first finding into a radiology report including a first impression created by a physician and displaying the radiology report. ### Claim 11 (depends on 1) The method of claim 1 wherein displaying comprises displaying a comparison of the first finding with a physician created finding. ### Claim 12 A method for machine training to generate findings, the method comprising: defining a first model to receive images and output findings, wherein the first model is defined as a first neural network configured to detect regions of interest and a hierarchal recurrent neural network or a bidirectional encoder representations from transformers network configured to generate the findings from a projection of the detected regions of interest output by the first neural network; defining a second model to receive findings and output impressions; machine training the first model, at least in part, based on losses from the output impressions compared to ground truth impressions; and storing the machine-trained first model. ### Claim 13 (depends on 12) The method of claim 12 wherein defining the first model comprises defining the first model as a vision model configured to receive the images and a natural language processing model configured to output the findings as text from an output of the vision model. ### Claim 14 (depends on 13) The method of claim 13 wherein machine training comprises training in a sequence where the loss from the output impressions compared to the ground truth impressions is back propagated to the second model in a first training and then the vision model and the natural language processing model are trained in a second training. ### Claim 15 (depends on 13) The method of claim 13 wherein defining the first model comprises defining the first model as the hierarchal recurrent neural network, the hierarchal recurrent neural network comprising a sentence-level recurrent neural network outputting to a word-level recurrent neural network. ### Claim 16 (depends on 12) The method of claim 12 wherein machine training comprises training where values of learnable parameters of the first model are changed based on backpropagation from the losses. ### Claim 17 (depends on 12) The method of claim 12 wherein machine training comprises machine training the second model based on the losses where the second model also receives input of patient background information relative to each training data sample, the patient background information encoded with an attentional encoder. ### Claim 18 A system for creating an anatomical observation, the system comprising: a medical records database having stored therein an image of a patient; a processor configured to input the image to a machine-trained model configured to create a first anatomical observation in response to input of the image, the machine-trained model having been trained with a loss based on diagnostic conclusion derived from second anatomical observations, wherein the machine-trained model comprises a machine-trained first neural network configured to detect regions of interest and a machine-trained hierarchal recurrent neural network or a machine-trained bidirectional encoder representations from transformers network configured to generate the first anatomical observation from a projection of the detected regions of interest output by the machine-learned first neural network; and a display configured to output the first anatomical observation for the patient.