# US 2021/0241884 A1 · Convolutional localization networks for intelligent captioning of medical images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0241884 A1 - Document ID: US20210241884A1 - Status: Published application - Kind: A1 - Application number: 17/053,153 - Filed: 2019-05-07 - Published: 2021-08-05 - Earliest claimed: 2018-05-08 - Assignees: Koninklijke Philips N.V. - Applicants: Koninklijke Philips N.V. - Inventors: Christine Menking Swisher; Sheikh Sadid Al Hasan; Jonathan Rubin; Cristhian Mauricio Potes Blandon; Yuan Ling; Oladimeji Feyisetan Farri; Rithesh Sreenivasan - Web page: https://patentgrep.com/patent/US20210241884A1 ## CPC - G06V 20/64: Three-dimensional [3D] objects - G06F 40/10: Text processing (natural language analysis G06F40/20; semantic analysis G06F40/30; processing or translation of natural language G06F40/40) - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06T 7/0012: {Biomedical image inspection} - G06T 7/11: Region-based segmentation - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/82: using neural networks - G16H 30/40: for processing medical images, e.g. editing - G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30004: Biomedical image processing - G06V 2201/031: of internal organs - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA ## Abstract A method (100) for generating a textual description of a medical image, comprising: receiving (130) a medical image of an anatomical region, the image comprising one or more abnormalities; segmenting (140) the anatomical region in the received medical image from a remainder of the image; identifying (150) at least one of the one or more abnormalities in the segmented anatomical region; extracting (160) one or more features from the identified abnormality; generating (170), using the extracted features and a trained text generation model, a textual description of the identified abnormality; and reporting (180), via a user interface of the system, the generated textual description of the identified abnormality. ## Claims ### Claim 1 A method for generating a textual description of a medical image, comprising: receiving a medical image of an anatomical region and associated Digital Imaging and Communications in Medicine (DICOM) tags, the image comprising one or more abnormalities; segmenting the anatomical region in the received medical image from a remainder of the image; identifying at least one of the one or more abnormalities in the segmented anatomical region; extracting one or more features from the identified abnormality using the DICOM tags; generating, using the extracted features and a trained text generation model, a textual description of the identified abnormality; and reporting, via a user interface of the system, the generated textual description of the identified abnormality. ### Claim 2 (depends on 1) The method of claim 1, wherein the medical image is a three-dimensional image. ### Claim 3 (depends on 1) The method of claim 1, wherein identifying the abnormality in the segmented anatomical region further comprises localizing the identified abnormality in three-dimensional space. ### Claim 4 (depends on 1) The method of claim 1, further comprising the step of combining generated textual description from two or more identified abnormalities into a single textual description. ### Claim 5 (depends on 1) The method of claim 1, wherein two or more extracted features are merged to generate a feature vector for the identified abnormality. ### Claim 6 (depends on 1) The method of claim 1, further comprising the step of utilizing, by a clinician, the reported textual description for diagnosis or treatment. ### Claim 7 (depends on 1) The method of claim 1, further comprising the step of training, using a training data set, the text generation model to generate a textual description from an identified abnormality. ### Claim 8 (depends on 1) The method of claim 1, wherein the step of identifying at least one of the one or more abnormalities in the segmented anatomical region comprises a convolutional neural network and/or a long short-term memory (LSTM) network. ### Claim 9 (depends on 1) The method of claim 1, wherein the step of generating a textual description of the identified abnormality comprises a long short-term memory (LSTM) network. ### Claim 10 A system for generating a textual description of a medical image, comprising: an image segmenting module configured to receive a medical image of an anatomical region, the image comprising one or more abnormalities, and further configured to segment the anatomical region in the received medical image from a remainder of the image; an abnormality identification module configured to identify at least one of the one or more abnormalities in the segmented anatomical region; a feature extraction module configured to extract one or more features from the identified abnormality using the DICOM tags; a text generation module configured to generate, using the extracted features and a trained text generation model, a textual description of the identified abnormality; and a user interface configured to report the generated textual description. ### Claim 11 (depends on 10) The system of claim 10, wherein the medical image is a three-dimensional image. ### Claim 12 (depends on 10) The system of claim 10, wherein the feature extraction module is configured to combine two or more extracted features to generate a feature vector for the identified abnormality. ### Claim 13 (depends on 10) The system of claim 10, wherein the text generation module is configured to combine generated a generated textual description from two or more identified abnormalities into a single textual description. ### Claim 14 (depends on 10) The system of claim 10, wherein the text generation module is configured to be trained, using a training data set, to generate a textual description from an identified abnormality. ### Claim 15 (depends on 10) The system of claim 10, wherein the system comprises one or more of a convolutional neural network and a long short-term memory (LSTM) network.