# US 11,790,523 B2 · Autonomous diagnosis of a disorder in a patient from image analysis > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,790,523 B2 - Document ID: US11790523B2 - Status: Granted - Kind: B2 - Application number: 16/175,318 - Filed: 2018-10-30 - Published: 2019-05-02 - Granted: 2023-10-17 - Earliest claimed: 2015-04-06 - Assignees: Digital Diagnostics Inc. - Applicants: Digital Diagnostics Inc. - Inventors: Meindert Niemeijer; Ryan Amelon; Warren Clarida; Michael D. Abramoff - Web page: https://patentgrep.com/patent/US11790523B2 ## CPC - G06T 7/0012: {Biomedical image inspection} - G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections - G06N 3/09: Supervised learning - G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/82: using neural networks - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30041: Eye; Retina; Ophthalmic - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A device receives an input image of a portion of a patient's body, and applies the input image to a feature extraction model, the feature extraction model comprising a trained machine learning model that is configured to generate an output that comprises, for each respective location of a plurality of locations in the input image, an indication that the input image contains an object of interest that is indicative of a presence of a disease state at the respective location. The device applies the output of the feature extraction model to a diagnostic model, the diagnostic model comprising a trained machine learning model that is configured to output a diagnosis of a disease condition in the patient based on the output of the feature extraction model. The device outputs the determined diagnosis of a disease condition in the patient obtained from the diagnostic model. ## Claims ### Claim 1 A method for diagnosing a disease condition in a patient, the method comprising: receiving an input image of a portion of a patient's body; applying the input image to a feature extraction model, the input image comprising a plurality of locations, the feature extraction model comprising a first trained machine learning model that is configured to generate an output that comprises, for each respective location of each of the plurality of locations in the input image, an indication that the input image contains an object of interest at the respective location, where the object of interest is indicative of a disease; applying the output of the feature extraction model to a diagnostic model, the diagnostic model comprising a second trained machine learning model that is configured to output a diagnosis of a disease condition in the patient based on taking as input the indications for each of the plurality of locations in the input image as output by the feature extraction model, where the second trained machine learning model is trained by: applying ground truth training data that comprises labels about whether a corresponding image depicts a disease and, at each location of a plurality of locations of the corresponding image, indicia that the corresponding image contains an object of interest that is indicative of a disease, to the second machine learning model, and adjusting the second machine learning model to fit the ground truth training data, thereby causing the second machine learning model to become trained; and outputting the determined diagnosis of the disease condition in the patient obtained from the diagnostic model. ### Claim 2 (depends on 1) The method of claim 1, wherein applying the input image to the feature extraction model comprises: obtaining a set of samples of the input image, each sample corresponding to a location in the input image; and for each sample of the set of samples, applying the sample to the feature extraction model, the feature extraction model configured to output an indication that the sample contains an object of interest. ### Claim 3 (depends on 2) The method of claim 2, wherein the set of samples is obtained by at least one of random sampling and grid sampling. ### Claim 4 (depends on 1) The method of claim 1, wherein the indication that the input image contains an object of interest at one or more locations comprises a mathematical representation of the object of interest. ### Claim 5 (depends on 1) The method of claim 1, wherein the output of the feature extraction model comprises a heat map corresponding to indications that the input image contains an object of interest at the plurality of locations in the input image. ### Claim 6 (depends on 1) The method of claim 1, wherein the output of the feature extraction model comprises a point-wise output corresponding to indications that the input image contains an object of interest at the plurality of locations in the input image. ### Claim 7 (depends on 1) The method of claim 1, wherein one or more of the objects of interest is indicative of disease. ### Claim 8 (depends on 1) The method of claim 1, wherein the portion of the patient's body includes at least a portion of the patient's eye, and the determined diagnosis of the disease condition in the patient comprises a diagnosis of a disorder manifesting in the retina. ### Claim 9 (depends on 8) The method of claim 8, wherein one or more of the objects of interest is selected from a group consisting of: a microaneurysm, a dot hemorrhage, a flame-shaped hemorrhage, a sub-intimal hemorrhage, a sub-retinal hemorrhage, a pre-retinal hemorrhage, a micro-infarction, a cotton-wool spot, and a yellow exudate. ### Claim 10 (depends on 1) The method of claim 1, wherein the input image is obtained by at least one of: computed tomography (CT), magnetic resonance imaging (MRI), computed radiography, magnetic resonance, angioscopy, optical coherence tomography, color flow Doppler, cystoscopy, diaphanography, echocardiography, fluorescein angiography, laparoscopy, magnetic resonance angiography, positron emission tomography, single-photon emission computed tomography, x-ray angiography, nuclear medicine, biomagnetic imaging, colposcopy, duplex Doppler, digital microscopy, endoscopy, fundoscopy, laser surface scanning, magnetic resonance spectroscopy, radiographic imaging, thermography, and radio fluoroscopy. ### Claim 11 A non-transitory computer-readable medium comprising memory with instructions encoded thereon for diagnosing a disease condition in a patient, the instructions, when executed, causing one or more processors to perform operations, the instructions comprising instructions to: receive an input image of a portion of a patient's body; apply the input image to a feature extraction model, the input image comprising a plurality of locations, the feature extraction model comprising a first trained machine learning model that is configured to generate an output that comprises, for each respective location of each of the plurality of locations in the input image, an indication that the input image contains an object of interest at the respective location, where the object of interest is indicative of a disease; apply the output of the feature extraction model to a diagnostic model, the diagnostic model comprising a second trained machine learning model that is configured to output a diagnosis of a disease condition in the patient based on taking as input the indications for each of the plurality of locations in the input image as output by the feature extraction model, where the second trained machine learning model is trained by: applying ground truth training data that comprises labels about whether a corresponding image depicts a disease and, at each location of a plurality of locations of the corresponding image, indicia that the corresponding image contains an object of interest that is indicative of a disease, to the second machine learning model, and adjusting the second machine learning model to fit the ground truth training data, thereby causing the second machine learning model to become trained; and output the determined diagnosis of the disease condition in the patient obtained from the diagnostic model. ### Claim 12 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein the instructions to apply the input image to the feature extraction model comprise instructions to: obtain a set of samples of the input image, each sample corresponding to a location in the input image; and for each sample of the set of samples, apply the sample to the feature extraction model, the feature extraction model configured to output an indication that the sample contains an object of interest. ### Claim 13 (depends on 12) The non-transitory computer-readable medium of claim 12, wherein the set of samples is obtained by at least one of random sampling and grid sampling. ### Claim 14 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein the indication that the input image contains an object of interest at one or more locations comprises a mathematical representation of the object of interest. ### Claim 15 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein the output of the feature extraction model comprises a heat map corresponding to indications that the input image contains an object of interest at the plurality of locations in the input image. ### Claim 16 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein the output of the feature extraction model comprises a point-wise output corresponding to indications that the input image contains an object of interest at the plurality of locations in the input image. ### Claim 17 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein one or more of the objects of interest is indicative of disease. ### Claim 18 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein the portion of the patient's body includes at least a portion of the patient's eye, and the determined diagnosis of the disease condition in the patient comprises a diagnosis of a disorder manifesting in the retina. ### Claim 19 (depends on 18) The non-transitory computer-readable medium of claim 18, wherein one or more of the objects of interest is selected from a group consisting of: a microaneurysm, a dot hemorrhage, a flame-shaped hemorrhage, a sub-intimal hemorrhage, a sub-retinal hemorrhage, a pre-retinal hemorrhage, a micro-infarction, a cotton-wool spot, and a yellow exudate. ### Claim 20 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein the input image is obtained by at least one of: computed tomography (CT), magnetic resonance imaging (MRI), computed radiography, magnetic resonance, angioscopy, optical coherence tomography, color flow Doppler, cystoscopy, diaphanography, echocardiography, fluorescein angiography, laparoscopy, magnetic resonance angiography, positron emission tomography, single-photon emission computed tomography, x-ray angiography, nuclear medicine, biomagnetic imaging, colposcopy, duplex Doppler, digital microscopy, endoscopy, fundoscopy, laser surface scanning, magnetic resonance spectroscopy, radiographic imaging, thermography, and radio fluoroscopy. ### Claim 21 A system for diagnosing a disease condition in a patient, the system comprising: memory with instructions encoded thereon; and one or more processors that, when executing the instructions, are caused to perform operations comprising: receiving an input image of a portion of a patient's body; applying the input image to a feature extraction model, the input image comprising a plurality of locations, the feature extraction model comprising a trained machine learning model that is configured to generate an output that comprises, for each respective location of each of the plurality of locations in the input image, an indication that the input image contains an object of interest at the respective location, where the object of interest is indicative of a disease; applying the output of the feature extraction model to a diagnostic model, the diagnostic model comprising a second trained machine learning model that is configured to output a diagnosis of a disease condition in the patient based on taking as input the indications for each of the plurality of locations in the input image as output by the feature extraction model, where the second trained machine learning model is trained by: applying ground truth training data that comprises labels about whether a corresponding image depicts a disease and, at each location of a plurality of locations of the corresponding image, indicia that the corresponding image contains an object of interest that is indicative of a disease, to the second machine learning model, and adjusting the second machine learning model to fit the ground truth training data, thereby causing the second machine learning model to become trained; and outputting the determined diagnosis of the disease condition in the patient obtained from the diagnostic model. ### Claim 22 (depends on 21) The system of claim 21, wherein applying the input image to the feature extraction model comprises: obtaining a set of samples of the input image, each sample corresponding to a location in the input image; and for each sample of the set of samples, applying the sample to the feature extraction model, the feature detection model configured to output an indication that the sample contains an object of interest. ### Claim 23 (depends on 22) The system of claim 22, wherein the set of samples is obtained by at least one of random sampling and grid sampling. ### Claim 24 (depends on 21) The system of claim 21, wherein the indication that the input image contains an object of interest at one or more locations comprises a mathematical representation of the object of interest. ### Claim 25 (depends on 21) The system of claim 21, wherein the output of the feature extraction model comprises a heat map corresponding to indications that the input image contains an object of interest at the plurality of locations in the input image. ### Claim 26 (depends on 21) The system of claim 21, wherein the output of the feature extraction model comprises a point-wise output corresponding to indications that the input image contains an object of interest at the plurality of locations in the input image. ### Claim 27 (depends on 21) The system of claim 21, wherein one or more of the objects of interest is indicative of disease. ### Claim 28 (depends on 21) The system of claim 21, wherein the portion of the patient's body includes at least a portion of the patient's eye, and the determined diagnosis of the disease condition in the patient comprises a diagnosis of a disorder manifesting in the retina. ### Claim 29 (depends on 28) The system of claim 28, wherein one or more of the objects of interest is selected from a group consisting of: a microaneurysm, a dot hemorrhage, a flame-shaped hemorrhage, a sub-intimal hemorrhage, a sub-retinal hemorrhage, a pre-retinal hemorrhage, a micro-infarction, a cotton-wool spot, and a yellow exudate. ### Claim 30 (depends on 21) The system of claim 21, wherein the input image is obtained by at least one of: computed tomography (CT), magnetic resonance imaging (MRI), computed radiography, magnetic resonance, angioscopy, optical coherence tomography, color flow Doppler, cystoscopy, diaphanography, echocardiography, fluorescein angiography, laparoscopy, magnetic resonance angiography, positron emission tomography, single-photon emission computed tomography, x-ray angiography, nuclear medicine, biomagnetic imaging, colposcopy, duplex Doppler, digital microscopy, endoscopy, fundoscopy, laser surface scanning, magnetic resonance spectroscopy, radiographic imaging, thermography, and radio fluoroscopy.