# US 12,217,418 B2 · Method and system for machine learning classification based on structure or material segmentation in an image > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,418 B2 - Document ID: US12217418B2 - Status: Granted - Kind: B2 - Application number: 17/584,752 - Filed: 2022-01-26 - Published: 2022-05-12 - Granted: 2025-02-04 - Earliest claimed: 2019-06-21 - Assignees: Curvebeam Ai Limited - Applicants: StraxCorp Pty. Ltd. - Inventors: Yu Peng - Web page: https://patentgrep.com/patent/US12217418B2 ## CPC - G06T 7/0012: {Biomedical image inspection} - A61B 5/4509: {Bone density determination} - A61B 5/4842: {Monitoring progression or stage of a disease} - A61B 5/7267: {involving training the classification device} - A61B 5/7275: {Determining trends in physiological measurement data; Predicting development of a medical condition based on physiological measurements, e.g. determining a risk factor} - G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade} - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06F 18/2431: Multiple classes - G06T 7/11: Region-based segmentation - G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines - G06F 18/24323: {Tree-organised classifiers} - G06T 2207/10081: Computed x-ray tomography [CT] - G06T 2207/20076: Probabilistic image processing - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30008: Bone - G06T 2207/30096: Tumor; Lesion - G06V 2201/03: Recognition of patterns in medical or anatomical images - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A system and method for classifying a structure or material in an image of a subject. The system comprises: a segmenter configured to segment an image into one or more segmentations that correspond to respective structures or materials in the image, and to generate from the segmentations one or more segmentation maps of the image (each of the segmentation maps representing the image) including categorizations of pixels or voxels of the segmentation maps assigned from one or more respective predefined sets of categories; a classifier that implements a classification machine learning model configured to generate, based on the segmentations maps, one or more classifications and to assign to the classifications respective scores indicative of a likelihood that the structure or material, or the subject, falls into the respective classifications; and an output for outputting a result indicative of the classifications and scores. ## Claims ### Claim 1 A system for classifying a structure or material in an image of a subject, comprising: a segmenter configured to segment an image into one or more segmentations that correspond to respective structures or materials in the image, and to generate from the segmentations one or more segmentation maps of the image including categorizations of pixels or voxels of the segmentation maps assigned from one or more respective predefined sets of categories; a classifier that implements a trained classification machine learning model configured to generate, based on the segmentations maps, one or more classifications and to assign to the classifications respective scores indicative of a likelihood that the structure or material, or the subject, falls into the respective classifications; and an output for outputting a result indicative of the classifications and scores; wherein the segmenter comprises: a structure segmenter configured to generate structure segmentation maps including categorizations of the pixels or voxels assigned from a predefined set of structure categories and to employ a structure segmentation machine learning model to generate the structure segmentation maps, a material segmenter configured to generate material segmentation maps including categorizations of the pixels or voxels assigned from a predefined set of material categories and to employ a material segmentation machine learning model to generate the material segmentation maps, and/or an abnormality segmenter configured to generate abnormality segmentation maps including categorizations of the pixels or voxels assigned from a predefined set of abnormality or normality categories and to employ an abnormality segmentation model to generate the abnormality segmentation maps. ### Claim 2 (depends on 1) A system as claimed in claim 1, wherein the classifier generates the one or more classifications based on the segmentations maps and non-image data pertaining to the subject. ### Claim 3 (depends on 1) A system as claimed in claim 1, further comprising a segmentation map processor configured to down-sample or otherwise process the segmentation maps before the segmentation maps are input by the classifier. ### Claim 4 (depends on 1) A system as claimed in claim 1, wherein the classification machine learning model comprises (a) a neural network, a support vector machine, and/or a decision tree; or (b) a neural network that includes convolutional neural network layers and fully-connected neural network layers. ### Claim 5 (depends on 1) A system as claimed in claim 1, wherein the image is a medical image, and the classifications correspond to (i) probabilities that the structure or material, or the subject, will sustain a specified condition or symptom in respective timeframes; (ii) probabilities that the structure or material, or the subject, will sustain a specified condition or symptom in respective timeframes that include a shorter-term timeframe, a longer-term timeframe, and at least one intermediate-term timeframe intermediate the shorter-term timeframe and the longer-term timeframe; (iii) probabilities that the structure or material, or the subject, will sustain respective conditions or symptoms; (iv) probabilities of respective rates of disease or pathology progression; (v) probabilities of respective rates of disease or pathology progression, the classifications comprising classifications corresponding any one or more of: stable, modest deterioration, and accelerated deterioration; (vi) probabilities of efficacy of respective treatment options; (vii) probabilities of efficacy of respective treatment options, the treatment options including an antiresorptive treatment and/or an anabolic treatment; (viii) respective medical conditions; and/or (ix) respective medical conditions that include any one or more of: osteomalacia, tumour, osteonecrosis and infection. ### Claim 6 (depends on 1) A system as claimed in claim 1, wherein the trained classification machine learning model is a model trained with image data and non-image data relating to training subjects, and generates the respective scores based on image data and non-image data relating to the subject. ### Claim 7 A computer-implemented method for classifying a structure or material in an image of a subject, comprising: segmenting an image into one or more segmentations that correspond to respective structures or materials in the image; generating from the segmentations one or more segmentation maps of the image including categorizations of pixels or voxels of the segmentation maps assigned from respective predefined sets of categories of the structure or material; using a trained classification machine learning model to generate, based on the segmentations maps, one or more classifications and to assign to the classifications respective scores indicative of a likelihood that the structure or material, or the subject, falls into the respective classifications; and outputting a result indicative of the classifications and scores; wherein forming the one or more segmentations comprises: generating structure segmentation maps including categorizations of the pixels or voxels assigned from a predefined set of structure categories and employing a structure segmentation machine learning model to generate the structure segmentation maps, generating material segmentation maps including categorizations of the pixels or voxels assigned from a predefined set of material categories and employing a material segmentation machine learning model to generate the material segmentation maps, and/or generating abnormality segmentation maps including categorizations of the pixels or voxels assigned from a predefined set of abnormality or normality categories and employing an abnormality segmentation model to generate the abnormality segmentation maps. ### Claim 8 (depends on 7) A method as claimed in claim 7, wherein the trained classification machine learning model generates (i) the one or more classifications based on the segmentations maps and non-image data pertaining to the subject, and/or (ii) the respective scores based on image data and non-image data relating to the subject having been trained with image data and non-image data relating to training subjects. ### Claim 9 (depends on 7) A method as claimed in claim 7, wherein the image is a medical image, and the classification machine learning model is configured to generate one or more classifications that correspond to (i) probabilities that the structure or material, or the subject, will sustain a specified condition or symptom in respective timeframes; (ii) probabilities that the structure or material, or the subject, will sustain a specified condition or symptom in respective timeframes that include a shorter-term timeframe, a longer-term timeframe, and at least one intermediate-term timeframe intermediate the shorter-term timeframe and the longer-term timeframe; (iii) probabilities that the structure or material, or the subject, will sustain respective conditions or symptoms; (iv) probabilities of respective rates of disease or pathology progression; (v) probabilities of respective rates of disease or pathology progression, the classifications comprising classifications corresponding any one or more of: stable, modest deterioration, and accelerated deterioration; (vi) probabilities of efficacy of respective treatment options; (vii) probabilities of efficacy of respective treatment options, the treatment options including an antiresorptive treatment and/or an anabolic treatment; (viii) respective medical conditions; and/or (ix) respective medical conditions that include any one or more of: osteomalacia, tumour, osteonecrosis and infection. ### Claim 10 (depends on 7) A computer-implemented diagnostic method, comprising the method of claim 7. ### Claim 11 (depends on 7) A non-transitory computer-readable medium, comprising a computer program comprising program code configured, when executed by one of more computing devices, to implemented the method of claim 7.