# US 2020/0401843 A1 · Method and system for machine learning classification based on structure or material segmentation in an image > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401843 A1 - Document ID: US20200401843A1 - Status: Published application - Kind: A1 - Application number: 16/448,474 - Filed: 2019-06-21 - Published: 2020-12-24 - Earliest claimed: 2019-06-21 - Assignees: Straxcorp Pty Ltd - Applicants: StraxCorp Pty. Ltd. - Inventors: Yu PENG - Web page: https://patentgrep.com/patent/US20200401843A1 ## CPC - G06T 7/0012: {Biomedical image inspection} - A61B 5/4509: {Bone density determination} - A61B 5/4842: {Monitoring progression or stage of a disease} - A61B 5/7267: {involving training the classification device} - A61B 5/7275: {Determining trends in physiological measurement data; Predicting development of a medical condition based on physiological measurements, e.g. determining a risk factor} - G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade} - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06F 18/2431: Multiple classes - G06T 7/11: Region-based segmentation - G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines - G06F 18/24323: {Tree-organised classifiers} - G06T 2207/10081: Computed x-ray tomography [CT] - G06T 2207/20076: Probabilistic image processing - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30008: Bone - G06T 2207/30096: Tumor; Lesion - G06V 2201/03: Recognition of patterns in medical or anatomical images - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A system and method for classifying a structure or material in an image of a subject. The system comprises: a segmenter configured to form one or more segmentations of a structure or material in an image and generate from the segmentations one or more segmentation maps of the image including categorizations of pixels or voxels of the segmentation maps assigned from one or more respective predefined sets of categories; a classifier that implements a classification machine learning model configured to generate, based on the segmentations maps, one or more classifications and to assign to the classifications respective scores indicative of a likelihood that the structure or material, or the subject, falls into the respective classifications; and an output for outputting a result indicative of the classifications and scores. ## Claims ### Claim 1 A system for classifying a structure or material in an image of a subject, comprising: a segmenter configured to segment an image into one or more segmentations that correspond to respective structures or materials in the image, and to generate from the segmentations one or more segmentation maps of the image including categorizations of pixels or voxels of the segmentation maps assigned from one or more respective predefined sets of categories; a classifier that implements a trained classification machine learning model configured to generate, based on the segmentations maps, one or more classifications and to assign to the classifications respective scores indicative of a likelihood that the structure or material, or the subject, falls into the respective classifications; and an output for outputting a result indicative of the classifications and scores. ### Claim 2 (depends on 1) A system as claimed in claim 1, wherein the classifier generates the one or more classifications based on the segmentations maps and non-image data pertaining to the subject. ### Claim 3 (depends on 1) A system as claimed in claim 1, wherein the segmenter comprises: i) a structure segmenter configured to generate structure segmentation maps including categorizations of the pixels or voxels assigned from a predefined set of structure categories, ii) a material segmenter configured to generate material segmentation maps including categorizations of the pixels or voxels assigned from a predefined set of material categories, and/or iii) an abnormality segmenter configured to generate abnormality segmentation maps including categorizations of the pixels or voxels assigned from a predefined set of abnormality or normality categories. ### Claim 4 (depends on 3) A system as claimed in claim 3, wherein the structure segmenter is configured to employ a structure segmentation machine learning model to generate the structure segmentation maps, the material segmenter is configured to employ a material segmentation machine learning model to generate the material segmentation maps, and the abnormality segmenter is configured to employ an abnormality segmentation model to generate the abnormality segmentation maps. ### Claim 5 (depends on 1) A system as claimed in claim 1, further comprising a segmentation map processor configured to down-sample or otherwise process the segmentation maps before the segmentation maps are input by the classifier. ### Claim 6 (depends on 1) A system as claimed in claim 1, wherein the classification machine learning model comprises (a) a neural network, a support vector machine, and/or a decision tree; or (b) a neural network that includes convolutional neural network layers and fully-connected neural network layers. ### Claim 7 (depends on 1) A system as claimed in claim 1, wherein the image is a medical image, and the classifications correspond to (i) probabilities that the structure or material, or the subject, will sustain a specified condition or symptom in respective timeframes; (ii) probabilities that the structure or material, or the subject, will sustain a specified condition or symptom in respective timeframes that include a shorter-term timeframe, a longer-term timeframe, and at least one intermediate-term timeframe intermediate the shorter-term timeframe and the longer-term timeframe; (iii) probabilities that the structure or material, or the subject, will sustain respective conditions or symptoms; (iv) probabilities of respective rates of disease or pathology progression; (v) probabilities of respective rates of disease or pathology progression, the classifications comprising classifications corresponding any one or more of: stable, modest deterioration, and accelerated deterioration; (vi) probabilities of efficacy of respective treatment options; (vii) probabilities of efficacy of respective treatment options, the treatment options including an antiresorptive treatment and/or an anabolic treatment; (viii) respective medical conditions; and/or (ix) respective medical conditions that include any one or more of: osteomalacia, tumour, osteonecrosis and infection. ### Claim 8 (canceled) ### Claim 9 (depends on 1) A system as claimed in claim 1, wherein the trained classification machine learning model is a model trained with image data and non-image data relating to training subjects, and generates the respective scores based on image data and non-image data relating to the subject. ### Claim 10 A computer-implemented method for classifying a structure or material in an image of a subject, comprising: segmenting an image into one or more segmentations that correspond to respective structures or materials in the image; generating from the segmentations one or more segmentation maps of the image including categorizations of pixels or voxels of the segmentation maps assigned from respective predefined sets of categories of the structure or material; using a trained classification machine learning model to generate, based on the segmentations maps, one or more classifications and to assign to the classifications respective scores indicative of a likelihood that the structure or material, or the subject, falls into the respective classifications; and outputting a result indicative of the classifications and scores. ### Claim 11 (depends on 10) A method as claimed in claim 10, wherein the trained classification machine learning model generates (i) the one or more classifications based on the segmentations maps and non-image data pertaining to the subject, and/or (ii) the respective scores based on image data and non-image data relating to the subject having been trained with image data and non-image data relating to training subjects. ### Claim 12 (depends on 10) A method as claimed in claim 10, wherein forming the one or more segmentations comprises: i) generating structure segmentation maps including categorizations of the pixels or voxels assigned from a predefined set of structure categories, ii) generating material segmentation maps including categorizations of the pixels or voxels assigned from a predefined set of material categories, and/or iii) generating abnormality segmentation maps including categorizations of the pixels or voxels assigned from a predefined set of abnormality or normality categories. ### Claim 13 (depends on 12) A method as claimed in claim 12, including employing a structure segmentation machine learning model to generate the structure segmentation maps, a material segmentation machine learning model to generate the material segmentation maps, and an abnormality segmentation model to generate the abnormality segmentation maps. ### Claim 14 A computer-implemented method for training a classification machine learning model for classifying a structure or material in an image of a subject, the method comprising: dividing annotated segmentation maps and annotated non-image data into a training set and a testing set, the annotated segmentation maps obtained by segmenting each of one or more images into one or more segmentations that correspond to respective structures or materials in the respective image; (a) implementing a classification machine learning model, including initializing parameters of the classification machine learning model; (b) updating the parameters of the classification machine learning model by running a learning algorithm on the training data; (c) testing the classification machine learning model on the testing data; (d) evaluating whether the classification machine learning model has satisfactory performance; repeating steps (a) to (d) when the performance is found in step (d) not to be satisfactory; and outputting the classification machine learning model for deployment as a trained classification machine learning model, or flagging the classification machine learning model as a trained classification machine learning model. ### Claim 15 (depends on 14) A method as claimed in claim 14, including, when the performance is found in step (d) not to be satisfactory, receiving and using more image and non-image data for training the classification machine learning model. ### Claim 16 (depends on 14) A method as claimed in claim 14, wherein the classification machine learning model comprises a neural network having a plurality of layers comprising artificial neurons, wherein the parameters comprise layer numbers, neuron numbers, neuron weights, and neuron function parameters; and testing the classification machine learning model includes testing the classification machine learning model on the testing data. ### Claim 17 (depends on 14) A method as claimed in claim 14, including dividing the annotated segmentation maps and the annotated non-image data into the training set, a development set and the testing set, and using the development data to investigate the learning procedure and to tune the parameters. ### Claim 18 (depends on 10) A computer-implemented diagnostic method, comprising the method of claim 10. ### Claim 19 (depends on 10) A computer program, comprising program code configured, when executed by one of more computing devices, to implement+ed the method of claim 10. ### Claim 20 (depends on 19) A computer-readable medium, comprising the computer program of claim 19. ### Claim 21 (depends on 14) A classification machine learning model trained according to the method of claim 14. ### Claim 22 (depends on 14) A method as claimed in claim 14, wherein the image is a medical image, and the classification machine learning model is configured to generate one or more classifications that correspond to (i) probabilities that the structure or material, or the subject, will sustain a specified condition or symptom in respective timeframes; (ii) probabilities that the structure or material, or the subject, will sustain a specified condition or symptom in respective timeframes that include a shorter-term timeframe, a longer-term timeframe, and at least one intermediate-term timeframe intermediate the shorter-term timeframe and the longer-term timeframe; (iii) probabilities that the structure or material, or the subject, will sustain respective conditions or symptoms; (iv) probabilities of respective rates of disease or pathology progression; (v) probabilities of respective rates of disease or pathology progression, the classifications comprising classifications corresponding any one or more of: stable, modest deterioration, and accelerated deterioration; (vi) probabilities of efficacy of respective treatment options; (vii) probabilities of efficacy of respective treatment options, the treatment options including an antiresorptive treatment and/or an anabolic treatment; (viii) respective medical conditions; and/or (ix) respective medical conditions that include any one or more of: osteomalacia, tumour, osteonecrosis and infection. ### Claim 23 A system for training a classification machine learning model for classifying a structure or material in an image of a subject, the system comprising a processor configured to: divide annotated segmentation maps and annotated non-image data into a training set and a testing set, the annotated segmentation maps obtained by segmenting each of one or more images into one or more segmentations that correspond to respective structures or materials in the respective image; (a) implement a classification machine learning model, including initializing parameters of the classification machine learning model; (b) update the parameters of the classification machine learning model by running a learning algorithm on the training data; (c) test the classification machine learning model on the testing data; (d) evaluate whether the classification machine learning model has satisfactory performance; repeat steps (a) to (d) when the performance is found in step (d) not to be satisfactory; and output the classification machine learning model for deployment as a trained classification machine learning model, or flag the classification machine learning model as a trained classification machine learning model. ### Claim 24 (depends on 10) A classification of a structure or material in an image of a subject, generated according to the method of claim 10.