# US 2021/0279866 A1 · Machine learning image processing techniques > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0279866 A1 - Document ID: US20210279866A1 - Status: Published application - Kind: A1 - Application number: 17/194,235 - Filed: 2021-03-06 - Published: 2021-09-09 - Earliest claimed: 2020-03-06 - Assignees: BostonGene Corporation - Applicants: BostonGene Corporation - Inventors: Viktor Svekolkin; Ilia Galkin; Ekaterina Postovalova; Ravshan Ataullakhanov; Alexander Bagaev; Arina Varlamova; Pavel Ovcharov - Web page: https://patentgrep.com/patent/US20210279866A1 ## CPC - G06T 7/0012: {Biomedical image inspection} - G06F 18/22: Matching criteria, e.g. proximity measures - G06F 18/2323: based on graph theory, e.g. minimum spanning trees [MST] or graph cuts - G06F 18/2431: Multiple classes - G06T 7/11: Region-based segmentation - G06T 7/174: involving the use of two or more images - G06T 7/194: involving foreground-background segmentation - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/761: {Proximity, similarity or dissimilarity measures} - G06V 10/7635: {based on graphs, e.g. graph cuts or spectral clustering} - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 20/695: {Preprocessing, e.g. image segmentation} - G06V 20/698: {Matching; Classification} - G06T 2207/10056: Microscopic image - G06T 2207/10064: Fluorescence image - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Techniques for processing multiplexed immunofluorescence (MxIF) images. The techniques include: obtaining at least one MxIF image of a same tissue sample; obtaining information indicative of locations of cells in the at least one MxIF image; identifying multiple groups of cells in the at least one MxIF image at least in part by: determining feature values for at least some of the cells using the at least one MxIF image and the information indicative of locations of the at least some cells in the at least one MxIF image; and grouping the at least some of the cells into the multiple groups using the determined feature values; and determining at least one characteristic of the tissue sample using the multiple cell groups. ## Claims ### Claim 1 A method, comprising: using at least one computer hardware processor to perform: obtaining at least one multiplexed immunofluorescence (MxIF) image of a same tissue sample; obtaining, using a machine learning technique, information indicative of locations of cells in the at least one MxIF image; identifying multiple groups of cells in the at least one MxIF image at least in part by: determining feature values for at least some of the cells using the at least one MxIF image and the information indicative of locations of cells in the at least one MxIF image; and grouping the at least some of the cells into the multiple groups using the determined feature values; and determining at least one characteristic of the tissue sample using the multiple groups. ### Claim 2 (depends on 1) The method of claim 1, wherein the at least some of the cells include a first cell, and wherein determining the feature values comprises determining first feature values for the first cell using at least one pixel value associated with a location of the first cell in the at least one MxIF image. ### Claim 3 (depends on 2) The method of claim 2, wherein determining the first feature values for the first cell comprises using pixel values associated with respective locations of the first cell in multiple channels of the at least one MxIF image. ### Claim 4 (depends on 3) The method of claim 3, wherein: the multiple channels are associated with respective markers in a plurality of markers; determining the first feature values using the pixel values associated with the respective locations of the first cell in the multiple channels of the at least one MxIF image comprises determining a marker expression signature that includes, for each particular marker of one or more of the plurality of markers, a likelihood that the particular marker is expressed in the first cell; and grouping the at least some of the cells into the multiple groups comprises: determining a predicted cell type for the first cell using the marker expression signature and cell typing data; and associating the first cell with one of the multiple groups based on the predicted cell type. ### Claim 5 (depends on 4) The method of claim 4, wherein: the cell typing data comprises at least one marker expression signature for each of a plurality of cell types; and the at least one marker expression signature for each particular cell type of the plurality of cell types comprises data indicative of which of the plurality of markers is expressed in cells of the particular cell type. ### Claim 6 (depends on 4) The method of claim 4, further comprising: providing, as input to a first trained neural network configured to determine marker expression signatures: (a) the at least one MxIF image and (b) information, from the information indicative of the locations of the cells, indicative of a location of the first cell in at least some of the multiple channels; and obtaining the marker expression signature as output from the first trained neural network. ### Claim 7 (depends on 5) The method of claim 5, wherein determining the predicted cell type of the first cell comprises comparing the marker expression signature of the first cell with the at least one marker expression signature for each of the plurality of cell types to determine the predicted cell type. ### Claim 8 (depends on 3) The method of claim 3, wherein using pixel values associated with respective locations of the first cell in the multiple channels comprises, for each of the multiple channels: identifying a set of pixels for the first cell using the location of the first cell in the channel; and determining a feature value of the first feature values for the first cell based on values of pixels in the set of pixels by determining an average pixel intensity of the values of the pixels in the set of pixels. ### Claim 9 (depends on 1) The method of claim 1, wherein determining the at least one characteristic comprises determining information about cell types in the tissue sample. ### Claim 10 (depends on 9) The method of claim 9, wherein determining the information about the cell types comprises identifying cell types of individual cells of the at least some of the cells in the at least one MxIF image, wherein the cell types comprise one or more of endothelial cells, epithelial cells, macrophages, T cells, malignant cells, NK cells, B cells, and acini cells. ### Claim 11 (depends on 1) The method of claim 1, wherein determining the at least one characteristic comprises determining at least one of: an acini mask indicating locations of acini in the at least one MxIF image; a stromal mask indicating locations of stroma in the at least one MxIF image; a tumor mask indicating locations of a tumor in the at least one MxIF image; and a cell cluster mask indicating locations of one or more cell clusters in the at least one MxIF image. ### Claim 12 (depends on 1) The method of claim 1, wherein determining the at least one characteristic comprises determining one or more cell clusters in the tissue sample based on the multiple groups comprising: determining a first set of cell features for each of the at least some cells by generating a graph comprising a node for each of the at least some cells, and edges between the nodes; and determining the one or more cell clusters based on the first set of cell features of the at least some cells. ### Claim 13 (depends on 12) The method of claim 12, wherein determining the one or more cell clusters in the tissue sample comprises: providing the graph as input to a graph neural network to obtain embedded features in a latent space; clustering the embedded features to obtain clusters of the nodes; and using the clusters of the nodes to determine the one or more cell clusters. ### Claim 14 (depends on 12) The method of claim 12, further comprising determining the first set of cell features for each node of the graph based on a group of the multiple groups that includes the cell, lengths of edges of the node in the graph, and mask data. ### Claim 15 (depends on 13) The method of claim 13, wherein the trained graph neural network comprises one or more convolutional layers including at least a last graph convolutional layer, and the embedded features are generated based on activations of the last graph convolutional layer. ### Claim 16 (depends on 1) The method of claim 1, wherein obtaining the at least one MxIF image of the tissue sample comprises: performing background subtraction on a first channel of the at least one MxIF image. ### Claim 17 (depends on 16) The method of claim 16, wherein performing the background subtraction comprises providing the first channel as input to a second trained neural network configured to perform background subtraction, wherein the second trained neural network comprises at least one million parameters. ### Claim 18 (depends on 1) The method of claim 1, wherein: the at least one MxIF image comprises a plurality of immunofluorescence images; and obtaining the information indicative of the locations of the cells in the at least one immunofluorescence image comprises analyzing the plurality of immunofluorescence images to identify one or more damaged portions of the tissue sample. ### Claim 19 (depends on 18) The method of claim 18, wherein analyzing the plurality of immunofluorescence images comprises: inputting a corresponding portion of each of at least two of the plurality of immunofluorescence images to a third trained neural network configured to identify differences between immunofluorescence images, wherein the at least two immunofluorescence images comprise images of a same marker; and obtaining from the third trained neural network at least one classification of the portion of the tissue sample. ### Claim 20 (depends on 19) The method of claim 19, wherein the at least one classification comprises a set of classifications comprising a first classification of no cells, a second classification of undamaged cells and/or a third classification of damaged cells; wherein the set of classifications comprises, for each classification, an associated confidence of the classification applying to the portion; and selecting, based on the confidences, a final classification from the set of classifications for the portion. ### Claim 21 (depends on 20) The method of claim 20, further comprising: generating a patch mask indicating the final classification of the portion and final classifications for a plurality of other portions of the at least two immunofluorescence images; removing, based on the patch mask, a portion of a segmentation mask for the tissue sample so that cells of the tissue sample associated with the removed portion are not included in the multiple groups. ### Claim 22 (depends on 1) The method of claim 1, wherein obtaining the information indicative of the locations of the cells in the at least one MxIF image using the machine learning technique comprises: applying a fourth trained neural network to at least one channel of the at least one MxIF image to generate the information indicative of the locations of the cells. ### Claim 23 (depends on 22) The method of claim 22, wherein: the fourth trained neural network is implemented using a U-Net architecture or a region-based convolutional neural network architecture; and the fourth trained neural network comprises at least one million parameters. ### Claim 24 (depends on 22) The method of claim 22, further comprising: training the fourth trained neural network using a set of training immunofluorescence images of tissue samples as input images and associated output images comprising information indicative of locations of cells in the input images. ### Claim 25 (depends on 1) The method of claim 1, wherein grouping the at least some of the cells into the multiple groups comprises performing cell clustering to group the at least some of the cells into the multiple groups based on the determined feature values. ### Claim 26 (depends on 1) The method of claim 1, wherein grouping the at least some of the cells into the multiple groups comprises: analyzing the determined feature values to determine relationships among the at least some of the cells; and determining the multiple groups based on the determined relationships such that each cell in a group of the multiple groups has feature values that are indicative of a relationship among cells in the group. ### Claim 27 A system, comprising: at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform: obtaining at least one multiplexed immunofluorescence (MxIF) image of a same tissue sample; obtaining, using a machine learning technique, information indicative of locations of cells in the MxIF image; identifying multiple groups of cells in the at least one MxIF image at least in part by: determining feature values for at least some of the cells using the at least one MxIF image and the information indicative of locations of cells; and grouping the at least some of the cells into the multiple groups using the determined feature values; and determining at least one characteristic of the tissue sample using the multiple groups. ### Claim 28 At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform: obtaining at least one multiplexed immunofluorescence (MxIF) image of a same tissue sample; obtaining, using a machine learning technique, information indicative of locations of cells in the MxIF image; identifying multiple groups of cells in the at least one MxIF image at least in part by: determining feature values for at least some of the cells using the at least one MxIF image and the information indicative of locations of cells in the at least one MxIF image; and grouping the at least some of the cells into the multiple groups using the determined feature values; and determining at least one characteristic of the tissue sample using the multiple groups. ### Claim 29 A method, comprising: using at least one computer hardware processor to perform: obtaining at least one MxIF image of a tissue sample; obtaining, using a machine learning technique, information indicative of a location of at least one cell in the at least one MxIF image; determining a marker expression signature for the cell in the tissue sample based on a plurality of markers expressed in the at least one MxIF image and the information indicative of the location of the cell in the at least one MxIF image; and comparing the marker expression signature to cell typing data that comprises at least one marker expression signature for a plurality of different types of cells to determine a cell type for the cell. ### Claim 30 (depends on 29) The method of claim 29, further comprising determining at least one characteristic of the tissue sample based on the cell type of the cell.