# US 2025/0166178 A1 · Techniques for tertiary lymphoid structure (tls) detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0166178 A1
- Document ID: US20250166178A1
- Status: Published application
- Kind: A1
- Application number: 18/918,401
- Filed: 2024-10-17
- Published: 2025-05-22
- Earliest claimed: 2023-11-20
- Assignees: BostonGene Corporation
- Applicants: BostonGene Corporation
- Inventors: Daniil Dymov; Vladimir Kushnarev; Nadezhda Lukashevich; Viktor Svekolkin; Aleksandr Sarachakov
- Web page: https://patentgrep.com/patent/US20250166178A1

## CPC

- G06T 7/0012: {Biomedical image inspection}
- G06T 5/20: using local operators
- G06T 7/12: Edge-based segmentation
- G06T 7/13: Edge detection
- G06T 7/62: of area, perimeter, diameter or volume
- G06V 10/46: Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features (colour feature extraction G06V10/56)
- G06V 10/82: using neural networks
- G16H 20/00: ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
- G06T 2207/10056: Microscopic image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G06T 2207/30096: Tumor; Lesion
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

Described herein are techniques for identifying at least one tertiary lymphoid structure (TLS) in an image of tissue. In some embodiments, the techniques include: obtaining a TLS mask indicating, for each particular pixel of multiple pixels of the image, a value indicative of a likelihood that the particular pixel is part of the at least one TLS; processing at least a portion of the image using a trained neural network model to obtain a tumor infiltrating lymphocyte (TIL) mask indicating, for each particular pixel of pixels of at least the portion of the image, a value indicative of a likelihood that the particular pixel is part of a TIL; identifying boundaries of the at least one TLS using the TLS mask and the TIL mask; and identifying a characteristic of the at least one TLS using the boundaries of the at least one TLS.

## Claims

### Claim 1

A method for identifying at least one tertiary lymphoid structure (TLS) in an image of tissue previously-obtained from a subject, the method comprising:
using at least one computer hardware processor to perform:
obtaining a TLS mask indicating, for each particular pixel of multiple pixels of the image, a respective numeric value indicative of a likelihood that the particular pixel is part of the at least one TLS;
processing at least a portion of the image using a trained neural network model to obtain a tumor infiltrating lymphocyte (TIL) mask indicating, for each particular pixel of pixels of at least the portion of the image, a respective numeric value indicative of a likelihood that the particular pixel is part of a TIL, wherein the trained neural network model is trained to predict, for the particular pixel, the respective numeric value indicative of the likelihood that the particular pixel is part of the TIL;
identifying boundaries of the at least one TLS in the image using the TLS mask and the TIL mask; and
identifying one or more characteristics of the at least one TLS using the boundaries of the at least one TLS in the image.

### Claim 2 (depends on 1)

The method of claim 1, wherein identifying the boundaries of the at least one TLS in the image comprises:
identifying the boundaries based on an overlap between the TLS mask and the TIL mask, the identifying comprising:
identifying one or more pixels for which the TIL mask indicates a respective one or more numeric values that are greater than or equal to a first threshold and for which the TLS mask indicates a respective one or more numeric values that are greater than or equal to a second threshold.

### Claim 3 (depends on 2)

The method of claim 2, wherein identifying the one or more pixels for which the TLS mask indicates the respective one or more numeric values that are greater than or equal to the second threshold comprises:
generating a binary version of the TLS mask; and
identifying contours of at least one portion of the image by applying a border-following algorithm to the binary version of the TLS mask, wherein the one or more pixels are positioned within the contours of the at least one portion of the image.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
obtaining a set of sub-images of the image of the tissue,
wherein processing the image using the trained neural network model to obtain the TIL mask comprises:
processing the set of sub-images using the trained neural network model to obtain a respective set of sub-image masks, each sub-image mask in the set of sub-image masks indicating, for a respective sub-image, a respective numeric value indicative of a likelihood that pixels in the respective sub-image are part of a TIL; and
generating the TIL mask using the set of sub-image masks corresponding to the set of sub-images of the image of the tissue.

### Claim 5 (depends on 1)

The method of claim 1, wherein the trained neural network model is a TIL neural network model, and wherein obtaining the TLS mask comprises processing the image using a TLS neural network model to obtain the TLS mask, the method further comprising:
obtaining a set of overlapping sub-images of the image of the tissue,
wherein processing the image using the TLS neural network model to obtain the TLS mask comprises:
processing the set of overlapping sub-images using the TLS neural network model to obtain a respective set of pixel-level sub-image masks, each pixel-level sub-image mask in the set of pixel-level sub-image masks indicating, for each particular pixel of multiple individual pixels in a respective particular sub-image, a respective probability that the particular pixel is part of a TLS; and
generating the TLS mask using the set of pixel-level sub-image masks corresponding to the set of overlapping sub-images of the image of the tissue.

### Claim 6 (depends on 1)

The method of claim 1, wherein identifying the boundaries of the at least one TLS in the image comprises:
identifying respective boundaries for a plurality of TLSs in the image; and
applying a filter to the respective boundaries identified for the plurality of TLSs in the tissue to obtain the boundaries of the at least one TLS in the image, the applying comprising, for each particular TLS of the plurality of TLSs:
determining a size of the particular TLS in the image;
determining whether the size is greater than or equal to a threshold; and
filtering out the respective boundaries of the particular TLS after determining that the size is not greater than or equal to the threshold.

### Claim 7 (depends on 1)

The method of claim 1, wherein identifying the one or more characteristics of the at least one TLS comprises identifying at least one characteristic selected from the group consisting of: a number of TLSs in at least the portion of the image, the number of TLSs in at least the portion of the image normalized by an area of at least the portion of the image, a total area of TLSs in at least the portion of the image, the total area of the TLSs in at least the portion of the image normalized by the area of at least the portion of the image, median area of TLSs in at least the portion of the image, and the median area of the TLSs in at least the portion of the image normalized by the area of at least the portion of the image.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
identifying boundaries of at least one TIL in the image using the TIL mask; and
identifying one or more characteristics of the at least one TIL using the boundaries of the at least one TIL, the identifying comprising:
determining a total area of TILs in at least the portion of the image and/or
determining the total area of the TILs in at least the portion of the image normalized by an area of at least the portion of the image.

### Claim 9 (depends on 1)

The method of claim 1,
wherein identifying the boundaries of the at least one TLS in the image comprises identifying the boundaries of the at least one TLS using the TLS mask, the TIL mask, and at least one other feature in the image,
wherein the at least one other feature in the image comprises at least one feature selected from the group consisting of: tissue in the image, tumor tissue in the image, and non-tumor tissue in the image, and
wherein identifying the one or more characteristics of the at least one TLS comprises identifying a total area of the at least one TLS in the tumor tissue in the image and/or identifying a number of the at least one TLS in the tumor tissue in the image.

### Claim 10 (depends on 1)

The method of claim 1,
wherein the image of the tissue is a whole slide image (WSI),
wherein the image is a three-channel image comprising at least 10,000 by 10,000 pixel values per channel, and
wherein the trained neural network model comprises at least 10 million, at least 25 million, at least 50 million, or at least 100 million parameters.

### Claim 11 (depends on 1)

The method of claim 1, wherein the trained neural network model comprises:
a deconvolution neural network portion;
an adapter neural network portion having an input coupled to an output of the deconvolution neural network portion; and
a classification neural network portion having an input coupled to an output of the adapter neural network portion.

### Claim 12 (depends on 1)

The method of claim 1, wherein the trained neural network model is a TIL neural network model, and wherein obtaining the TLS mask comprises processing the image using a TLS neural network model.

### Claim 13 (depends on 12)

The method of claim 12, wherein the TLS neural network model comprises at least 10 million, at least 25 million, at least 50 million, or at least 100 million parameters.

### Claim 14 (depends on 13)

The method of claim 13, wherein the TLS neural network model comprises an encoder sub-model, a decoder sub-model, and an auxiliary classifier sub-model.

### Claim 15 (depends on 1)

The method of claim 1,
wherein the subject has, is suspected of having, or is at risk of having cancer, and
wherein the cancer is lung adenocarcinoma, breast cancer, cervical squamous cell carcinoma, lung squamous cell carcinoma, head and neck squamous cell carcinoma, gastric adenocarcinoma, colorectal adenocarcinoma, liver adenocarcinoma, pancreatic adenocarcinoma, or melanoma.

### Claim 16 (depends on 1)

The method of claim 1, further comprising:
determining, based on the one or more characteristics of the at least one TLS, to administer an immunotherapy to the subject.

### Claim 17 (depends on 16)

The method of claim 16, further comprising:
administering the immunotherapy to the subject.

### Claim 18 (depends on 17)

The method of claim 17, wherein administering the immunotherapy to the subject comprises administering pembrolizumab, nivolumab, atezolizumab, or durvalumab.

### Claim 19

A system, comprising:
at least one computer hardware processor; and
at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform a method for identifying at least one tertiary lymphoid structure (TLS) in an image of tissue previously-obtained from a subject, the method comprising:
obtaining a TLS mask indicating, for each particular pixel of multiple pixels of the image, a respective numeric value indicative of a likelihood that the particular pixel is part of the at least one TLS;
processing at least a portion of the image using a trained neural network model to obtain a tumor infiltrating lymphocyte (TIL) mask indicating, for each particular pixel of pixels of at least the portion of the image, a respective numeric value indicative of a likelihood that the particular pixel is part of a TIL, wherein the trained neural network model is trained to predict, for the particular pixel, the respective numeric value indicative of the likelihood that the particular pixel is part of the TIL;
identifying boundaries of the at least one TLS in the image using the TLS mask and the TIL mask; and
identifying one or more characteristics of the at least one TLS using the boundaries of the at least one TLS in the image.

### Claim 20

At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform a method for identifying at least one tertiary lymphoid structure (TLS) in an image of tissue previously-obtained from a subject, the method comprising:
obtaining a TLS mask indicating, for each particular pixel of multiple pixels of the image, a respective numeric value indicative of a likelihood that the particular pixel is part of the at least one TLS;
processing at least a portion of the image using a trained neural network model to obtain a tumor infiltrating lymphocyte (TIL) mask indicating, for each particular pixel of pixels of at least the portion of the image, a respective numeric value indicative of a likelihood that the particular pixel is part of a TIL, wherein the trained neural network model is trained to predict, for the particular pixel, the respective numeric value indicative of the likelihood that the particular pixel is part of the TIL;
identifying boundaries of the at least one TLS in the image using the TLS mask and the TIL mask; and
identifying one or more characteristics of the at least one TLS using the boundaries of the at least one TLS in the image.
