# US 2024/0203605 A1 · Convolutional neural network for subvisible particulate classification of biopharmaceuticals

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0203605 A1
- Document ID: US20240203605A1
- Status: Published application
- Kind: A1
- Application number: 18/390,291
- Filed: 2023-12-20
- Published: 2024-06-20
- Earliest claimed: 2022-12-20
- Assignees: Merck Sharp & Dohme Corp.
- Applicants: Merck Sharp & Dohme LLC
- Inventors: Shubing Wang; Andy I-Hsuen Liaw; Yue-Ming Chen; Yongchao Su; Daniel Skomski
- Web page: https://patentgrep.com/patent/US20240203605A1

## CPC

- G16H 70/40: relating to drugs, e.g. their side effects or intended usage
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06T 7/0012: {Biomedical image inspection}
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An analysis system implements a particulate classification model trained as a convolutional neural network to assess sterile formulation quality. The system obtains micro flow imaging (MFI) data for a sterile formulation drug product, wherein the MFI data includes an image depicting a plurality of sub-visible particulates in the sterile formulation drug product. The system detects a plurality of features in the image, wherein each feature is bounded by a bounding box. The system performs preprocessing to determine and remove one or more of features to be artifacts based on characteristics of the features, wherein the remaining features are determined to be sub-visible particulates. The system applies a particulate classification model to each sub-visible particulate to determine a particulate classification label. The system determines a quality metric for the sterile formulation drug product based on a count of sub-visible particulates in each particulate classification label.

## Claims

### Claim 1

A computer-implemented method of sub-visible particle classification, the method comprising:
obtaining imaging data for a sterile formulation drug product, wherein the imaging data includes an image depicting a plurality of sub-visible particulates in the sterile formulation drug product;
detecting a plurality of features in the image, wherein each feature is bounded by a bounding box;
determining one or more of features to be artifacts based on characteristics of the features;
removing the artifacts from the plurality of features to generate a set of sub-visible particulates;
applying a particulate classification model to each sub-visible particulate to determine a particulate classification label, wherein the particulate classification model is a convolutional neural network; and
determining a quality metric for the sterile formulation drug product based on a count of sub-visible particulates in each particulate classification label.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein detecting the plurality of features comprises:
determining a background signal in the image covering a majority of pixels in the image; and
detecting features as pixels having a contrast to the background signal.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein detecting the plurality of features further comprises:
aggregating adjacent features into a single feature based on a distance of the two features being below a threshold distance.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein determining the one or more of features to be artifacts comprises:
determining an aspect ratio of the bounding box of each feature; and
determining a first feature to be a stripe-type artifact with the aspect ratio being below a threshold aspect ratio.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein determining the one or more features to be artifacts comprises:
determining a position of each feature in the image; and
determining a first feature to be a dirty-lens-type artifact with the position matching a position of a prior feature in a prior image in the imaging data.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the particulate classification labels include: air bubble, silicone oil, air-silicone hybrid, protein aggregate, and background.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the particulate classification model is trained with training data comprising images of sub-visible particulates annotated with particulate classification labels.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the quality metric is based on a weight vector that scales each count of sub-visible particulates in each particulate classification label.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the particulate classification model is further configured to output a confidence score for each sub-visible particulate.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, further comprising:
returning the quality metric to a client device of a user and the counts of sub-visible particulates in the particulate classification labels.

### Claim 11

A non-transitory computer-readable storage medium storing instructions for sub-visible particle classification, the instructions that, when executed by a computer processor, cause the computer processor to perform operations comprising:
obtaining imaging data for a sterile formulation drug product, wherein the imaging data includes an image depicting a plurality of sub-visible particulates in the sterile formulation drug product;
detecting a plurality of features in the image, wherein each feature is bounded by a bounding box;
determining one or more of features to be artifacts based on characteristics of the features;
removing the artifacts from the plurality of features to generate a set of sub-visible particulates;
applying a particulate classification model to each sub-visible particulate to determine a particulate classification label, wherein the particulate classification model is a convolutional neural network; and
determining a quality metric for the sterile formulation drug product based on a count of sub-visible particulates in each particulate classification label.

### Claim 12 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein detecting the plurality of features comprises:
determining a background signal in the image covering a majority of pixels in the image; and
detecting features as pixels having a contrast to the background signal.

### Claim 13 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein detecting the plurality of features further comprises:
aggregating adjacent features into a single feature based on a distance of the two features being below a threshold distance.

### Claim 14 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein determining the one or more of features to be artifacts comprises:
determining an aspect ratio of the bounding box of each feature; and
determining a first feature to be a stripe-type artifact with the aspect ratio being below a threshold aspect ratio.

### Claim 15 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein determining the one or more features to be artifacts comprises:
determining a position of each feature in the image; and
determining a first feature to be a dirty-lens-type artifact with the position matching a position of a prior feature in a prior image in the imaging data.

### Claim 16 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the particulate classification labels include: air bubble, silicone oil, air-silicone hybrid, protein aggregate, and background.

### Claim 17 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the particulate classification model is trained with training data comprising images of sub-visible particulates annotated with particulate classification labels.

### Claim 18 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the quality metric is based on a weight vector that scales each count of sub-visible particulates in each particulate classification label.

### Claim 19 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the particulate classification model is further configured to output a confidence score for each sub-visible particulate.

### Claim 20 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, the operations further comprising:
returning the quality metric to a client device of a user and the counts of sub-visible particulates in the particulate classification labels.
