# US 12,008,825 B2 · Classification workflow for flexible image based particle sorting

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,825 B2
- Document ID: US12008825B2
- Status: Granted
- Kind: B2
- Application number: 17/531,124
- Filed: 2021-11-19
- Published: 2022-05-19
- Granted: 2024-06-11
- Earliest claimed: 2020-11-19
- Assignees: Sony Group Corporation; Sony Corporation of America
- Applicants: Sony Group Corporation; Sony Corporation of America
- Inventors: Ming-Chang Liu; Michael Zordan; Ko-Kai Albert Huang; Su-Hui Chiang; Liyu Gong
- Web page: https://patentgrep.com/patent/US12008825B2

## CPC

- G06V 20/698: {Matching; Classification}
- G01N 15/1429: Signal processing
- G01N 15/1433: using image recognition
- G01N 15/149: specially adapted for sorting particles, e.g. by their size or optical properties
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/23: Clustering techniques
- G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines
- G06V 10/762: using clustering, e.g. of similar faces in social networks
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G01N 2015/1006: {for cytology}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An image-based classification workflow uses unsupervised clustering to help a user identify subpopulations of interest for sorting. Labeled cell images are used to fine-tune the supervised classification network for a specific experiment. The workflow allows the user to select the populations to sort in the same manner for a variety of applications. The supervised classification network is very fast, allowing it to make real-time sort decisions as the cell travels through a device. The workflow is more automated and has fewer user steps, which improves the ease of use. The workflow uses machine learning to avoid human error and bias from manual gating. The workflow does not require the user to be an expert in image processing, thus increasing the ease of use.

## Claims

### Claim 1

A method comprising:
pre-training a feature encoder using cell images;
performing unsupervised clustering to identify populations, wherein the unsupervised clustering receives output from pre-training the feature encoder, wherein performing the unsupervised clustering includes classifying cells of the cell images into clusters, wherein a user labels the clusters after the unsupervised clustering based on viewing the cluster and representative information, wherein clusters are labeled as “sort” or “do not sort;”
implementing a classifier to fine-tune supervised classification; and
performing real-time classification of cells during active sorting using the classifier, wherein the real-time classification processes an entire image of each cell.

### Claim 2 (depends on 1)

The method of claim 1 wherein the feature encoder detects and measures feature values from the cell images.

### Claim 3 (depends on 1)

The method of claim 1 wherein the feature encoder is implemented using a neural network.

### Claim 4 (depends on 1)

The method of claim 1 wherein the feature encoder is scalable to accommodate 1 to 12 image channels.

### Claim 5 (depends on 1)

The method of claim 1 further comprising manually or automatically determining which population of cells to sort based on results of the unsupervised clustering.

### Claim 6 (depends on 1)

The method of claim 1 wherein classifier results from the unsupervised clustering are used by the classifier to fine-tune a convolutional neural network.

### Claim 7 (depends on 1)

The method of claim 1 wherein the classifier is configured to be re-trained for each experiment.

### Claim 8

An apparatus comprising:
a non-transitory memory for storing an application, the application for:
pre-training a feature encoder using cell images;
performing unsupervised clustering to identify populations, wherein the unsupervised clustering receives output from pre-training the feature encoder, wherein performing the unsupervised clustering includes classifying cells of the cell images into clusters, wherein a user labels the clusters after the unsupervised clustering based on viewing the cluster and representative information, wherein clusters are labeled as “sort” or “do not sort;”
implementing a classifier to fine-tune supervised classification; and
performing real-time classification of cells during active sorting using the classifier, wherein the real-time classification processes an entire image of each cell; and
a processor coupled to the memory, the processor configured for processing the application.

### Claim 9 (depends on 8)

The apparatus of claim 8 wherein the feature encoder detects and measures feature values from the cell images.

### Claim 10 (depends on 8)

The apparatus of claim 8 wherein the feature encoder is implemented using a neural network.

### Claim 11 (depends on 8)

The apparatus of claim 8 wherein the feature encoder is scalable to accommodate 1 to 12 image channels.

### Claim 12 (depends on 8)

The apparatus of claim 8 further automatically determining which population of cells to sort based on results of the unsupervised clustering.

### Claim 13 (depends on 8)

The apparatus of claim 8 wherein classifier results from the unsupervised clustering are used by the classifier to fine-tune a convolutional neural network.

### Claim 14 (depends on 8)

The apparatus of claim 8 wherein the classifier is configured to be re-trained for each experiment.

### Claim 15

A system comprising:
a first device configured for acquiring cell images; and
a second device configured for:
pre-training a feature encoder using the cell images;
performing unsupervised clustering to identify populations, wherein the unsupervised clustering receives output from pre-training the feature encoder, wherein performing the unsupervised clustering includes classifying cells of the cell images into clusters, wherein a user labels the clusters after the unsupervised clustering based on viewing the cluster and representative information, wherein clusters are labeled as “sort” or “do not sort;”
implementing a classifier to fine-tune supervised classification; and
performing real-time classification of cells during active sorting using the classifier, wherein the real-time classification processes an entire image of each cell.

### Claim 16 (depends on 15)

The system of claim 15 wherein the feature encoder detects and measures feature values from the cell images.

### Claim 17 (depends on 15)

The system of claim 15 wherein the feature encoder is implemented using a neural network.

### Claim 18 (depends on 15)

The system of claim 15 wherein the feature encoder is scalable to accommodate 1 to 12 image channels.

### Claim 19 (depends on 15)

The system of claim 15 further comprising manually or automatically determining which population of cells to sort based on results of the unsupervised clustering.

### Claim 20 (depends on 15)

The system of claim 15 wherein classifier results from the unsupervised clustering are used by the classifier to fine-tune a convolutional neural network.

### Claim 21 (depends on 15)

The system of claim 15 wherein the classifier is configured to be re-trained for each experiment.