# US 2023/0066976 A1 · Systematic characterization of objects in a biological sample

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0066976 A1
- Document ID: US20230066976A1
- Status: Published application
- Kind: A1
- Application number: 17/795,755
- Filed: 2021-01-29
- Published: 2023-03-02
- Earliest claimed: 2020-01-30
- Assignees: Vitadx International
- Applicants: Vitadx International
- Inventors: Thibaut TROUDE; Florent COUZINIE-DEVY; Riadh FEZZANI; Marin SCALBERT; Shuxian LI
- Web page: https://patentgrep.com/patent/US20230066976A1

## CPC

- G01N 15/1433: using image recognition
- G06F 18/2414: {Smoothing the distance, e.g. radial basis function networks [RBFN]}
- G06T 7/0012: {Biomedical image inspection}
- G06V 10/457: {by analysing connectivity, e.g. edge linking, connected component analysis or slices}
- G06V 20/698: {Matching; Classification}
- G01N 2015/1006: {for cytology}
- G01N 2015/1486: {Counting the particles}
- G06T 2207/10056: Microscopic image
- G06T 2207/20081: Training; Learning
- G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method for classifying and counting objects recoverable from a urine sample processed onto a slide. The method includes the following steps: receiving at least one digitalized image of the whole slide; detecting connected components by segmentation of the image of the whole slide; classifying the detected connected components into countable connected components and uncountable connected components using a classifier; for the countable connected components using an object detection model to obtaining the number of objects for each class; for the uncountable components using a semantic segmentation model to obtaining the number of objects for each class; summing up the number of objects for each class obtained from the semantic segmentation model and the object detection model outputting a number of objects for each class.

## Claims

### Claim at position 1

-15. (canceled)

### Claim 16

A computer-implemented method for classifying and counting objects recoverable from a urine sample processed onto a slide, said method comprising:
receiving at least one digitalized image of the whole slide;
detecting connected components by segmentation of the image of the whole slide;
classifying the detected connected components into countable connected components and uncountable connected components using a classifier;
for the countable connected components:
inputting each countable connected component into an object detection model so as to detect objects and obtain an output comprising a bounding box and an associated class for each detected object;
counting the bounding boxes associated to each class obtaining a number of objects for each class;
for the uncountable components:
inputting each uncountable connected component into a semantic segmentation model and obtaining as output a segmentation mask in which all pixels are classified into one class among multiple predefined available classes;
for each class, counting the number of objects as the ratio between a total pixel's area of the class, obtained as the number of pixels of the segmentation mask associate to said class, and an average area of the object of said class;
summing up the number of objects for each class obtained from the semantic segmentation model and the object detection model;
outputting the number of objects for each class;
wherein said classes for the semantic segmentation model and the object detection model are the same.

### Claim 17 (depends on 16)

The method according to claim 16, wherein the semantic segmentation model is a U-Net.

### Claim 18 (depends on 16)

The method according to claim 16, wherein the object detection model is a Faster-RCNN, CenterNet, SOLO or a YOLO.

### Claim 19 (depends on 16)

The method according to claim 16, wherein the received digitalized image results from a brightfield optical system.

### Claim 20 (depends on 16)

The method according to claim 16, wherein the semantic segmentation model and the object detection model are trained using a dataset of labelled digitalized images.

### Claim 21 (depends on 20)

The method according to claim 20, wherein the semantic segmentation model and the object detection model are trained using a stochastic gradient descent training method.

### Claim 22 (depends on 16)

The method according to claim 16, wherein each class of the semantic segmentation model and the object detection model is associated to at least one of the objects in the following list:
leukocytes: basophils, neutrophils, macrophage, monocyte and eosinophil;
red blood cells;
bacteria;
urinary crystal;
cylinder;
healthy and atypical urothelial cells;
squamous cells;
reactive urothelial cells, and/or
yeast.

### Claim 23 (depends on 16)

The method according to claim 16, further comprising displaying the total count of objects for each class alongside to the digitalized image of at least one portion of the whole slide.

### Claim 24 (depends on 23)

The method according to claim 23, further comprising displaying at least one string of characters providing an information on the presence or absence of objects in a class with the digitalized image.

### Claim 25 (depends on 16)

The method according to claim 16, wherein the sample is colored by a Pap stain.

### Claim 26

A system for classifying and counting objects recoverable from a urine sample processed onto a slide, said system comprising:
at least one input adapted to receive at least one digitalized image of the whole slide comprising multiple objects;
at least one processor configured to:
detect connected components by segmentation of the image of the whole slide;
classify the detected connected components into countable connected components and uncountable connected components using a classifier;
for the countable connected components:
(i) input each countable connected component into an object detection model configured to detect objects and to output for each object one bounding box and an associated class;
(ii) count the bounding boxes associated to each class obtaining the number of objects for each class;
for the uncountable components:
(i) input each uncountable connected component into a semantic segmentation model and obtain as output a segmentation mask in which all pixels are classified into one class among the predefined available classes;
(ii) for each class, count the number of objects as the ratio between a total pixel's area of the class, obtained as the number of pixels of the segmentation mask associate to said class, and an average area of the object of said class;
summing up the number of objects for each class obtained from the semantic segmentation model and the object detection model;
wherein said classes for the semantic segmentation model and the object detection model are the same; and
at least one output adapted to provide a number of objects for each class.

### Claim 27 (depends on 26)

The system according to claim 26, wherein the object detection model is a Faster-RCNN, CenterNet, SOLO or a YOLO.

### Claim 28 (depends on 26)

The system according to claim 26, wherein the semantic segmentation model and the object detection model are trained using a dataset of labelled digitalized images.

### Claim 29 (depends on 26)

The system according to claim 26, wherein the semantic segmentation model and the object detection model are trained using a stochastic gradient descent training method.

### Claim 30

A computer program product for classifying and counting objects recoverable from a urine sample, the computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method comprising:
receiving at least one digitalized image of the whole slide;
detecting connected components by segmentation of the image of the whole slide;
classifying the detected connected components into countable connected components and uncountable connected components using a classifier;
for the countable connected components:
inputting each countable connected component into an object detection model so as to detect objects and obtain an output comprising a bounding box and an associated class for each detected object;
counting the bounding boxes associated to each class obtaining a number of objects for each class;
for the uncountable components:
inputting each uncountable connected component into a semantic segmentation model and obtaining as output a segmentation mask in which all pixels are classified into one class among multiple predefined available classes;
for each class, counting the number of objects as the ratio between a total pixel's area of the class, obtained as the number of pixels of the segmentation mask associate to said class, and an average area of the object of said class;
summing up the number of objects for each class obtained from the semantic segmentation model and the object detection model;
outputting the number of objects for each class;
wherein said classes for the semantic segmentation model and the object detection model are the same.

### Claim 31 (depends on 29)

The method according to claim 29, wherein the semantic segmentation model and the object detection model are trained using a dataset of labelled digitalized images.

### Claim 32

A non-transitory computer-readable storage medium comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method comprising
receiving at least one digitalized image of the whole slide;
detecting connected components by segmentation of the image of the whole slide;
classifying the detected connected components into countable connected components and uncountable connected components using a classifier;
for the countable connected components:
inputting each countable connected component into an object detection model so as to detect objects and obtain an output comprising a bounding box and an associated class for each detected object;
counting the bounding boxes associated to each class obtaining a number of objects for each class;
for the uncountable components:
inputting each uncountable connected component into a semantic segmentation model and obtaining as output a segmentation mask in which all pixels are classified into one class among multiple predefined available classes;
for each class, counting the number of objects as the ratio between a total pixel's area of the class, obtained as the number of pixels of the segmentation mask associate to said class, and an average area of the object of said class;
summing up the number of objects for each class obtained from the semantic segmentation model and the object detection model;
outputting the number of objects for each class;
wherein said classes for the semantic segmentation model and the object detection model are the same.

### Claim 33 (depends on 32)

The non-transitory computer-readable storage medium according to claim 32, wherein the semantic segmentation model and the object detection model are trained using a dataset of labelled digitalized images.
