# US 11,308,355 B2 · Methods and apparatus for classification

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,355 B2
- Document ID: US11308355B2
- Status: Granted
- Kind: B2
- Application number: 16/882,487
- Filed: 2020-05-24
- Published: 2020-09-10
- Granted: 2022-04-19
- Earliest claimed: 2017-08-18
- Assignees: Massachusetts Institute of Technology
- Applicants: Massachusetts Institute of Technology
- Inventors: Pratik Shah; Gregory Yauney
- Web page: https://patentgrep.com/patent/US11308355B2

## CPC

- G06V 10/764: using classification, e.g. of video objects
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2178: {based on feedback of a supervisor}
- G06F 18/2413: based on distances to training or reference patterns
- G06F 18/251: {of input or preprocessed data}
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/7784: {based on feedback from supervisors}
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A human expert may initially label a white light image of teeth, and computer vision may initially label a filtered fluorescent image of the same teeth. Each label may indicate presence or absence of dental plaque at a pixel. The images may be registered. For each pixel of the registered images, a union label may be calculated, which is the union of the expert label and computer vision label. The union labels may be applied to the white light image. This process may be repeated to create a training set of union-labeled white light images. A classifier may be trained on this training set. Once trained, the classifier may classify a previously unseen white light image, by predicting union labels for that image. Alternatively, the items that are initially labeled may comprise images captured by two different imaging modalities, or may comprise different types of sensor measurements.

## Claims

### Claim 1

A method comprising:
(a) creating a set of union-labeled modality A images in such a way that each union-labeled image in the set is created by
(i) capturing a modality A image,
(ii) capturing a modality B image,
(iii) applying initial labels to the modality A image on a region-by-region basis,
(iv) applying initial labels to the modality B image on a region-by-region basis,
(v) performing registration of the modality A image and modality B image, and
(vi) after the registration, computing union labels for the modality A image in such a way that, for each specific region in a group of regions of the modality A image, a union label for the specific region is a union of (A) the initial label for the specific region and (B) the initial label for a corresponding region of the modality B image;
(b) training a classifier on at least the set of union-labeled modality A images; and
(c) after the training, calculating, with the classifier, labels for a previously unseen modality A image, on a region-by-region basis.

### Claim 2 (depends on 1)

The method of claim 1, wherein each union-labeled image in the set captures a different physical object or different portion of a physical object than that which is captured in each other union-labeled image in the set.

### Claim 3 (depends on 1)

The method of claim 1, wherein, for each particular union-labeled modality A image in the set:
(a) the particular union-labeled modality A image is an image of tissue before the tissue has been stained, and
(b) a modality B image, which corresponds to the union-labeled modality A image, is an image of the tissue after the tissue has been stained.

### Claim 4 (depends on 1)

The method of claim 1, wherein, for each particular union-labeled modality A image in the set:
(a) the particular union-labeled modality A image is an image of tissue before the tissue has been stained by hematoxylin and eosin (H&E) stain, and
(b) a modality B image, which corresponds to the union-labeled modality A image, is an image of the tissue after the tissue has been stained by H&E stain.

### Claim 5 (depends on 1)

The method of claim 1, wherein each modality A image and each modality B image is a tomographic image.

### Claim 6 (depends on 1)

The method of claim 1, wherein:
(a) each modality A image is a positron emission tomography image; and
(b) each modality B image is a magnetic resonance imaging image.

### Claim 7 (depends on 1)

The method of claim 1, wherein:
(a) each modality A image is a positron emission tomography image; and
(b) each modality B image is an x-ray computed tomography image.

### Claim 8 (depends on 1)

The method of claim 1, wherein each modality A image is a white light image and each modality B image is a fluorescent image.

### Claim 9 (depends on 1)

The method of claim 1, wherein each modality A image and each modality B image is an image of all or a portion of one or more teeth.

### Claim 10 (depends on 1)

The method of claim 1, wherein each initial label and each union label indicates either presence of, or absence of, dental plaque.

### Claim 11 (depends on 1)

The method of claim 1, wherein each initial label and each union label indicates either presence of, or absence of, gingivitis.

### Claim 12 (depends on 1)

The method of claim 1, wherein:
(a) each initial label of a modality A image is inputted by a human being; and
(b) each initial label of a modality B image is generated by a computer vision algorithm.

### Claim 13 (depends on 1)

The method of claim 1, wherein each region mentioned in claim 1 consists of only a single pixel.

### Claim 14 (depends on 1)

The method of claim 1, wherein each region mentioned in claim 1 consists of multiple pixels.

### Claim 15

A system comprising:
(a) a modality A imaging device;
(b) a modality B imaging device; and
(c) one or more computers;
wherein
(i) the one or more computers are programmed to acquire a set of union-labeled modality A images, in such a way that for each specific union-labeled image in the set, the one or more computers are programmed
(A) to output an instruction for the modality A imaging device to capture a modality A image,
(B) to output an instruction for the modality B imaging device to capture a modality B image,
(C) to apply initial labels to the modality A image on region-by-region basis,
(D) to apply initial labels to the modality B image on a region-by-region basis,
(E) to perform registration of the modality A image and modality B image, and
(F) to compute, after the registration, union labels for the modality A image in such a way that, for each specific region in a group of regions of the modality A image, a union label for the specific region is a union of (I) the initial label for the specific region and (II) the initial label for a corresponding region of the modality B image, and
(ii) the one or more computers are also programmed
(A) to train a classifier on the set of union-labeled modality A images; and
(B) after the training, to calculate, with the classifier, labels for a previously unseen modality A image, on a region-by-region basis.

### Claim 16 (depends on 15)

The system of claim 15, wherein:
(a) each modality A image is a white light image and each modality B image is a fluorescent image; and
(b) each initial label and union label indicates either presence of, or absence of, dental plaque.

### Claim 17 (depends on 15)

The system of claim 15, wherein each union-labeled image in the set captures a different physical object or different region of a physical object than that which is captured in each other union-labeled image in the set.

### Claim 18 (depends on 15)

The system of claim 15, wherein, for each particular union-labeled modality A image in the set:
(a) the particular union-labeled modality A image is an image of tissue before the tissue has been stained, and
(b) a modality B image, which corresponds to the union-labeled modality A image, is an image of the tissue after the tissue has been stained.

### Claim 19 (depends on 15)

The method of claim 15, wherein the one or more computers are also programmed:
(a) to accept input from one or more human beings, which input specifies the initial labels for the modality A images;
(b) to generate, by a computer vision algorithm, the initial labels for the modality B images.

### Claim 20 (depends on 15)

The system of claim 15 wherein the modality A imaging device and the modality B imaging device are each a tomographic imaging device.

### Claim 21 (depends on 15)

The system of claim 15, wherein:
(a) each modality A image is a white light image and each modality B image is a fluorescent image; and
(b) each initial label and union label indicates either presence of, or absence of, gingivitis.