# US 11,657,600 B2 · Method for assessing oral health using a mobile device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,657,600 B2
- Document ID: US11657600B2
- Status: Granted
- Kind: B2
- Application number: 17/765,795
- Filed: 2020-10-01
- Published: 2022-11-03
- Granted: 2023-05-23
- Earliest claimed: 2019-10-04
- Assignees: Adent ApS
- Applicants: Adent ApS
- Inventors: Richard Bundsgaard; Thøger Bundsgaard
- Web page: https://patentgrep.com/patent/US11657600B2

## CPC

- G06V 10/776: Validation; Performance evaluation
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- A61B 5/0022: {Monitoring a patient using a global network, e.g. telephone networks, internet}
- A61B 5/0077: {Devices for viewing the surface of the body, e.g. camera, magnifying lens}
- A61B 5/0088: {for oral or dental tissue}
- A61B 5/4547: {Evaluating teeth}
- A61B 5/4552: {Evaluating soft tissue within the mouth, e.g. gums or tongue}
- A61B 5/6898: {Portable consumer electronic devices, e.g. music players, telephones, tablet computers}
- A61B 5/7267: {involving training the classification device}
- A61B 5/7425: {Displaying combinations of multiple images regardless of image source, e.g. displaying a reference anatomical image with a live image}
- G06T 7/0012: {Biomedical image inspection}
- G06V 10/56: relating to colour
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/7796: {based on specific statistical tests}
- G06V 10/82: using neural networks
- G16H 30/40: for processing medical images, e.g. editing
- G16H 50/30: for calculating health indices; for individual health risk assessment
- G06T 2207/10024: Color image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30036: Dental; Teeth
- G06V 2201/033: of skeletal patterns
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method for remotely assessing oral health of a user of a mobile device by obtaining, using the mobile device (40), at least one digital image (1) of said user's (30) oral cavity (31) and additional non-image data (2) comprising anamnestic information about the user (30). The digital image (1) is processed both using a statistical object detection algorithm (20) to extract at least one local visual feature (3) corresponding to a medical finding related to a sub-region of said user's oral cavity (31); and also using a statistical image recognition algorithm (21) to extract at least one global classification label (4) corresponding to a medical finding related to said user's oral cavity (31) as a whole. An assessment (10) of the oral health of said user (30) is determined based on the local visual feature(s) (3), the global classification label(s) (4) and the non-image data (2).

## Claims

### Claim 1

A computer-implemented method for assessing oral health of a user of a mobile device to determine whether the user is a candidate for an oral treatment, the method comprising:
obtaining (101) at least one digital image (1) of the user's (30) oral cavity (31) using a camera (41) of the mobile device (40);
obtaining (102) non-image data (2) associated with the user (30) using the mobile device (40), the non-image data (2) comprising anamnestic information about the user;
processing (103) the at least one digital image (1) using a statistical object detection algorithm (20) executed on a processor of the mobile device (40) or a processor of a remote server (50) to calculate the likelihood of at least one local visual feature (3) corresponding to a medical finding related to a sub-region of the user's oral cavity (31);
processing (104) the at least one digital image (1) using a statistical image recognition algorithm (21) executed on a processor of the mobile device (40) or a processor of a remote server (50) to calculate the likelihood of at least one global classification label (4) corresponding to a medical finding related to the user's oral cavity (31);
processing (201) the non-image data (2) and the at least one local visual feature (3) using a statistical or non-statistical algorithm executed on a processor of the mobile device (40) or a processor of a remote server (50) to extract a set of priority classification labels (7) corresponding to a predefined number of prioritized medical findings of the user's oral cavity (31), and relative likelihoods of the priority classification labels (7) based on predefined conditions;
arranging (202) the set of priority classification labels (7) into a subset of priority classification labels (7A) of highest relative likelihoods;
using (203) the subset of priority classification labels (7A) as further input for at least one of the statistical object detection algorithm (20) or the statistical image recognition algorithm (21) to enhance the predictability of the at least one local visual feature (3) or the at least one global classification label (4);
determining (105) an assessment (10) of the oral health of the user (30) using a processor of the mobile device (40) or a processor of a remote server (50) by feeding (1051) the likelihood of the at least one local visual feature (3) and the likelihood of the at least one global classification label (4) to a synthesizing algorithm (24) arranged to map local visual features (3) to global classification labels (4) and to determine a set of concepts (9), each concept (9) comprising at least one global classification label (4) and any number of related local visual features (3); and
validating (1052) the set of concepts (9) either automatically or by a human dental reviewer (32,33) based on medical certainty, wherein the medical certainty is determined based on at least one of:
predefined threshold values for the likelihoods of the global classification labels (4) and related local visual features (3) comprised in each concept; and
predefined threshold values for validity scores associated with each statistical algorithm used for predicting the likelihoods.

### Claim 2 (depends on 1)

A method according to claim 1, wherein the at least one digital image (1) is at least one of an intraoral or extraoral high resolution color photograph, preferably in the RGB or RGBA color space; and wherein the method further comprises extracting at least one image derivative (1A,1B,1C) from the digital image (1) by at least one of:
processing (1011) the at least one digital image (1) to generate at least one scaled image (1A) by scaling the at least one digital image (1) to optimal dimensions;
processing (1012) the at least one digital image (1) to extract at least one image patch (1B), each image patch (1B) comprising visual information of at least one tooth and its surrounding region within the oral cavity (31); and
processing (1013) the at least one digital image (1) to extract at least one image vector (1C), each image vector (1C) representing the digital image (1) in a reduced vector space; wherein processing the at least one digital image (1) using a statistical algorithm comprises feeding the at least one image derivative (1A,1B,1C) to a version of the statistical algorithm optimized for the respective image derivative (1A,1B,1C) input; and wherein
any classification resulting from feeding the at least one image patch (1B) to a statistical image recognition algorithm (21) optimized for image patch input can correspond to at least one of a global classification label (4), or a local visual feature (3) linked to the respective at least one tooth and its surrounding region within the user's oral cavity (31).

### Claim 3 (depends on 2)

A method according to claim 2, wherein processing (103) the at least one digital image (1) using a statistical object detection algorithm (20) arranged to calculate the likelihood of at least one local visual feature (3) comprises:
feeding (1031) the at least one digital image (1) or at least one of its image derivatives (1A,1B,1C) to at least one classification algorithm (23) arranged to predict each of the at least one local visual feature (3) independently, wherein the at least one classification algorithm (23) is arranged to further predict:
the likelihood of the respective local visual feature (3), and
at least one local reference (6) indicating a location corresponding to the respective local visual feature (3) on the digital image (1) or any of its image derivatives (1A,1B,1C).

### Claim 4 (depends on 2)

A method according to claim 2, further comprising:
processing (301) the at least one digital image (1) or at least one of its image derivatives (1A,1B,1C) using a second statistical image recognition algorithm (21A) arranged to predict a set of pre-classification labels (8) corresponding to the most likely possible medical findings of the user's oral cavity (31); and
using (302) the set of pre-classification labels (8) as input condition for at least one of the statistical object detection algorithm (20) or the statistical image recognition algorithm (21) to limit the possible result pool of either or both of the algorithms.

### Claim 5 (depends on 1)

A method according to claim 1, wherein the non-image data (2) comprises self-reported user input obtained using the mobile device in the form of at least one of a checklist, a slider bar, or free text; and wherein the method further comprises:
processing (1021) the non-image data (2) using a syntax analysis algorithm (22) to extract a structured database of non-image signals (5), the non-image signals (5) indicating at least one of
symptoms or signs of health regarding intraoral colors, surface structures, contours, wounds, texture, general health, taste in mouth, breath, saliva, healing process, pain;
symptom triggers related to touch or icing, biting down, movement of tongue or jaw; or
behavioral data regarding oral habits, general health, medicine intake, comorbidities of periodontitis, history of trauma, history of oral procedures, pregnancy, or tobacco usage.

### Claim 6 (depends on 1)

A method according to claim 1, wherein the statistical object detection algorithm (20) uses a neural network model, more preferably a convolutional neural network model.

### Claim 7 (depends on 1)

A method according to claim 1, wherein at least one of
the at least one local visual feature (3) is used as further input for the statistical image recognition algorithm (21) to enhance the predictability of the at least one global classification label (4); or
the at least one global classification label (4) is used as further input for the statistical object detection algorithm (20) to enhance the predictability of the at least one local visual feature (3).

### Claim 8 (depends on 1)

A method according to claim 1, wherein the at least one global classification label (4) corresponds to at least one of “gingivitis”, “periodontitis”, “dental cavities”, or “herpes labialis”.

### Claim 9 (depends on 8)

A method according to claim 8, wherein one or more predefined local visual feature (3) may increase the likelihood of at least one global classification label (4), such as:
“swelling gums”, “bleeding gums”, “redness of gums”, “dental plaque” may increase the likelihood of “gingivitis”;
“exposure of root”, “abrasion of tooth”, “staining of tooth”, “recession of gums” may increase the likelihood of “periodontitis”;
“white spot lesion”, “brown/black discoloration of tooth”, “enamel breakdown”, “shade on enamel”, “dental plaque” may increase the likelihood of “dental cavities”;
“inflamed papules and vesicles”, “fluid-filled blister”, “local redness of lip”, “open ulcer” may increase the likelihood of “herpes labialis”; and
“light coral pink gums”, “stippled surface gums”, “tightly fitting gumline on tooth” may increase the likelihood of “healthy gums”.

### Claim 10 (depends on 1)

A method according to claim 1, further comprising:
processing (1053) the set of concepts (9) using a specialist reference database and flagging any concept (9) comprising a classification label or a related local visual feature (3) that is defined in the specialist reference database as any one of
extraordinarily rare,
extraordinarily malignant, or
transcending into the domain of general health; and
if a concept (9) is flagged, forwarding (1054) it to a human specialist dental reviewer (32) for validation;
if medical certainty of a concept (9) is lower than a predefined medical certainty threshold but the concept (9) is not flagged, forwarding (1055) it to a human general dental reviewer (33) for validation; and
validating the remaining set of concepts (9) using (1056) a validation algorithm (25) arranged for automatic concept (9) validation based on a predefined set of validation rules.

### Claim 11 (depends on 10)

A method according to claim 10, further comprising:
processing (1057) the list of validated concepts (9A) to determine additional components (11) for the assessment (10), the additional components (11) comprising comments or prompts and possible or typical actions to take in case of an identified concept (9), global classification label (4) or related local visual feature (3), wherein determining the additional components (11) comprises at least one of:
determining a predefined component (11A) from multiple possible predefined components (11A) automatically based on predefined rules,
editing a predefined component (11A) manually by a human reviewer,
editing a predefined component (11A) automatically using a statistical natural language processing (NLP) algorithm,
determining a non-predefined component (11B) manually by a human reviewer, and
determining a non-predefined component (11B) automatically using a statistical natural language processing (NLP) algorithm; and
determining (105) an assessment (10) of the oral health of the user (30) based on the list of validated concepts (9A) and the additional components (11).

### Claim 12 (depends on 1)

A computer program product (42), encoded on a non-transitory computer-readable storage medium (43), operable to cause a processor (44) of a mobile device (40) or a processor of a remote server (50) to perform operations according to the method of claim 1.

### Claim 13 (depends on 1)

A system for assessing oral health of a user, the system comprising:
a mobile device (40) comprising a camera (41) for obtaining at least one digital image (1) of the user's oral cavity (31);
a machine-readable storage medium (43) comprising a program product (42);
a remote server (50) in data connection with the mobile device (40); and
one or more processors (44) operable to execute the program product (42) and perform operations according to the method of claim 1.