# US 2023/0410495 A1 · Tooth detection and labeling > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0410495 A1 - Document ID: US20230410495A1 - Status: Published application - Kind: A1 - Application number: 18/241,709 - Filed: 2023-09-01 - Published: 2023-12-21 - Earliest claimed: 2017-11-07 - Assignees: Align Technology, Inc. - Applicants: Align Technology, Inc. - Inventors: Ya Xue; Yingjie Li; Chao Shi; Aleksandr Anikin; Mikhail Toporkov; Aleksandr Sergeevich Karsakov - Web page: https://patentgrep.com/patent/US20230410495A1 ## CPC - G06V 10/82: using neural networks - G06F 18/24143: {Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]} - G06T 7/0012: {Biomedical image inspection} - G06T 7/13: Edge detection - G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components - G06V 30/19173: {Classification techniques} - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20132: Image cropping - G06T 2207/30004: Biomedical image processing - G06V 2201/033: of skeletal patterns - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method includes receiving an image of a face, processing the image using a first trained machine learning model to determine a bounding shape around teeth in the image, cropping the image based on the bounding shape to produce a cropped image, processing the cropped image using an edge detection operation to generate edge data for the cropped image, and processing the cropped image and the edge data using a second trained machine learning model to label edges in the cropped image. ## Claims ### Claim 1 A method comprising: receiving an image of a face; processing the image using a first trained machine learning model to determine a bounding shape around teeth in the image; cropping the image based on the bounding shape to produce a cropped image; processing the cropped image using an edge detection operation to generate edge data for the cropped image; and processing the cropped image and the edge data using a second trained machine learning model to label edges in the cropped image. ### Claim 2 (depends on 1) The method of claim 1, wherein the image is an image of a patient wearing an aligner over teeth, and wherein the labeled edges comprise one or more first labeled edges having a tooth edge classification and one or more second labeled edges having an aligner edge classification. ### Claim 3 (depends on 2) The method of claim 2, further comprising: determining a fit of the aligner on the teeth based on a comparison of the one or more first labeled edges to the one or more second labeled edges. ### Claim 4 (depends on 2) The method of claim 2, further comprising: determining a distance between a portion of an aligner edge and a portion of an adjacent tooth edge; determining whether the distance exceeds a threshold; and responsive to determining that the distance exceeds the threshold, determining that the aligner does not properly fit the teeth. ### Claim 5 (depends on 4) The method of claim 4, further comprising: identifying one or more teeth in the image; registering each tooth of the one or more teeth with a respective tooth label; and for each tooth of one or more teeth, using the respective tooth label associated with the tooth to identify a specific tooth associated with the determined distance between the portion of the aligner edge and the portion of the adjacent tooth edge. ### Claim 6 (depends on 4) The method of claim 4, further comprising: generating a notification indicating that the aligner does not properly fit the teeth. ### Claim 7 (depends on 1) The method of claim 1, wherein labeled edges comprise edge classification probabilities of a plurality of edge classifications, the plurality of edge classifications comprising a tooth edge classification, the method further comprising: determining the edge classification probabilities for each edge pixel of a plurality of edge pixels in the edge data; and applying a path finding operation to the labeled edges using the tooth edge classification as a cost basis to update the edge classification probabilities for one or more of the edge pixels. ### Claim 8 (depends on 1) The method of claim 1, wherein labeling the edges in the cropped image comprises assigning a separate label to each of a plurality of teeth in the cropped image. ### Claim 9 A non-transitory computer readable storage medium comprising instructions that, when executed by a processing device, cause the processing device to perform operations comprising: receiving an image of a face; processing the image using a first trained machine learning model to determine a bounding shape around teeth in the image; cropping the image based on the bounding shape to produce a cropped image; processing the cropped image using an edge detection operation to generate edge data for the cropped image; and processing the cropped image and the edge data using a second trained machine learning model to label edges in the cropped image. ### Claim 10 (depends on 9) The non-transitory computer readable storage medium of claim 9, wherein the image is an image of a patient wearing an aligner over teeth, and wherein the labeled edges comprise one or more first labeled edges having a tooth edge classification and one or more second labeled edges having an aligner edge classification. ### Claim 11 (depends on 10) The non-transitory computer readable storage medium of claim 10, the operations further comprising: determining a fit of the aligner on the teeth based on a comparison of the one or more first labeled edges to the one or more second labeled edges. ### Claim 12 (depends on 10) The non-transitory computer readable storage medium of claim 10, the operations further comprising: determining a distance between a portion of an aligner edge and a portion of an adjacent tooth edge; determining whether the distance exceeds a threshold; and responsive to determining that the distance exceeds the threshold, determining that the aligner does not properly fit the teeth. ### Claim 13 (depends on 12) The non-transitory computer readable storage medium of claim 12, the operations further comprising: identifying one or more teeth in the image; registering each tooth of the one or more teeth with a respective tooth label; and for each tooth of one or more teeth, using the respective tooth label associated with the tooth to identify a specific tooth associated with the determined distance between the portion of the aligner edge and the portion of the adjacent tooth edge. ### Claim 14 (depends on 12) The non-transitory computer readable storage medium of claim 12, the operations further comprising: generating a notification indicating that the aligner does not properly fit the teeth. ### Claim 15 (depends on 9) The non-transitory computer readable storage medium of claim 9, wherein labeled edges comprise edge classification probabilities of a plurality of edge classifications, the plurality of edge classifications comprising a tooth edge classification, the operations further comprising: determining the edge classification probabilities for each edge pixel of a plurality of edge pixels in the edge data; and applying a path finding operation to the labeled edges using the tooth edge classification as a cost basis to update the edge classification probabilities for one or more of the edge pixels. ### Claim 16 (depends on 9) The non-transitory computer readable storage medium of claim 9, wherein labeling the edges in the cropped image comprises assigning a separate label to each of a plurality of teeth in the cropped image. ### Claim 17 A computing device comprising: a memory; and a processing device operatively coupled to the memory, the processing device to: receive an image of a face; process the image using a first trained machine learning model to determine a bounding shape around teeth in the image; crop the image based on the bounding shape to produce a cropped image; process the cropped image using an edge detection operation to generate edge data for the cropped image; and process the cropped image and the edge data using a second trained machine learning model to label edges in the cropped image. ### Claim 18 (depends on 17) The computing device of claim 17, wherein the image is an image of a patient wearing an aligner over teeth, and wherein the labeled edges comprise on or more first labeled edges having a tooth edge classification and one or more second labeled edges having an aligner edge classification. ### Claim 19 (depends on 18) The computing device of claim 18, wherein the processing device is further to: determine a fit of the aligner on the teeth based on a comparison of the one or more first labeled edges to the one or more second labeled edges. ### Claim 20 (depends on 17) The computing device of claim 17, wherein the processing device is further to: determine a distance between a portion of an aligner edge and a portion of an adjacent tooth edge; determine whether the distance exceeds a threshold; and responsive to determining that the distance exceed the threshold, determine that the aligner does not properly fit the teeth.