# US 12,599,299 B2 · Method and apparatus for multimodal soft tissue diagnostics > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,599,299 B2 - Document ID: US12599299B2 - Status: Granted - Kind: B2 - Application number: 17/927,054 - Filed: 2021-05-20 - Published: 2023-07-13 - Granted: 2026-04-14 - Earliest claimed: 2020-05-26 - Assignees: DENTSPLY SIRONA Inc. - Applicants: DENTSPLY SIRONA Inc. - Inventors: Thomas Ertl - Web page: https://patentgrep.com/patent/US12599299B2 ## CPC - A61B 1/24: for the mouth, i.e. stomatoscopes, e.g. with tongue depressors; Instruments for opening or keeping open the mouth - A61B 1/000094: {extracting biological structures} - A61B 1/000096: {using artificial intelligence} - A61B 5/0035: {adapted for acquisition of images from more than one imaging mode, e.g. combining MRI and optical tomography} - A61B 5/0062: {Arrangements for scanning} - A61B 5/0071: {by measuring fluorescence emission} - A61B 5/0075: {by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy (A61B5/0071 takes precedence)} - A61B 5/0077: {Devices for viewing the surface of the body, e.g. camera, magnifying lens} - A61B 5/0088: {for oral or dental tissue} - A61B 5/1079: {using optical or photographic means} - A61B 5/444: {Evaluating skin marks, e.g. mole, nevi, tumour, scar} - G06T 7/30: Determination of transform parameters for the alignment of images, i.e. image registration - G06T 7/40: Analysis of texture (depth or shape recovery from texture G06T7/529) - G06V 10/764: using classification, e.g. of video objects - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/10064: Fluorescence image - G06T 2207/10101: Optical tomography; Optical coherence tomography [OCT] - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30096: Tumor; Lesion - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method and device for multimodal imaging of dermal and mucosal lesions. The method includes using at least two imaging modalities from which one is a 3D scan of the lesion, and, additionally providing information on the distance and angulation between scanning device and the dermis or mucosa and mapping at least the second modality over the 3D data. ## Claims ### Claim 1 An apparatus for multimodal imaging of a dermal or mucosal lesion comprising: a scanning device having illumination light sources, a beam splitter, and sensors, the sensors including a visible light 2D sensor and an infrared light 3D sensor, and the beam splitter being configured to spectrally split detected light into wavelengths appropriate for the visible light 2D sensor and the infrared light 3D sensor; and at least one processor configured to: compute images from raw data provided by the scanning device which is adapted to use at least two imaging modalities from which a first imaging modality generates 3D data for a 3D image in a 3D scan of a lesion, compute 3D information on a distance and angulation between the scanning device and a dermis or mucosa, and map at least an image generated by a second imaging modality over the 3D image of the 3D scan based on the 3D information. ### Claim 2 (depends on 1) The apparatus of claim 1, wherein the at least one processor is further configured to compute exact dimensions of the lesion by using the 3D information on the distance and angulation between the scanning device and the lesion. ### Claim 3 (depends on 1) The apparatus of claim 1, wherein the at least one processor is further configured to compute a 3D surface texture of the lesion by using the 3D information of the 3D scan. ### Claim 4 (depends on 1) The apparatus of claim 1, wherein the second imaging modality generates as the image at least one of a 2D image or an auto-fluorescence image by employing one sensor, wherein the 2D image is spectrally resolved with 3 or more channels. ### Claim 5 (depends on 4) The apparatus of claim 4, wherein spectrally resolved 2D data of the 2D image that is superimposed through the at least one processor onto the 3D data to support a correct registration of 3D images of the 3D data to form a complete 3D image of a region of interest, wherein the 3D images include at least one of a 3D texture image or subsurface structure image. ### Claim 6 (depends on 4) The apparatus of claim 4, wherein the scanning device comprises a mosaic type CMOS sensor with a plurality of different filters in combination with a lens array configured for 2D spectral imaging. ### Claim 7 (depends on 1) The apparatus of claim 1, wherein the apparatus is configured to capture 3D data of the 3D scan with a technology that suppresses volume scattering through confocal imaging, Optical Coherence Tomography (OCT), or combinations of confocal scanning or OCT with depth of focus based technologies. ### Claim 8 (depends on 1) The apparatus of claim 1, wherein (i) the scanning device is configured to use wavelengths between 350 and 400 nm for the 3D scan of a surface by employing one corresponding sensor for the 3D image or (ii) the scanning device is configured to use a wavelength longer than 840 nm for the 3D scan of a subsurface by employing one corresponding sensor for a subsurface structure image so as to operate at wavelength where dermal and mucosal lesions show a smaller scattering coefficient. ### Claim 9 (depends on 8) The apparatus of claim 8, wherein the scanning device is configured to use both wavelength ranges together for 3D scan of the surface and 3D scan of the subsurface. ### Claim 10 (depends on 9) The apparatus of claim 9, wherein the scanning device is configured to switch an illumination sequentially between different wavelengths, wherein the illumination light sources are coupled into the same light path with dichroitic mirrors. ### Claim 11 (depends on 9) The apparatus of claim 9, wherein a light path of illumination sources for different wavelengths is not fully congruent, provided that light source dimensions are small enough and slight angular deviation causing a displacement of an illumination pattern on associated sensor is corrected by calculation. ### Claim 12 (depends on 9) The apparatus of claim 9, wherein in the scanning device when using illumination with wavelength beyond 1000 nm, the scanning device comprises at least one beam splitter configured to separate an optical path for different sensors. ### Claim 13 (depends on 1) The apparatus of claim 1, wherein the first imaging modality or the second imaging modality is further configured for fluorescence imaging with excitation light in a UV/Blue range for fluorophores including flavin adenine dinucleotide (FAD), nicotinamide adenine dinucleotide (NADH), and collagen. ### Claim 14 (depends on 13) The apparatus of claim 13, wherein the scanning device comprises a blocking filter for excitation light using a same illumination pattern with a UV/Blue wavelength as used for 3D imaging and adapted to introduce a blocking filter for fluorescence detection into an imaging path after separation from the illumination path. ### Claim 15 (depends on 14) The apparatus of claim 14, wherein the blocking filter is in a 2D imaging path for a 2D image of the second imaging modality. ### Claim 16 (depends on 15) The apparatus of claim 15, wherein the blocking filter is placed in front of one 2D sensor for the 2D image. ### Claim 17 (depends on 13) The apparatus of claim 13, wherein the scanning device comprises a separate excitation light source placed on sides of a replaceable hood and a blocking filter is integrated in a window of the replaceable hood and the blocking filter is designed not to suppress structured light for 3D measurement. ### Claim 18 (depends on 17) The apparatus of claim 17, wherein the replaceable hood covers fluorescence excitation LEDs to illuminate a field of interest through the window wherein the replaceable hood is removable for sterilization, while the fluorescence excitation LEDs stay on the scanning device. ### Claim 19 (depends on 13) The apparatus of claim 13, wherein an excitation wavelength is below 350 nm, and a wavelength of an illumination pattern is in a range of 365 nm-405 nm. ### Claim 20 (depends on 1) The apparatus of claim 1, wherein the scanning device is adapted to use, in the second imaging modality, light in a red wavelength range for excitation of porphyrins and the at least one processor is configured to overlay a fluorescence image over the 3D data of the 3D scan. ### Claim 21 (depends on 1) The apparatus of claim 1, wherein the scanning device has one or more combinations of the following imaging modalities: use of visual wavelength 2D images together with 3D information used for recalculation of a magnification, distance and angulation of the lesion; visual wavelength 2D image overlaid over a 3D texture image of the lesion; use of visual wavelength 2D images and fluorescent image with 3D information used for recalculation of a magnification, distance and angulation; visual wavelength 2D image and fluorescent image overlaid over a 3D texture image of the lesion; visual wavelength 2D image overlaid over a 3D texture image of the lesion and a subsurface structure image; visual wavelength 2D image and fluorescent image overlaid over a 3D texture image of the lesion and a subsurface structure image. ### Claim 22 (depends on 1) The apparatus of claim 1, wherein an artificial neuronal network is integrated into the apparatus, and adapted to use images of a multimodal image database for training, and classifying multimodal images which are fed in the artificial neuronal network. ### Claim 23 (depends on 1) The apparatus of claim 1, wherein the 3D information is sent by a computer to a cloud-based artificial neuronal network for training of the cloud-based artificial neuronal network, and collected in a multimodal image database, wherein the cloud-based artificial neuronal network is connectable by the apparatus for classification of intra-oral lesions captured by the apparatus and provided to the cloud-based artificial neuronal network. ### Claim 24 (depends on 22, 23) The apparatus of claim 22 or 23, wherein additionally to the 3D information, other information including palpation results, removability of whitish layers, lesion history and risk factors such as smoking, or alcohol is used for training and retrieval of the cloud-based artificial neuronal network. ### Claim 25 (depends on 1) The apparatus of claim 1, wherein the scanning device comprises one InGaAs image sensor configured to cover at least a wavelength range from 1000 nm-1600 nm. ### Claim 26 (depends on 1) The apparatus of claim 1, wherein the scanning device comprises a CQD image sensor configured to extend sensitivity into a near-infrared (NIR) range covering at least additionally a wavelength range from 1000 nm-1400 nm. ### Claim 27 (depends on 1) The apparatus of according to claim 1, wherein the at least one processor is configured to additionally provide the 3D information on the distance and angulation between scanning device and the dermis or mucosa through use of an illumination pattern, a stereogrammetry, or time of flight. ### Claim 28 A method for multimodal imaging of a dermal or mucosal lesion comprising: computing images from raw data provided by a scanning device having illumination light sources and sensor, the scanning device adapted to use at least two imaging modalities from which a first imaging modality generates 3D data for a 3D image in a 3D scan of the dermal or mucosal lesion, the scanning device having illumination light sources, a beam splitter, and sensors, the sensors including a visible light 2D sensor and an infrared light 3D sensor, and the beam splitter being configured to spectrally split detected light into wavelengths appropriate for the visible light 2D sensor and the infrared light 3D sensor, computing 3D information on a distance and an angulation between the scanning device and a dermis or mucosa, and mapping at least an image generated by a second imaging modality over the 3D image of the 3D scan based on the 3D information.