# US 11,298,072 B2 · Dermoscopy diagnosis of cancerous lesions utilizing dual deep learning algorithms via visual and audio (sonification) outputs

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,298,072 B2
- Document ID: US11298072B2
- Status: Granted
- Kind: B2
- Application number: 16/445,684
- Filed: 2019-06-19
- Published: 2019-11-07
- Granted: 2022-04-12
- Earliest claimed: 2016-07-01
- Assignees: Bostel Technologies, LLC
- Applicants: Avi Dascalu
- Inventors: Avi Dascalu
- Web page: https://patentgrep.com/patent/US11298072B2

## CPC

- A61B 5/444: {Evaluating skin marks, e.g. mole, nevi, tumour, scar}
- A61B 5/0064: {Body surface scanning}
- A61B 5/7267: {involving training the classification device}
- A61B 5/7415: {Sound rendering of measured values, e.g. by pitch or volume variation (A61B5/741 takes precedence)}
- G06N 3/02: Neural networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- A61B 5/0077: {Devices for viewing the surface of the body, e.g. camera, magnifying lens}
- A61B 5/445: {Evaluating skin irritation or skin trauma, e.g. rash, eczema, wound, bed sore}
- A61B 5/7257: {using Fourier transforms}
- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06N 20/00: Machine learning
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06N 20/20: Ensemble learning
- G06T 7/0012: {Biomedical image inspection}
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30088: Skin; Dermal
- G06T 2207/30096: Tumor; Lesion
- G10L 21/06: Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids (G10L15/26 takes precedence)
- G10L 25/18: the extracted parameters being spectral information of each sub-band
- G10L 25/66: for extracting parameters related to health condition (detecting or measuring for diagnostic purposes A61B5/00)
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

The present invention provides for a system and method for the diagnosis of skin cancer, wherein visual data is acquired from a skin lesion by a passive or active optical, electronical, thermal or mechanical method, processing the acquired visual field image into a classifier, applying a dermoscopic classification analysis to the image and converting, by sonification techniques, the data to raw audio signals, wherein the raw audio signals are analyzed with a second machine learning algorithm to increase interpretability and the precision of diagnosis.

## Claims

### Claim 1

A method of diagnosing skin cancer in a subject, the method comprising:
obtaining an image of a tissue sample suspected of being cancerous;
extracting a plurality of features from the image, wherein the features include texture, brightness, shape, color, size, and/or quality of tissue;
classifying the plurality of features using a first computer-aided classifier;
introducing the classified plurality of features into a computer-aided clustering algorithm to obtain clusters of data relating to each tissue type in the image;
converting, by sonification techniques, the clusters of data to audio signals; and
analyzing the audio signals using a second computer-aided classifier, wherein an audio output from the second computer-aided classifier acoustically differentiates malignant from benign lesions,
wherein the subject has skin cancer if the audio output indicates the lesion is malignant.

### Claim 2 (depends on 1)

The method of claim 1, wherein the image is obtained using photography, dermoscopy, molecular scattering, thermography, multiphoton fluorescence microscopy, multiphoton excitation microscopy, optical coherence tomography, or confocal scanning laser microscopy.

### Claim 3 (depends on 1)

The method of claim 1, wherein the plurality of features of the image are extracted and grouped into a plurality of segments for classifying by the first computer-aided classifier.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first computer-aided classifier is a deep learning classifier.

### Claim 5 (depends on 4)

The method of claim 4, wherein the deep learning classifier comprises a convolution Neural Network.

### Claim 6 (depends on 1)

The method of claim 1, wherein the clustering algorithm is an unsupervised, supervised, semisupervised, or reinforcement learning clustering algorithm.

### Claim 7 (depends on 1)

The method of claim 1, wherein for each image an audio file is generated using k-means sonification algorithms.

### Claim 8 (depends on 1)

The method of claim 1, wherein the audio signals are selected from the group consisting of different pitch, loudness, timbre, duration, spatialization and temporal patterns of each visual feature.

### Claim 9 (depends on 1)

The method of claim 1, wherein the audio output is communicated by headphones, speaker, smartphone, computer, tablet, mobile device, watch, Bluetooth device or a device that provides a tactile or vibration signal.

### Claim 10 (depends on 1)

The method of claim 1, wherein the second computer-aided classifier is a deep learning classifier.

### Claim 11 (depends on 10)

The method of claim 10, wherein deep learning classifier comprises a Convolution Neural Network.

### Claim 12 (depends on 1)

The method of claim 1, wherein the sonification technique comprises a parameter mapping sonification method wherein a classification method and/or a clustering algorithm is used to isolate clusters of data related to each tissue type and then assign a unique audio signal to each tissue type.

### Claim 13 (depends on 1)

The method of claim 1, wherein the tissue sample suspected of being cancerous comprises a lesion selected from the group consisting of atypical melanocytic hyperplasia, atypical mole, dysplastic mole, melanoma, cancerous skin diseases, actinic keratosis, basal cell carcinoma, and squamous cell carcinoma.

### Claim 14 (depends on 1)

The method of claim 1, wherein the audio signal comprises different output sounds differentiated by at least one of frequency, duration, magnitude, spectrum, and spatial orientation.

### Claim 15 (depends on 14)

The method of claim 14, wherein malignant lesions sound comparatively more loud, sharp, or urgent than benign lesions.

### Claim 16 (depends on 1)

The method of claim 1, wherein the clustering algorithm can be visualized by grouping features by K-means, hierarchical clustering or feature agglomeration.

### Claim 17 (depends on 1)

The method of claim 1, wherein a visual output can be obtained from the first computer-aided classifier, wherein the visual output can be used to identify if the lesion is benign or malignant.

### Claim 18 (depends on 7)

The method of claim 7, wherein for each audio file, the method further comprises producing a Fast Fourier Transform (FFTs) and/or a spectrogram, which exhibit visually-discernible differences between benign and malignant lesions.