# US 10,373,312 B2 · Automated skin lesion segmentation using deep side layers > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,373,312 B2 - Document ID: US10373312B2 - Status: Granted - Kind: B2 - Application number: 15/642,717 - Filed: 2017-07-06 - Published: 2018-05-10 - Granted: 2019-08-06 - Earliest claimed: 2016-11-06 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Mani Abedini; SeyedBehzad Bozorgtabar; Rajib Chakravorty; Sergey Demyanov; Rahil Garnavi; Zongyuan Ge - Web page: https://patentgrep.com/patent/US10373312B2 ## CPC - G06T 7/0012: {Biomedical image inspection} - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30088: Skin; Dermal - G06T 2207/30096: Tumor; Lesion - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method for computer-aided diagnosis of skin lesions includes obtaining a dermoscopic image, convolving the dermoscopic image in a plurality of convolutional layers, obtaining deconvolved outputs of at least two convolutional layers of the plurality of convolutional layers, obtaining side-output feature maps by applying loss functions to the deconvolved outputs of the at least two convolutional layers, obtaining a first concatenated feature map by concatenating the side-output feature maps with different first weights, obtaining a second concatenated feature map by concatenating the side-output feature maps with different second weights, and producing a final score map by convolving the first and second concatenated feature maps in a final convolutional layer followed by a loss layer. Also disclosed: a computer-readable medium embodying instructions for the method, and an apparatus configured to implement the method. ## Claims ### Claim 1 A computer-implemented method comprising: obtaining a dermoscopic image; convolving the dermoscopic image in a plurality of convolutional layers; obtaining deconvolved outputs of at least two convolutional layers of the plurality of convolutional layers; obtaining side-output feature maps by applying loss functions to the deconvolved outputs of the at least two convolutional layers; obtaining a first concatenated feature map by concatenating the side-output feature maps with different first weights; obtaining a second concatenated feature map by concatenating the side-output feature maps with different second weights; and producing a final score map by convolving the first and second concatenated feature maps in a final convolutional layer followed by a final loss layer. ### Claim 2 (depends on 1) The method of claim 1 wherein the final loss layer directly notifies each side-output layer about the final objective of segmenting the skin lesion. ### Claim 3 (depends on 1) The method of claim 1 further comprising: obtaining a training dermoscopic image and a training final score map; convolving the training dermoscopic image in the plurality of convolutional layers; deconvolving the outputs of the at least two convolutional layers of the plurality of convolutional layers; obtaining trial side-output feature maps by applying loss functions to the deconvolved outputs of the at least two convolutional layers; obtaining a trial concatenated feature map by concatenating the trial side-output feature maps with different weights; convolving the trial concatenated feature map in the final convolutional layer followed by the final loss layer to produce a trial final score map; assessing variances of the trial final score map from the training final score map; adjusting filters of the plurality of convolutional layers in response to the variances; and repeating the preceding steps until the variances are less than a threshold variance vector. ### Claim 4 (depends on 3) The method of claim 3 further comprising adjusting deconvolution filters in response to the variances. ### Claim 5 (depends on 1) The method of claim 1 wherein the plurality of convolutional layers comprise a VGG-16 neural network. ### Claim 6 (depends on 5) The method of claim 5 wherein the at least two convolutional layers include conv2_2, conv3_3, conv4_3, and conv5_3 layers of the VGG-16 neural network. ### Claim 7 (depends on 5) The method of claim 5 wherein the plurality of convolutional layers comprise a conv5_4 layer of dimensions 14×14×512, in addition to the layers of the VGG-16 neural network, and the conv5_4 layer is one of the at least two convolutional layers. ### Claim 8 A non-transitory computer readable medium embodying computer executable instructions which when executed by a computer cause the computer to perform the method of: obtaining a dermoscopic image; convolving the dermoscopic image in a plurality of convolutional layers; obtaining deconvolved outputs of at least two convolutional layers of the plurality of convolutional layers; obtaining side-output feature maps by applying loss functions to the deconvolved outputs of the at least two convolutional layers; obtaining a first concatenated feature map by concatenating the side-output feature maps with different first weights; obtaining a second concatenated feature map by concatenating the side-output feature maps with different second weights; and producing a final score map by convolving the first and second concatenated feature maps in a final convolutional layer followed by a loss layer. ### Claim 9 (depends on 8) The medium of claim 8 wherein the final loss layer directly notifies each side-output layer about the final objective of segmenting the skin lesion. ### Claim 10 (depends on 8) The medium of claim 8, the method further comprising: obtaining a training dermoscopic image and a training final score map; convolving the training dermoscopic image in the plurality of convolutional layers; deconvolving the outputs of the at least two convolutional layers of the plurality of convolutional layers; obtaining trial side-output feature maps by applying loss functions to the deconvolved outputs of the at least two convolutional layers; obtaining a trial concatenated feature map by concatenating the trial side-output feature maps with different weights; convolving the trial concatenated feature map in the final convolutional layer followed by the loss layer to produce a trial final score map; assessing variances of the trial final score map from the training final score map; adjusting filters of the plurality of convolutional layers in response to the variances; and repeating the preceding steps until the variances are less than a threshold variance vector. ### Claim 11 (depends on 10) The medium of claim 10, the method further comprising adjusting deconvolution filters in response to the variances. ### Claim 12 (depends on 8) The medium of claim 8 wherein the plurality of convolutional layers comprise a VGG-16 neural network. ### Claim 13 (depends on 12) The medium of claim 12 wherein the at least two convolutional layers include conv2_2, conv3_3, conv4_3, and conv5_3 layers of the VGG-16 neural network. ### Claim 14 (depends on 12) The medium of claim 12 wherein the plurality of convolutional layers comprise a conv5_4 layer of dimensions 14×14×512, in addition to the layers of the VGG-16 neural network, and the conv5_4 layer is one of the at least two convolutional layers. ### Claim 15 An apparatus comprising: a memory in which computer executable instructions are stored; and at least one processor, coupled to said memory, and operative by the computer executable instructions to perform a method comprising: obtaining a dermoscopic image; convolving the dermoscopic image in a plurality of convolutional layers; obtaining deconvolved outputs of at least two convolutional layers of the plurality of convolutional layers; obtaining side-output feature maps by applying loss functions to the deconvolved outputs of the at least two convolutional layers; obtaining a first concatenated feature map by concatenating the side-output feature maps with different first weights; obtaining a second concatenated feature map by concatenating the side-output feature maps with different second weights; and producing a final score map by convolving the first and second concatenated feature maps in a final convolutional layer followed by a loss layer. ### Claim 16 (depends on 15) The apparatus of claim 15, the method further comprising: obtaining a training dermoscopic image and a training final score map; convolving the training dermoscopic image in the plurality of convolutional layers; deconvolving the outputs of the at least two convolutional layers of the plurality of convolutional layers; obtaining trial side-output feature maps by applying loss functions to the deconvolved outputs of the at least two convolutional layers; obtaining a trial concatenated feature map by concatenating the trial side-output feature maps with different weights; convolving the trial concatenated feature map in the final convolutional layer followed by the loss layer to produce a trial final score map; assessing variances of the trial final score map from the training final score map; adjusting filters of the plurality of convolutional layers in response to the variances; and repeating the preceding steps until the variances are less than a threshold variance vector. ### Claim 17 (depends on 16) The apparatus of claim 16, the method further comprising adjusting deconvolution filters in response to the variances. ### Claim 18 (depends on 15) The apparatus of claim 15 wherein the plurality of convolutional layers comprise a VGG-16 neural network. ### Claim 19 (depends on 18) The apparatus of claim 18 wherein the at least two convolutional layers include conv2_2, conv3_3, conv4_3, and conv5_3 layers of the VGG-16 neural network. ### Claim 20 (depends on 18) The apparatus of claim 18 wherein the plurality of convolutional layers comprise a conv5_4 layer of dimensions 14×14×512, in addition to the layers of the VGG-16 neural network, and the conv5_4 layer is one of the at least two convolutional layers.