# US 10,482,603 B1 · Medical image segmentation using an integrated edge guidance module and object segmentation network > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,603 B1 - Document ID: US10482603B1 - Status: Granted - Kind: B1 - Application number: 16/451,565 - Filed: 2019-06-25 - Granted: 2019-11-19 - Earliest claimed: 2019-06-25 - Assignees: Artificial Intelligence, Ltd. - Applicants: Inception Institute of Artificial Intelligence, Ltd. - Inventors: Huazhu Fu; Jianbing Shen; Zhijie Zhang; Hang Dai; Ling Shao - Web page: https://patentgrep.com/patent/US10482603B1 ## CPC - G06T 7/11: Region-based segmentation - G06T 7/12: Edge-based segmentation - G06F 17/15: Correlation function computation {including computation of convolution operations (arithmetic circuits for sum of products per se, e.g. multiply-accumulators G06F7/5443; digital filters, e.g. FIR, IIR, adaptive filters H03H17/00)} - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/09: Supervised learning - G06T 7/0012: {Biomedical image inspection} - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06T 2207/10072: Tomographic images - G06T 2207/10116: X-ray image - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30041: Eye; Retina; Ophthalmic - G06T 2207/30061: Lung - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract This disclosure relates to improved techniques for performing image segmentation functions using neural network architectures. The neural network architecture integrates an edge guidance module and object segmentation network into a single framework for detecting target objects and performing segmentation functions. The neural network architecture can be trained to generate edge-attention representations that preserve the edge information included in images. The neural network architecture can be trained to generate multi-scale feature information that preserves and enhances object-level feature information included in images. The edge-attention representations and multi-scale feature information can be fused to generate segmentation results that identify target object boundaries with increased accuracy. ## Claims ### Claim 1 A system for performing image segmentation on one or more images, comprising: one or more computing devices comprising one or more processors and one or more non-transitory storage devices for storing instructions, wherein execution of the instructions by the one or more processors causes the one or more computing devices to: receive an image comprising one or more objects at a neural network architecture comprising an encoder-decoder network, wherein the encoder-decoder network includes a plurality of encoding blocks and a plurality of decoding blocks; generate, using the neural network architecture, an edge-attention representation associated with the image, at least in part, utilizing encoding outputs of one or more encoding blocks included in the encoder-decoder network, wherein the edge-attention representation preserves local edge information included in the image; generate, using the neural network architecture, multi-scale feature information associated with the image, at least in part, utilizing decoding outputs of one or more decoding blocks included in the encoder-decoder network, wherein the multi-scale feature information preserves object-level features included in the image; generate, using the neural network architecture, segmentation results corresponding to the image using both the edge-attention representation and the multi-scale feature information derived from the image, wherein the segmentation results identify boundaries of the one or more objects in the image. ### Claim 2 (depends on 1) The system of claim 1, wherein generating the segmentation results using both the edge-attention representation and the multi-scale feature information includes: concatenating the edge-attention representation with the multi-scale feature information; and utilizing a convolutional layer to extract the one or more objects included in the image. ### Claim 3 (depends on 1) The system of claim 1, wherein generating the multi-scale feature information includes: generating, at each of the plurality of decoding blocks, the decoding outputs; providing the decoding outputs to weighted blocks that apply weights to identify relevant object-level features; and aggregating outputs from the weighted blocks to generate the multi-scale feature information. ### Claim 4 (depends on 1) The system of claim 1, wherein generating the edge-attention representation includes: generating, using a first encoding block of the encoder-decoder network, a first encoding output having a first resolution; generating, using a second encoding block of the encoder-decoder network, a second encoding output having a second resolution; upsampling the second encoding output to the first resolution; applying one or more convolutional functions to both the first encoding output and the upsampled second encoding output; and after applying the one or more convolutional functions, generating the edge-attention representation by concatenating the first encoding output and the upsampled second encoding output. ### Claim 5 (depends on 1) The system of claim 1, wherein the neural network architecture is trained, at least in part, using a segmentation loss function that is utilized to optimize generation of the edge-attention representation and the multi-scale feature information. ### Claim 6 (depends on 5) The system of claim 5, wherein the loss function utilizes a separate Lovász-Softmax loss to optimize generation of the edge-attention representation and the multi-scale feature information. ### Claim 7 (depends on 5) The system of claim 5, wherein the neural network architecture is trained using a set of training images that are annotated with pixel-level labels identifying boundaries of objects included in the training images. ### Claim 8 (depends on 1) The system of claim 1, wherein: the image is a medical image that includes one or more medical objects; the one or more medical objects include at least one of: an optic disk, an optic cup, an eye vessel, or a lung; and the segmentation results identify the boundaries of the one or more medical objects with pixel-level accuracy. ### Claim 9 (depends on 1) The system of claim 1, wherein: the encoder-decoder network comprises four encoding blocks and three decoding blocks; the encoding outputs from a first and second encoding block are utilized to generate the edge-attention representation; the decoding outputs from each of the three decoding blocks are utilized to generate the multi-scale feature information. ### Claim 10 (depends on 1) The system of claim 1, wherein: the system comprises a network; the image is transmitted to the neural network architecture over the network; the segmentation results generated by the neural network architecture are transmitted to at least one computing device over the network. ### Claim 11 A method for performing image segmentation on one or more images, comprising: receiving an image comprising one or more objects at a neural network architecture comprising an encoder-decoder network, wherein the encoder-decoder network includes a plurality of encoding blocks and a plurality of decoding blocks; generating, using the neural network architecture, an edge-attention representation associated with the image, at least in part, utilizing encoding outputs of one or more encoding blocks included in the encoder-decoder network, wherein the edge-attention representation preserves local edge information included in the image; generating, using the neural network architecture, multi-scale feature information associated with the image, at least in part, utilizing decoding outputs of one or more decoding blocks included in the encoder-decoder network, wherein the multi-scale feature information preserves object-level features included in the image; generating, using the neural network architecture, segmentation results corresponding to the image using both the edge-attention representation and the multi-scale feature information derived from the image, wherein the segmentation results identify boundaries of the one or more objects in the image. ### Claim 12 (depends on 11) The method of claim 11, wherein generating the segmentation results using both the edge-attention representation and the multi-scale feature information includes: concatenating the edge-attention representation with the multi-scale feature information; and utilizing a convolutional layer to extract the one or more objects included in the image. ### Claim 13 (depends on 11) The method of claim 11, wherein generating the multi-scale feature information includes: generating, at each of the plurality of decoding blocks, the decoding outputs; providing the decoding outputs to weighted blocks that apply weights to identify relevant object-level features; and aggregating outputs from the weighted blocks to generate the multi-scale feature information. ### Claim 14 (depends on 11) The method of claim 11, wherein generating the edge-attention representation includes: generating, using a first encoding block of the encoder-decoder network, a first encoding output having a first resolution; generating, using a second encoding block of the encoder-decoder network, a second encoding output having a second resolution; upsampling the second encoding output to the first resolution; applying one or more convolutional functions to both the first encoding output and the upsampled second encoding output; and after applying the one or more convolutional functions, generating the edge-attention representation by concatenating the first encoding output and the upsampled second encoding output. ### Claim 15 (depends on 11) The method of claim 11, wherein the neural network architecture is trained, at least in part, using a segmentation loss function that is utilized to optimize generation of the edge-attention representation and the multi-scale feature information. ### Claim 16 (depends on 15) The method of claim 15, wherein the loss function utilizes a separate Lovász-Softmax loss to optimize generation of the edge-attention representation and the multi-scale feature information. ### Claim 17 (depends on 15) The method of claim 15, wherein the neural network architecture is trained using a set of training images that are annotated with pixel-level labels identifying boundaries of objects included in the training images. ### Claim 18 (depends on 11) The method of claim 11, wherein: the image is a medical image that includes one or more medical objects; the one or more medical objects include at least one of: an optic disk, an optic cup, an eye vessel, or a lung; and the segmentation results identify the boundaries of the one or more medical objects with pixel-level accuracy. ### Claim 19 (depends on 11) The method of claim 11, wherein: the encoder-decoder network comprises four encoding blocks and three decoding blocks; the encoding outputs from a first and second encoding block are utilized to generate the edge-attention representation; the decoding outputs from each of the three decoding blocks are utilized to generate the multi-scale feature information. ### Claim 20 A computer program product comprising a non-transitory computer-readable medium, including instructions for causing a computer to: receive an image comprising one or more objects at a neural network architecture comprising an encoder-decoder network, wherein the encoder-decoder network includes a plurality of encoding blocks and a plurality of decoding blocks; generate, using the neural network architecture, an edge-attention representation associated with the image, at least in part, utilizing encoding outputs of one or more encoding blocks included in the encoder-decoder network, wherein the edge-attention representation preserves local edge information included in the image; generate, using the neural network architecture, multi-scale feature information associated with the image, at least in part, utilizing decoding outputs of one or more decoding blocks included in the encoder-decoder network, wherein the multi-scale feature information preserves object-level features included in the image; generate, using the neural network architecture, segmentation results corresponding to the image using both the edge-attention representation and the multi-scale feature information derived from the image, wherein the segmentation results identify boundaries of the one or more objects in the image.