# US 10,769,408 B2 · Method and system for automatic chromosome classification > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,769,408 B2 - Document ID: US10769408B2 - Status: Granted - Kind: B2 - Application number: 16/246,278 - Filed: 2019-01-11 - Published: 2020-01-09 - Granted: 2020-09-08 - Earliest claimed: 2018-07-06 - Assignees: Tata Consultancy Services Limited - Applicants: Tata Consultancy Services Limited - Inventors: Monika Sharma; Swati Jindal; Lovekesh Vig - Web page: https://patentgrep.com/patent/US10769408B2 ## CPC - G06N 3/08: Learning methods - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06T 7/74: {involving reference images or patches} - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 20/695: {Preprocessing, e.g. image segmentation} - G06V 20/698: {Matching; Classification} - G06T 2207/10004: Still image; Photographic image - G06T 2207/20081: Training; Learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Method and system for automatic chromosome classification is disclosed. The system, alternatively referred as a Residual Convolutional Recurrent Attention Neural Network (Res-CRANN), utilizes property of band sequence of chromosome bands for chromosome classification. The Res-CRANN is end-to-end trainable system, in which a sequence of feature vectors are extracted from the feature maps produced by convolutional layers of a Residual neural networks (ResNet), wherein the feature vectors correspond to visual features representing chromosome bands in an chromosome image. The sequence feature vectors are fed into Recurrent Neural Networks (RNN) augmented with an attention mechanism. The RNN learns the sequence of feature vectors and the attention module concentrates on a plurality of Regions-of-interest (ROIs) of the sequence of feature vectors, wherein the ROIs are specific to a class label of chromosomes. The Res-CRANN provides higher classification accuracy as compared to the state-of the-art methods for chromosome classification. ## Claims ### Claim 1 A processor implemented method for chromosome classification, the method comprising: receiving, via one or more hardware processors, a chromosome image comprising a chromosome with a plurality of chromosome bands; extracting, via one or more hardware processors, visual features associated with the plurality of chromosome bands of the chromosome by generating a plurality of feature maps with dimension G×H×K; obtaining, via one or more hardware processors, a plurality of feature vectors from the plurality of feature maps, with each feature vector Fg of dimension H*K, by applying horizontal slicing on the plurality of feature maps; concatenating, via one or more hardware processors, the plurality of feature vectors in sequence from a top chromosome band to a bottom chromosome band among the plurality of chromosome bands to generate a feature sequence; and classifying, via one or more hardware processors, the chromosome into a class label among a plurality of pre-determined class labels of chromosomes by processing the feature sequence using a Recurrent Neural Network (RNN) augmented with an attention mechanism, wherein the RNN learns the feature sequence and the attention mechanism concentrates on a plurality of Regions-of-interest (ROIs) of the feature sequence, wherein the ROIs are specific to a class label of chromosomes. ### Claim 2 (depends on 1) The method as claimed in claim 1, wherein the classifying the chromosome further comprises utilizing a softmax layer, which assigns a unique class label to the feature sequence. ### Claim 3 (depends on 1) The method as claimed in claim 1, wherein the step of generating the plurality of feature maps by extracting the visual features of the chromosome is performed using a Residual Neural Network (ResNet)-50, wherein each feature map is a mathematical representation of the chromosome image. ### Claim 4 (depends on 1) The method as claimed in claim 1, wherein the plurality of chromosome bands of the chromosome image are read from the top chromosome band to the bottom chromosome band as characters are read from left-to-right in a text-sequence. ### Claim 5 (depends on 1) The method as claimed in claim 1, wherein the attention mechanism is an attention module comprising a neural network based learning mechanism, wherein the neural network based learning mechanism focuses on the ROI of the chromosome image with high resolution while perceiving a neighboring area of the chromosome image with low resolution. ### Claim 6 A system comprising: one or more data storage devices operatively coupled to one or more hardware processors and configured to store instructions configured for execution by the one or more hardware processors to: receive a chromosome image comprising a chromosome with a plurality of chromosome bands; extract visual features associated with the plurality of chromosome bands of the chromosome by generating a plurality of feature maps with dimension G×H×K; obtain a plurality of feature vectors from the plurality of feature maps, with each feature vector Fg of dimension H*K, by applying horizontal slicing on the plurality of feature maps; concatenate the plurality of feature vectors in sequence from a top chromosome band to a bottom chromosome band among the plurality of chromosome bands to generate a feature sequence; and classify the chromosome into a class label among a plurality of pre-determined class labels of chromosomes by processing the feature sequence using a Recurrent Neural Network (RNN) augmented with an attention mechanism, wherein the RNN learns the feature sequence and the attention mechanism concentrates on a plurality of Regions-of-interest (ROIs) of the feature sequence, wherein the ROIs are specific to a class label of chromosomes. ### Claim 7 (depends on 6) The system as claimed in claim 6, wherein the system is further configured to classify the chromosome utilizing a softmax layer, which assigns a unique class label to the feature sequence. ### Claim 8 (depends on 6) The system as claimed in claim 6, wherein the system is configured to generate the plurality of feature maps by extracting the visual features of the chromosome using a Residual Neural Network (ResNet)-50, wherein each feature map is a mathematical representation of the chromosome image. ### Claim 9 (depends on 6) The system as claimed in claim 6, wherein the system is configured to read the plurality of chromosome bands of the chromosome image from the top chromosome band to the bottom chromosome band as characters are read from left-to-right in a text-sequence. ### Claim 10 (depends on 6) The system as claimed in claim 6, wherein the attention mechanism of the system is an attention module comprising a neural network based learning mechanism, wherein the neural network based learning mechanism focuses on the ROI of the chromosome image with high resolution while perceiving a neighboring area of the chromosome image with low resolution. ### Claim 11 A non-transitory computer readable medium storing instructions which, when executed by a hardware processor, causes the hardware processor to perform actions comprising: receiving a chromosome image comprising a chromosome with a plurality of chromosome bands; extracting visual features associated with the plurality of chromosome bands of the chromosome by generating a plurality of feature maps with dimension G×H×K; obtaining a plurality of feature vectors from the plurality of feature maps, with each feature vector Fg of dimension H*K, by applying horizontal slicing on the plurality of feature maps; concatenating the plurality of feature vectors in sequence from a top chromosome band to a bottom chromosome band among the plurality of chromosome bands to generate a feature sequence; and classifying the chromosome into a class label among a plurality of pre-determined class labels of chromosomes by processing the feature sequence using a Recurrent Neural Network (RNN) augmented with an attention mechanism, wherein the RNN learns the feature sequence and the attention mechanism concentrates on a plurality of Regions-of-interest (ROIs) of the feature sequence, wherein the ROIs are specific to a class label of chromosomes. ### Claim 12 (depends on 11) The non-transitory computer readable medium of claim 11, wherein the classifying the chromosome further comprises utilizing a softmax layer, which assigns a unique class label to the feature sequence. ### Claim 13 (depends on 11) The non-transitory computer readable medium of claim 11, wherein the step of generating the plurality of feature maps by extracting the visual features of the chromosome is performed using a Residual Neural Network (ResNet)-50, wherein each feature map is a mathematical representation of the chromosome image. ### Claim 14 (depends on 11) The non-transitory computer readable medium of claim 11, wherein the plurality of chromosome bands of the chromosome image are read from the top chromosome band to the bottom chromosome band as characters are read from left-to-right in a text-sequence. ### Claim 15 (depends on 11) The non-transitory computer readable medium of claim 11, wherein the attention mechanism is an attention module comprising a neural network based learning mechanism, wherein the neural network based learning mechanism focuses on the ROI of the chromosome image with high resolution while perceiving a neighboring area of the chromosome image with low resolution.