# US 10,769,191 B2 · Classifying data objects > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,769,191 B2 - Document ID: US10769191B2 - Status: Granted - Kind: B2 - Application number: 14/576,907 - Filed: 2014-12-19 - Published: 2015-06-25 - Granted: 2020-09-08 - Earliest claimed: 2013-12-20 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Gregory Sean Corrado; Tomas Mikolov; Samy Bengio; Yoram Singer; Jonathon Shlens; Andrea L. Frome; Jeffrey Adgate Dean; Mohammad Norouzi - Web page: https://patentgrep.com/patent/US10769191B2 ## CPC - G06F 16/35: Clustering; Classification - G06F 16/50: of still image data - G06F 16/55: Clustering; Classification - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods, systems, and apparatus, including computer programs encoded on computer storage media, for classifying data objects. One of the methods includes obtaining data that associates each term in a vocabulary of terms with a respective high-dimensional representation of the term; obtaining classification data for a data object, wherein the classification data includes a respective score for each of a plurality of categories, and wherein each of the categories is associated with a respective category label; computing an aggregate high-dimensional representation for the data object from high-dimensional representations for the category labels associated with the categories and the respective scores; identifying a first term in the vocabulary of terms having a high-dimensional representation that is closest to the aggregate high-dimensional representation; and selecting the first term as a category label for the data object. ## Claims ### Claim 1 A method performed by one or more computers, the method comprising: obtaining term representation data that has a respective high-dimensional representation for each category label of a plurality of category labels that each correspond to a different category of a plurality of categories, wherein the high-dimensional representation of the category label is a numeric representation of the category label in a high-dimensional space, and wherein each category label is one or more words that describe the corresponding category; obtaining a new image to be classified; processing the new image using a deep convolutional neural network, wherein the deep convolutional network comprises a plurality of neural network layers and has been configured through training to: generate a plurality of initial classification scores for the new image, wherein each initial classification score corresponds to a different category from the plurality of categories and represents a likelihood as determined by the deep convolutional neural network that the new image includes an image of an object that belongs to the corresponding category; and generating a new classification for the new image using, for each category of the plurality of categories, both (i) the initial classification score for the category generated by the deep convolutional neural network and (ii) the high-dimensional representation for the category label corresponding to the category, comprising: for each category of the plurality of categories: multiplying the high-dimensional representation of the category label corresponding to the category by the initial classification generated by the deep convolutional neural network for the category to generate a product for the category; combining the products for each of the categories to generate an aggregate high-dimensional representation for the new image; identifying a first term in a vocabulary of terms having a high-dimensional representation that is closest to the aggregate high-dimensional representation; and selecting the first term as a category label for the new image. ### Claim 2 (depends on 1) The method of claim 1, further comprising: identifying a second term in the vocabulary of terms having a high-dimensional representation that is second closest to the aggregate high-dimensional representation; and selecting the second term as another category label for the new image. ### Claim 3 (depends on 1) The method of claim 1, wherein a high-dimensional representation is a high-dimensional vector of floating point values. ### Claim 4 (depends on 1) The method of claim 1, wherein the high-dimensional space is a one thousand dimensional space. ### Claim 5 (depends on 1) The method of claim 1, wherein identifying the first term comprises selecting a term having a high-dimensional representation that has a largest cosine similarity value with the aggregate high-dimensional representation. ### Claim 6 (depends on 1) The method of claim 1, wherein processing the new image using the deep convolutional neural network comprises: processing the new image using the plurality of neural network layers to obtain a plurality of initial scores; selecting a pre-determined number of highest initial scores; and applying a transformation to each of the selected initial scores to generate the respective classification scores for the plurality of categories. ### Claim 7 (depends on 6) The method of claim 6, wherein applying the transformation to each of the selected scores comprises normalizing the selected scores. ### Claim 8 (depends on 1) The method of claim 1, wherein combining the products for each of the categories further comprises: summing the products for each of the categories. ### Claim 9 (depends on 1) The method of claim 1, wherein obtaining the term representation data that has a respective high-dimensional representation for each category label of a plurality of category labels that each correspond to a different category of a plurality of categories comprises: training a machine learning system configured to process each category label of the plurality of category labels to obtain the respective high-dimensional representation of the category label and to associate each category label with the respective high-dimensional representation of the category label. ### Claim 10 A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising: obtaining term representation data that has a respective high-dimensional representation for each category label of a plurality of category labels that each correspond to a different category of a plurality of categories, wherein the high-dimensional representation of the category label is a numeric representation of the category label in a high-dimensional space, and wherein each category label is one or more words that describe the corresponding category; obtaining a new image to be classified; processing the new image using a deep convolutional neural network, wherein the deep convolutional network comprises a plurality of neural network layers and has been configured through training to: generate a plurality of initial classification scores for the new image, wherein each initial classification score corresponds to a different category from the plurality of categories and represents a likelihood as determined by the deep convolutional neural network that the new image includes an image of an object that belongs to the corresponding category; and generating a new classification for the new image using, for each category of the plurality of categories, both (i) the initial classification score for the category generated by the deep convolutional neural network and (ii) the high-dimensional representation for the category label corresponding to the category, comprising: for each category of the plurality of categories: multiplying the high-dimensional representation of the category label corresponding to the category by the initial classification score for the category generated by the deep convolutional neural network to generate a product for the category; combining the products for each of the categories to generate an aggregate high-dimensional representation for the new image; identifying a first term in a vocabulary of terms having a high-dimensional representation that is closest to the aggregate high-dimensional representation; and selecting the first term as a category label for the new image. ### Claim 11 (depends on 10) The system of claim 10, the operations further comprising: identifying a second term in the vocabulary of terms having a high-dimensional representation that is second closest to the aggregate high-dimensional representation; and selecting the second term as another category label for the new image. ### Claim 12 (depends on 10) The system of claim 10, wherein identifying the first term comprises selecting a term having a high-dimensional representation that has a largest cosine similarity value with the aggregate high-dimensional representation. ### Claim 13 (depends on 10) The system of claim 10, wherein processing the new image using the deep convolutional neural network comprises: processing the new image using the plurality of neural network layers to obtain a plurality of initial scores; selecting a pre-determined number of highest initial scores; and applying a transformation to each of the selected initial scores to generate the respective classification scores for the plurality of categories. ### Claim 14 (depends on 13) The system of claim 13, wherein applying the transformation to each of the selected scores comprises normalizing the selected scores. ### Claim 15 (depends on 10) The system of claim 10, wherein combining the products for each of the categories further comprises: summing the products for each of the categories. ### Claim 16 A non-transitory computer storage medium encoded with a computer program, the computer program comprising instructions that when executed by one or more computers cause the one or more computers to perform comprising: obtaining term representation data that has a respective high-dimensional representation for each category label of a plurality of category labels that each correspond to a different category of a plurality of categories, wherein the high-dimensional representation of the category label is a numeric representation of the category label in a high-dimensional space, and wherein each category label is one or more words that describe the corresponding category; obtaining a new image to be classified; processing the new image using a deep convolutional neural network, wherein the deep convolutional network comprises a plurality of neural network layers and has been configured through training to: generate a plurality of initial classification scores for the new image, wherein each initial classification score corresponds to a different category from the plurality of categories and represents a likelihood as determined by the deep convolutional neural network that the new image includes an image of an object that belongs to the corresponding category; and generating a new classification for the new image using, for each category of the plurality of categories, both (i) the initial classification score for the category generated by the deep convolutional neural network and (ii) the high-dimensional representation for the category label corresponding to the category, comprising: for each category of the plurality of categories: multiplying the high-dimensional representation of the category label corresponding to the category by the initial classification score for the category generated by the deep convolutional neural network to generate a product for the category; combining the products for each of the categories to generate an aggregate high-dimensional representation for the new image; identifying a first term in a vocabulary of terms having a high-dimensional representation that is closest to the aggregate high-dimensional representation; and selecting the first term as a category label for the new image.