# US 10,769,491 B2 · Machine learning system for generating classification data and part localization data for objects depicted in images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,769,491 B2 - Document ID: US10769491B2 - Status: Granted - Kind: B2 - Application number: 16/120,023 - Filed: 2018-08-31 - Published: 2019-03-07 - Granted: 2020-09-08 - Earliest claimed: 2017-09-01 - Assignees: SRI International - Applicants: SRI International - Inventors: Bogdan Calin Mihai Matei; Xiyang Dai; John Benjamin Southall; Nhon Hoc Trinh; Harpreet Sawhney - Web page: https://patentgrep.com/patent/US10769491B2 ## CPC - G06V 10/82: using neural networks - G06F 18/2113: by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation - G06F 18/2413: based on distances to training or reference patterns - G06F 18/24133: {Distances to prototypes} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 20/00: Machine learning - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/764: using classification, e.g. of video objects - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Techniques are disclosed for identifying discriminative, fine-grained features of an object in an image. In one example, an input device receives an image. A machine learning system includes a model comprising a first set, a second set, and a third set of filters. The machine learning system applies the first set of filters to the received image to generate an intermediate representation of the received image. The machine learning system applies the second set of filters to the intermediate representation to generate part localization data identifying sub-parts of an object and one or more regions of the image in which the sub-parts are located. The machine learning system applies the third set of filters to the intermediate representation to generate classification data identifying a subordinate category to which the object belongs. The system uses the part localization and classification data to perform fine-grained classification of the object. ## Claims ### Claim 1 A system for identifying discriminative, fine-grained features of an object in an image, comprising: an input device configured to receive the image of the object; a computation engine comprising processing circuitry for executing a machine learning system; and an output device configured to output part localization data for the object and classification data for the object, wherein the machine learning system comprises a model comprising a first set of filters, a second set of filters, and a third set of filters, wherein the machine learning system is further configured to apply the first set of filters to the received image to generate an intermediate representation of the received image suitable as an input to both the second set of filters and third set of filters, wherein the machine learning system is further configured to apply the second set of filters to the intermediate representation of the received image to generate the part localization data for the object, wherein the part localization data for the object comprises data identifying one or more sub-parts of the object and one or more regions of the received image in which the one or more sub-parts of the object are located, and wherein the machine learning system is further configured to apply the third set of filters to the intermediate representation of the received image to generate the classification data for the object, wherein the data identifying one or more sub-parts of the object and one or more regions of the received image in which the one or more sub-parts of the object are located along with the classification data for the object results in a more discriminative, fine-grained identification of one or more features of the object in the image. ### Claim 2 (depends on 1) The system of claim 1, wherein the classification data for the object comprises data identifying a subordinate category within a basic level category for the object. ### Claim 3 (depends on 2) The system of claim 2, wherein the machine learning system is further configured to uniquely distinguish the object from other similar objects in the subordinate category within the basic level category for the object based on the part localization data for the object and the classification data for the object. ### Claim 4 (depends on 1) The system of claim 1, wherein the machine learning system is configured to train the model by: applying training data comprising images of objects labeled with corresponding part localization data for the objects and corresponding classification data for the objects to train the first set of filters and the third set of filters together; applying the training data to train the first set of filters and the second set of filters together; and applying the training data to train the first set of filters, the second set of filters, and the third set of filters together. ### Claim 5 (depends on 1) The system of claim 1, wherein the machine learning system is configured to train the model by: applying training data comprising images of objects labeled with corresponding part localization data for the objects and corresponding classification data for the objects to train the first set of filters and the third set of filters together; after applying the training data to train the first set of filters and the third set of filters, applying the training data to train the first set of filters and the second set of filters together; and after applying the training data to train the first set of filters and the second set of filters, applying the training data to train the first set of filters, the second set of filters, and the third set of filters together. ### Claim 6 (depends on 1) The system of claim 1, wherein the model comprises a convolutional neural network model. ### Claim 7 (depends on 1) The system of claim 1, wherein the second set of filters comprises a set of convolutional filters. ### Claim 8 (depends on 1) The system of claim 1, wherein the third set of filters comprises a set of one or more convolutional filters and one or more bilinear pooling filters. ### Claim 9 (depends on 1) The system of claim 1, wherein the image of the object comprises an image of a vehicle, wherein the part localization data for the object further comprises part localization data for the vehicle, the part localization data for the vehicle comprising data identifying one or more components of the vehicle and one or more regions of the received image in which the one or more components of the vehicle are located, and wherein the classification data for the object comprises classification data for the vehicle, the classification data for the vehicle comprising data identifying a make and model of the vehicle. ### Claim 10 (depends on 9) The system of claim 9, wherein the machine learning system is further configured to uniquely distinguish the vehicle from other similar vehicles of the same make and model based on the part localization data for the vehicle and the classification data for the vehicle. ### Claim 11 (depends on 1) The system of claim 1, wherein the image of the object comprises an image of an animal, wherein the part localization data for the object further comprises part localization data for the animal, the part localization data for the animal comprising data identifying one or more body parts of the animal and one or more regions of the received image in which the one or more body parts of the animal are located, and wherein the classification data for the object comprises classification data for the animal, the classification data for the animal comprising data identifying a species of the animal. ### Claim 12 A method for identifying discriminative, fine-grained features of an object in an image, the method comprising: receiving, by an input device, the image of the object; applying, by a machine learning system executed by processing circuitry of a computation engine, a first set of filters of a model to the received image to generate an intermediate representation of the received image suitable as an input to both a second set of filters and a third set of filters of the model; applying, by the machine learning system, the second set of filters to the intermediate representation of the received image to generate part localization data for the object, wherein the part localization data for the object comprises data identifying one or more sub-parts of the object and one or more regions of the received image in which the one or more sub-parts of the object are located; applying, by the machine learning system, the third set of filters to the intermediate representation of the received image to generate classification data for the object; and outputting, by an output device, the part localization data for the object and the classification data for the object, wherein the data identifying one or more sub-parts of the object and one or more regions of the received image in which the one or more sub-parts of the object are located along with the classification data for the object results in a more discriminative, fine-grained identification of one or more features of the object in the image. ### Claim 13 (depends on 12) The method of claim 12, wherein the classification data for the object comprises data identifying a subordinate category within a basic level category to which the object belongs. ### Claim 14 (depends on 13) The method of claim 13, further comprising uniquely distinguishing, by the machine learning system, the object from other similar objects in the subordinate category of similar objects based on the part localization data for the object and the classification data for the object. ### Claim 15 (depends on 12) The method of claim 12, further comprising training, by the machine learning system, the model by: applying training data comprising images of objects labeled with corresponding part localization data for the objects and corresponding classification data for the objects to train the first set of filters and the third set of filters together; applying the training data to train the first set of filters and the second set of filters together; and applying the training data to train the first set of filters, the second set of filters, and the third set of filters together. ### Claim 16 (depends on 12) The method of claim 12, wherein the image of the object comprises an image of a vehicle, wherein applying the second set of filters of the model to the intermediate representation of the received image to generate the part localization data for the object comprises applying the second set of filters of the model to the intermediate representation of the received image to generate part localization data for the vehicle, wherein the part localization data for the vehicle comprises data identifying one or more components of the vehicle and one or more regions of the received image in which the one or more components of the vehicle are located, and wherein applying the third set of filters of the model to the intermediate representation of the received image to generate the part localization data for the object comprises applying the third set of filters of the model to the intermediate representation of the received image to generate classification data for the vehicle, wherein the classification data for the vehicle comprises data identifying a make and model of the vehicle. ### Claim 17 A non-transitory computer-readable medium comprising instructions that, when executed, cause processing circuitry of a computation device to execute a machine learning system configured to identify discriminative, fine-grained features of an object in an image, wherein the machine learning system is configured to: receive the image of the object; apply a first set of filters of a model to the received image to generate an intermediate representation of the received image suitable as an input to both a second set of filters and a third set of filters of the model; apply a second set of filters of the model to the intermediate representation of the received image to generate part localization data for the object, wherein the part localization data for the object comprises data identifying one or more sub-parts of the object and one or more regions of the received image in which the one or more sub-parts of the object are located; apply a third set of filters to the intermediate representation of the received image to generate classification data for the object; and output the part localization data for the object and the classification data for the object, wherein the data identifying one or more sub-parts of the object and one or more regions of the received image in which the one or more sub-parts of the object are located along with the classification data for the object results in a more discriminative, fine-grained identification of one or more features of the object in the image. ### Claim 18 A method for training a first set of filters, a second set of filters, and a third set of filters of a model of a machine learning system, executed by processing circuitry of a computation engine, to identify discriminative, fine-grained features of an object in an image, the method comprising: applying, by the machine learning system, training data comprising images of objects labeled with corresponding part localization data for the objects and corresponding classification data for the objects to train the first set of filters and the third set of filters together to generate classification data for the objects, wherein the classification data for the objects comprises data identifying a subordinate category within a basic level category to which the objects belong, applying the training data to train the first set of filters and the second set of filters together to generate part localization data for the objects, wherein the part localization data for the objects comprises data identifying one or more sub-parts of each of the objects and one or more regions of the received image in which the one or more sub-parts of each of the objects are located; applying the training data to train the first set of filters, the second set of filters, and the third set of filters together to generate both the part localization data for the objects and the classification data for the objects; and outputting the model.