# US 2019/0318207 A1 · Rare instance classifiers > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0318207 A1 - Document ID: US20190318207A1 - Status: Published application - Kind: A1 - Application number: 16/447,842 - Filed: 2019-06-20 - Published: 2019-10-17 - Earliest claimed: 2017-06-22 - Assignees: Waymo LLC - Applicants: Waymo LLC - Inventors: Wan-Yen Lo; Abhijit Ogale; Yang Gao - Web page: https://patentgrep.com/patent/US20190318207A1 ## CPC - G06N 3/084: Backpropagation, e.g. using gradient descent - G05D 1/0246: {using a video camera in combination with image processing means} - G06F 18/24: Classification techniques - G06F 18/2414: {Smoothing the distance, e.g. radial basis function networks [RBFN]} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06V 20/582: {of traffic signs} - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract In some implementations, an image classification system of an autonomous or semi-autonomous vehicle is capable of improving multi-object classification by reducing repeated incorrect classification of objects that are considered rarely occurring objects. The system can include a common instance classifier that is trained to identify and recognize general objects (e.g., commonly occurring objects and rarely occurring objects) as belonging to specified object categories, and a rare instance classifier that is trained to compute one or more rarity scores representing likelihoods that an input image is correctly classified by the common instance classifier. The output of the rare instance classifier can be used to adjust the classification output of the common instance classifier such that the likelihood of input images being incorrectly classified is reduced. ## Claims ### Claim 1 A method of training a rare instance neural network, the method comprising: receiving a plurality of labeled training images; identifying a subset of the labeled training images that are likely to be misclassified by a common instance neural network that has been trained on the labeled training images to classify images into object categories; generating, from the subset of the labeled training images, training data for a rare instance neural network, the rare instance neural network being configured to process an input image to generate a rarity output that includes a rarity score that represents a likelihood that the input image will be incorrectly classified by the trained common instance neural network; and training the rare instance neural network on the training data to generate rarity scores for the subset of labeled training images that indicate that the subset of labeled training images are likely to be incorrectly classified by the common instance neural network. ### Claim 2 (depends on 1) The method of claim 1, wherein the rare instance neural network has fewer parameters than the common instance neural network. ### Claim 3 (depends on 1) The method of claim 1, wherein identifying a subset of the labeled training images that are likely to be misclassified by a common instance neural network that has been trained on the labeled training images to classify images into object categories comprises: processing the training images using the trained common instance neural network to identify training images that are misclassified by the common instance neural network. ### Claim 4 (depends on 1) The method of claim 1, wherein identifying a subset of the labeled training images that are likely to be misclassified by a common instance neural network that has been trained on the labeled training images to classify images into object categories comprises: identifying labeled training images that depict objects of object types that occur rarely in the labeled training images. ### Claim 5 (depends on 1) The method of claim 1, wherein generating the training data comprises: generating transformed images from one or more of the labeled training images. ### Claim 6 (depends on 1) The method of claim 1, wherein training the rare instance neural network further comprises: training the rare instance neural network to generate, for the subset of labeled training images, rarity scores that satisfy a predetermined threshold. ### Claim 7 (depends on 6) The method of claim 6, wherein training further comprises: training the rare instance neural network to generate, for labeled training images that are not in the subset of labeled training images, rarity scores that do not satisfy the predetermined threshold. ### Claim 8 A system comprising: one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising: receiving a plurality of labeled training images; identifying a subset of the labeled training images that are likely to be misclassified by a common instance neural network that has been trained on the labeled training images to classify images into object categories; generating, from the subset of the labeled training images, training data for a rare instance neural network, the rare instance neural network being configured to process an input image to generate a rarity output that includes a rarity score that represents a likelihood that the input image will be incorrectly classified by the trained common instance neural network; and training the rare instance neural network on the training data to generate rarity scores for the subset of labeled training images that indicate that the subset of labeled training images are likely to be incorrectly classified by the common instance neural network. ### Claim 9 (depends on 8) The system of claim 8, wherein the rare instance neural network has fewer parameters than the common instance neural network. ### Claim 10 (depends on 8) The system of claim 8, wherein identifying a subset of the labeled training images that are likely to be misclassified by a common instance neural network that has been trained on the labeled training images to classify images into object categories comprises: processing the training images using the trained common instance neural network to identify training images that are misclassified by the common instance neural network. ### Claim 11 (depends on 8) The system of claim 8, wherein identifying a subset of the labeled training images that are likely to be misclassified by a common instance neural network that has been trained on the labeled training images to classify images into object categories comprises: identifying labeled training images that depict objects of object types that occur rarely in the labeled training images. ### Claim 12 (depends on 8) The system of claim 8, wherein generating the training data comprises: generating transformed images from one or more of the labeled training images. ### Claim 13 (depends on 8) The system of claim 8, wherein training the rare instance neural network further comprises: training the rare instance neural network to generate, for the subset of labeled training images, rarity scores that satisfy a predetermined threshold. ### Claim 14 (depends on 13) The system of claim 13, wherein training further comprises: training the rare instance neural network to generate, for labeled training images that are not in the subset of labeled training images, rarity scores that do not satisfy the predetermined threshold. ### Claim 15 One or more non-transitory computer-readable storage media encoded with computer program instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: receiving a plurality of labeled training images; identifying a subset of the labeled training images that are likely to be misclassified by a common instance neural network that has been trained on the labeled training images to classify images into object categories; generating, from the subset of the labeled training images, training data for a rare instance neural network, the rare instance neural network being configured to process an input image to generate a rarity output that includes a rarity score that represents a likelihood that the input image will be incorrectly classified by the trained common instance neural network; and training the rare instance neural network on the training data to generate rarity scores for the subset of labeled training images that indicate that the subset of labeled training images are likely to be incorrectly classified by the common instance neural network. ### Claim 16 (depends on 15) The media of claim 15, wherein the rare instance neural network has fewer parameters than the common instance neural network. ### Claim 17 (depends on 15) The media of claim 15, wherein identifying a subset of the labeled training images that are likely to be misclassified by a common instance neural network that has been trained on the labeled training images to classify images into object categories comprises: processing the training images using the trained common instance neural network to identify training images that are misclassified by the common instance neural network. ### Claim 18 (depends on 15) The media of claim 15, wherein identifying a subset of the labeled training images that are likely to be misclassified by a common instance neural network that has been trained on the labeled training images to classify images into object categories comprises: identifying labeled training images that depict objects of object types that occur rarely in the labeled training images. ### Claim 19 (depends on 15) The media of claim 15, wherein generating the training data comprises: generating transformed images from one or more of the labeled training images. ### Claim 20 (depends on 15) The media of claim 15, wherein training the rare instance neural network further comprises: training the rare instance neural network to generate, for the subset of labeled training images, rarity scores that satisfy a predetermined threshold.