Image based object detection

A method comprising:

• accessing an image from an image sensor;
• applying a convolutional neural network to the image to obtain localization data to detect an object depicted in the image and to obtain classification data to classify the object, the classification data comprising predictions corresponding to respective classes that are nodes in a hierarchical tree of classes, in which the convolutional neural network has been trained in part using training images with associated localization labels and classification labels and has been trained in part using training images with associated classification labels that lack localization labels;
• annotating the image based on the localization data and the classification data to obtain an annotated image;
• and
• storing, displaying, or transmitting the annotated image.

A method for training a convolutional neural network for object detection and classification comprising:

• training the convolutional neural network using training images with associated localization labels and classification labels, wherein the convolutional neural network outputs localization data and classification data based on an image that is input to the convolutional neural network, the classification data comprising predictions corresponding to respective classes that are nodes in a hierarchical tree of classes;
• training the convolutional neural network using training images with associated classification labels that lack localization labels;
• and
• storing or transmitting the convolutional neural network.

One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

• access the image from the image sensor;
• apply a convolutional neural network to the image to obtain localization data to detect an object depicted in the image and to obtain classification data to classify the object, the classification data comprising predictions corresponding to respective classes that are nodes in a hierarchical tree of classes, in which the convolutional neural network has been trained in part using training images with associated localization labels and classification labels and has been trained in part using training images with associated classification labels that lack localization labels;
• and
• annotate the image based on the localization data and the classification data.

A system comprising:

• a processing apparatus that is configured to:
• train the convolutional neural network using training images with associated localization labels and classification labels, wherein the convolutional neural network outputs localization data and classification data based on an image that is input to the convolutional neural network, the classification data comprising predictions corresponding to respective classes that are nodes in a hierarchical tree of classes;
• train the convolutional neural network using training images with associated classification labels that lack localization labels;
• and
• store or transmit the convolutional neural network.

One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

• train the convolutional neural network using training images with associated localization labels and classification labels, wherein the convolutional neural network outputs localization data and classification data based on an image that is input to the convolutional neural network, the classification data comprising predictions corresponding to respective classes that are nodes in a hierarchical tree of classes;
• train the convolutional neural network using training images with associated classification labels that lack localization labels;
• and
• store or transmit the convolutional neural network.