Open-vocabulary object detection with vision and language supervision

A computer program product for training a neural network to predict object categories without manual annotation, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:

• feeding training datasets including at least images and data annotations to an object detection neural network;
• converting, by a text prompter, the data annotations into natural text inputs;
• converting, by a text embedder, the natural text inputs into embeddings;
• minimizing objective functions, including a localization loss, a detection classification loss, a weak alignment loss that determines a soft assignment between all detections from the images and words in the data annotations, and a global alignment loss that determines feature embeddings that maximize similarity between the images and the data annotations, that utilize the embeddings based on a determined alignment for summarizing the images and the data annotations during training to adjust parameters of the object detection neural network;
• and
• generating control instructions for an autonomous vehicle to avoid a collision by predicting, by the object detection neural network, objects within images and videos of a traffic scene that includes the autonomous vehicle.

A computer processing system for training a neural network to predict object categories without manual annotation, comprising:

• a memory device for storing program code;
• and
• a processor device, operatively coupled to the memory device, for running the program code to:
• feed training datasets including at least images and data annotations to an object detection neural network;
• convert, by a text prompter, the data annotations into natural text inputs;
• convert, by a text embedder, the natural text inputs into embeddings;
• minimize objective functions, including a localization loss, a detection classification loss, a weak alignment loss that determines a soft assignment between all detections from the images and words in the data annotations, and a global alignment loss that determines feature embeddings that maximize similarity between the images and the data annotations, that utilize the embeddings based on a determined alignment for summarizing the images and the data annotations during training to adjust parameters of the object detection neural network;
• and
• generate control instructions for an autonomous vehicle to avoid a collision by predicting, with the object detection neural network, objects within images and videos of a traffic scene that includes the autonomous vehicle.