Open-vocabulary object detection in images

A method performed by one or more computers, the method comprising:

• obtaining:
• (i) an input image, and (ii) a set of one or more query embeddings, wherein each query embedding represents a respective category of object, wherein one or more of the query embeddings are image-conditioned query embeddings are each derived from a respective set of one or more query images;
• processing the input image and the set of query embeddings using an object detection neural network to generate object detection data for the input image, comprising:
• processing the input image using an image encoding subnetwork of the object detection neural network to generate a set of object embeddings;
• processing each object embedding using a localization subnetwork of the object detection neural network to generate localization data defining a corresponding region of the input image;
• and
• processing:
• (i) the set of object embeddings, and (ii) the set of query embeddings, using a classification subnetwork of the object detection neural network to generate, for each object embedding, a respective classification score distribution over the set of query embeddings,
• wherein the respective classification score distribution for each of the object embeddings defines, for each query embedding, a likelihood that the region of the input image corresponding to the object embedding depicts an object that is included in the category represented by the query embedding.

A system comprising:

• one or more computers;
• and
• one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
• obtaining:
• (i) an input image, and (ii) a set of one or more query embeddings, wherein each query embedding represents a respective category of object, wherein one or more of the query embeddings are image-conditioned query embeddings are each derived from a respective set of one or more query images;
• processing the input image and the set of query embeddings using an object detection neural network to generate object detection data for the input image, comprising:
• processing the input image using an image encoding subnetwork of the object detection neural network to generate a set of object embeddings;
• processing each object embedding using a localization subnetwork of the object detection neural network to generate localization data defining a corresponding region of the input image;
• and
• processing:
• (i) the set of object embeddings, and (ii) the set of query embeddings, using a classification subnetwork of the object detection neural network to generate, for each object embedding, a respective classification score distribution over the set of query embeddings,
• wherein the respective classification score distribution for each of the object embeddings defines, for each query embedding, a likelihood that the region of the input image corresponding to the object embedding depicts an object that is included in the category represented by the query embedding.

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• obtaining:
• (i) an input image, and (ii) a set of one or more query embeddings, wherein each query embedding represents a respective category of object, wherein one or more of the query embeddings are image-conditioned query embeddings are each derived from a respective set of one or more query images;
• processing the input image and the set of query embeddings using an object detection neural network to generate object detection data for the input image, comprising:
• processing the input image using an image encoding subnetwork of the object detection neural network to generate a set of object embeddings;
• processing each object embedding using a localization subnetwork of the object detection neural network to generate localization data defining a corresponding region of the input image;
• and
• processing:
• (i) the set of object embeddings, and (ii) the set of query embeddings, using a classification subnetwork of the object detection neural network to generate, for each object embedding, a respective classification score distribution over the set of query embeddings,
• wherein the respective classification score distribution for each of the object embeddings defines, for each query embedding, a likelihood that the region of the input image corresponding to the object embedding depicts an object that is included in the category represented by the query embedding.