Performing perception tasks by leveraging auto-regressive neural networks

A system comprising:

• one or more computers;
• and
• one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the respective operations comprising:
• obtaining one or more query images and a plurality of context images;
• generating a sequence of discrete tokens representing the context images;
• generating one or more continuous tokens representing the one or more query images;
• processing an input comprising the sequence of discrete tokens representing the context images and the one or more continuous tokens representing the one or more query images using a token processing neural network to generate one or more updated continuous tokens representing the one or more query images;
• and
• processing the one or more updated continuous tokens to generate a respective output for each of one or more prediction tasks.

One or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• obtaining one or more query images and a plurality of context images;
• generating a sequence of discrete tokens representing the context images;
• generating one or more continuous tokens representing the one or more query images;
• processing an input comprising the sequence of discrete tokens representing the context images and the one or more continuous tokens representing the one or more query images using a token processing neural network to generate one or more updated continuous tokens representing the one or more query images;
• and
• processing the one or more updated continuous tokens to generate a respective output for each of one or more prediction tasks.