Computer vision neural networks with learned tokenization
AssigneeGoogle Inc.
InventorsMichael Sahngwon Ryoo, Anthony Jacob Piergiovanni, Anelia Angelova, Anurag Arnab, Mostafa Dehghani
Life of this patent
Estimating the term…
Abstract
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing a network input using a computer vision neural network with learned tokenization.
Claims20
A method performed by one or more computers, the method comprising:
- receiving a network input comprising one or more input images;
- processing the one or more input images to generate an intermediate tensor that comprises one or more frames, each frame having a first spatial dimension;
- applying a tokenizer to the respective frames to map each frame to a fixed number of token vectors, wherein the tokenizer applies a learned, input-dependent transformation to each frame to map the frame to a set of token vectors;
- and
- processing the token vectors for the one or more frames using one or more subsequent neural network components to generate a network output for a computer vision task for the one or more input images.
A system comprising one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
- receiving a network input comprising one or more input images;
- processing the one or more input images to generate an intermediate tensor that comprises one or more frames, each frame having a first spatial dimension;
- applying a tokenizer to the respective frames to map each frame to a fixed number of token vectors, wherein the tokenizer applies a learned, input-dependent transformation to each frame to map the frame to a set of token vectors;
- and
- processing the token vectors for the one or more frames using one or more subsequent neural network components to generate a network output for a computer vision task for the one or more input images.
One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
- receiving a network input comprising one or more input images;
- processing the one or more input images to generate an intermediate tensor that comprises one or more frames, each frame having a first spatial dimension;
- applying a tokenizer to the respective frames to map each frame to a fixed number of token vectors, wherein the tokenizer applies a learned, input-dependent transformation to each frame to map the frame to a set of token vectors;
- and
- processing the token vectors for the one or more frames using one or more subsequent neural network components to generate a network output for a computer vision task for the one or more input images.
Dependencies are inferred from the claim text.