Computer vision models using global and local information

A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement a neural network that is configured to perform a machine learning task by receiving an input image and processing the input image to generate a network output, the neural network comprising:

• a plurality of wrapped convolutional neural network layers, wherein each wrapped convolutional neural network layer of the plurality of wrapped convolutional neural network layers comprises:
• a respective convolutional neural network layer configured to receive, for each time step of a plurality of time steps, a layer input and to process the layer input to generate an initial output for the current time step, and
• a respective note-taking module configured to receive the initial output and to process the initial output to generate a feature vector for the current time step, wherein the feature vector represents local information of the wrapped convolutional neural network layer;
• and
• a summarization module configured to receive the feature vectors generated by the respective note-taking modules for the current time step and to process the feature vectors to generate a revision vector for the current time step, wherein the revision vector represents global information of the plurality of wrapped convolutional neural network layers;
• wherein the respective note-taking module comprises a pre-processing layer, a pooling layer, and a multi-layer perceptron (MLP) layer, and wherein the respective note-taking module is configured to process the initial output to generate the feature vector for the current time step by:
• applying, using the pre-processing layer, a pre-processing operation to the initial output to generate a first intermediate output,
• applying, using the pooling layer, a pooling operation to the first intermediate output to generate a second intermediate output,
• flattening the second intermediate output to generate a third intermediate output, and
• processing, using the MLP layer, the third intermediate output to generate the feature vector for the current time step.

A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement a neural network that is configured to perform a machine learning task by receiving an input image and processing the input image to generate a network output, the neural network comprising:

• a plurality of wrapped convolutional neural network layers, wherein each wrapped convolutional neural network layer of the plurality of wrapped convolutional neural network layers comprises:
• a respective convolutional neural network layer configured to receive, for each time step of a plurality of time steps, a layer input and to process the layer input to generate an initial output for the current time step, and
• a respective note-taking module configured to receive the initial output and to process the initial output to generate a feature vector for the current time step, wherein the feature vector represents local information of the wrapped convolutional neural network layer;
• and
• a summarization module configured to receive the feature vectors generated by the respective note-taking modules for the current time step and to process the feature vectors to generate a revision vector for the current time step, wherein the revision vector represents global information of the plurality of wrapped convolutional neural network layers;
• wherein the respective note-taking module comprises a pooling layer and a multi-layer perceptron (MLP) layer, and wherein the respective note-taking module is configured to process the initial output to generate the feature vector for the current time step by:
• applying, using a pooling layer, a pooling operation to the initial output to generate a second intermediate output,
• flattening the second intermediate output to generate a third intermediate output, and
• processing, using the MLP layer, the third intermediate output to generate the feature vector for the current time step.

A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement a neural network that is configured to perform a machine learning task by receiving an input image and processing the input image to generate a network output, the neural network comprising:

• a plurality of wrapped convolutional neural network layers, wherein each wrapped convolutional neural network layer of the plurality of wrapped convolutional neural network layers comprises:
• a respective convolutional neural network layer configured to receive, for each time step of a plurality of time steps, a layer input and to process the layer input to generate an initial output for the current time step, and
• a respective note-taking module configured to receive the initial output and to process the initial output to generate a feature vector for the current time step, wherein the feature vector represents local information of the wrapped convolutional neural network layer;
• and
• a summarization module configured to receive the feature vectors generated by the respective note-taking modules for the current time step and to process the feature vectors to generate a revision vector for the current time step, wherein the revision vector represents global information of the plurality of wrapped convolutional neural network layers;
• wherein the summarization module comprises a second multi-layer perception (MLP) layer and is configured to process the feature vectors to generate a revision vector for the current time step by:
• concatenating the feature vectors generated by the respective note-taking modules for the current time step to generate a concatenated feature vector, and
• processing the concatenated feature vector using the second MLP layer to generate the revision vector for the current time step.

A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement a neural network that is configured to perform a machine learning task by receiving an input image and processing the input image to generate a network output, the neural network comprising:

• a plurality of wrapped convolutional neural network layers, wherein each wrapped convolutional neural network layer of the plurality of wrapped convolutional neural network layers comprises:
• a respective convolutional neural network layer configured to receive, for each time step of a plurality of time steps, a layer input and to process the layer input to generate an initial output for the current time step, and
• a respective note-taking module configured to receive the initial output and to process the initial output to generate a feature vector for the current time step, wherein the feature vector represents local information of the wrapped convolutional neural network layer;
• and
• a summarization module configured to receive the feature vectors generated by the respective note-taking modules for the current time step and to process the feature vectors to generate a revision vector for the current time step, wherein the revision vector represents global information of the plurality of wrapped convolutional neural network layers;
• wherein the initial output is a three-dimensional (3D) vector.

A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement a neural network that is configured to perform a machine learning task by receiving an input image and processing the input image to generate a network output, the neural network comprising:

• a plurality of wrapped convolutional neural network layers, wherein each wrapped convolutional neural network layer of the plurality of wrapped convolutional neural network layers comprises:
• a respective convolutional neural network layer configured to receive, for each time step of a plurality of time steps, a layer input and to process the layer input to generate an initial output for the current time step, and
• a respective note-taking module configured to receive the initial output and to process the initial output to generate a feature vector for the current time step, wherein the feature vector represents local information of the wrapped convolutional neural network layer;
• and
• a summarization module configured to receive the feature vectors generated by the respective note-taking modules for the current time step and to process the feature vectors to generate a revision vector for the current time step, wherein the revision vector represents global information of the plurality of wrapped convolutional neural network layers;
• wherein the revision vector is a one-dimensional (1D) vector.

A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement a neural network that is configured to perform a machine learning task by receiving an input image and processing the input image to generate a network output, the neural network comprising:

• a plurality of wrapped convolutional neural network layers, wherein each wrapped convolutional neural network layer of the plurality of wrapped convolutional neural network layers comprises:
• a respective convolutional neural network layer configured to receive, for each time step of a plurality of time steps, a layer input and to process the layer input to generate an initial output for the current time step, and
• a respective note-taking module configured to receive the initial output and to process the initial output to generate a feature vector for the current time step, wherein the feature vector represents local information of the wrapped convolutional neural network layer;
• and
• a summarization module configured to receive the feature vectors generated by the respective note-taking modules for the current time step and to process the feature vectors to generate a revision vector for the current time step, wherein the revision vector represents global information of the plurality of wrapped convolutional neural network layers;
• wherein the neural network is a Generative Adversarial Network (GAN).

A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement a neural network that is configured to perform a machine learning task by receiving an input image and processing the input image to generate a network output, the neural network comprising:

• a plurality of wrapped convolutional neural network layers, wherein each wrapped convolutional neural network layer of the plurality of wrapped convolutional neural network layers comprises:
• a respective convolutional neural network layer configured to receive, for each time step of a plurality of time steps, a layer input and to process the layer input to generate an initial output for the current time step, and
• a respective note-taking module configured to receive the initial output and to process the initial output to generate a feature vector for the current time step, wherein the feature vector represents local information of the wrapped convolutional neural network layer;
• and
• a summarization module configured to receive the feature vectors generated by the respective note-taking modules for the current time step and to process the feature vectors to generate a revision vector for the current time step, wherein the revision vector represents global information of the plurality of wrapped convolutional neural network layers;
• the machine learning task is one of the following:
• a super resolution task, a depth estimation task, a style transfer task, an image classification task, an object recognition task, an optical character recognition (OCR) task, or a semantic segmentation task.

A computer-implemented method comprising:

• performing the following operations at each of a plurality of time steps:
• for each of a plurality of wrapped convolutional neural network layers of a neural network:
• receiving, by the wrapped convolutional neural network layer, a layer input and processing the layer input to generate an initial output for the current time step, and
• receiving, by the wrapped convolutional neural network layer, the initial output and processing the initial output to generate a feature vector for the current time step, wherein the feature vector represents local information of the wrapped convolutional neural network layer;
• and
• receiving, by a summarization module of the neural network, the feature vectors generated for the current time step and processing the feature vectors to generate a revision vector for the current time step, wherein the revision vector represents global information of the plurality of wrapped convolutional neural network layers;
• further comprising:
• at each time step after a first time step in the plurality of time steps, performing the following operations for each of the plurality of wrapped convolutional neural network layers:
• receiving an amendment module input for the current time step, the amendment module input comprising one or more of:
• (i) a previous revision vector generated by the summarization module for a previous time step, (ii) a previous feature vector generated by the wrapped convolutional neural network layer for the previous time step, or (iii) the feature vector generated by the wrapped convolutional neural network layer for the current time step,
• processing the amendment module input to generate a set of amendment parameters, and
• processing the initial output for the current time step based on the set of amendment parameters to generate a layer output for the wrapped convolutional neural network layer.

A computer-implemented method comprising:

• performing the following operations at each of a plurality of time steps:
• for each of a plurality of wrapped convolutional neural network layers of a neural network:
• receiving, by the wrapped convolutional neural network layer, a layer input and processing the layer input to generate an initial output for the current time step, and
• receiving, by the wrapped convolutional neural network layer, the initial output and processing the initial output to generate a feature vector for the current time step, wherein the feature vector represents local information of the wrapped convolutional neural network layer;
• and
• receiving, by a summarization module of the neural network, the feature vectors generated for the current time step and processing the feature vectors to generate a revision vector for the current time step, wherein the revision vector represents global information of the plurality of wrapped convolutional neural network layers;
• wherein the layer input is one of the following:
• (i) an input image of the neural network, (ii) an input derived from the input image, (iii) a layer output of a preceding wrapped convolutional neural network layer in the plurality of wrapped convolutional neural network layers, or (iv) an output of another component of the neural network.

One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the following operations at each of a plurality of time steps:

• for each of a plurality of wrapped convolutional neural network layers of a neural network:
• receiving, by the wrapped convolutional neural network layer, a layer input and processing the layer input to generate an initial output for the current time step, and
• receiving, by the wrapped convolutional neural network layer, the initial output and processing the initial output to generate a feature vector for the current time step, wherein the feature vector represents local information of the wrapped convolutional neural network layer;
• and
• receiving, by a summarization module of the neural network, the feature vectors generated for the current time step and processing the feature vectors to generate a revision vector for the current time step, wherein the revision vector represents global information of the plurality of wrapped convolutional neural network layers;
• at each time step after the first time step in the plurality of time steps, performing the following operations for each of the plurality of wrapped convolutional neural network layers:
• receiving an amendment module input for the current time step, the amendment module input comprising one or more of:
• (i) a previous revision vector generated by the summarization module for a previous time step, (ii) a previous feature vector generated by the wrapped convolutional neural network layer for the previous time step, or (iii) the feature vector generated by the wrapped convolutional neural network layer for the current time step,
• processing the amendment module input to generate a set of amendment parameters, and
• processing the initial output for the current time step based on the set of amendment parameters to generate a layer output for the wrapped convolutional neural network layer.