Learning visual concepts using neural networks

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a symbol encoder neural network and a symbol decoder neural network using a trained image encoder neural network having a plurality of image encoder parameters, wherein the image encoder neural network is configured to:

• receive an input image;
• and
• process the input image in accordance with the image encoder parameters to generate an image encoder output that parameterizes a distribution over possible values for each of a plurality of generative visual factors of variation;
• wherein the symbol encoder neural network has a plurality of symbol encoder parameters and is configured to:
• receive a symbol input comprising one or more symbols from a vocabulary of symbols;
• and
• process the symbol input in accordance with the symbol encoder parameters to generate a symbol encoder output that parameterizes a distribution over possible values for each of the plurality of generative visual factors of variation;
• wherein the symbol decoder neural network has a plurality of symbol decoder parameters and is configured to:
• receive a symbol decoder input comprising a respective value for each of the plurality of generative visual factors;
• and
• process the symbol decoder input in accordance with the symbol decoder parameters to generate a symbol output that includes one or more symbols from the vocabulary of symbols;
• and
• wherein the training comprises determining trained values of the symbol encoder parameters and the symbol decoder parameters by repeatedly:
• receiving a training symbol input comprising one or more symbols from the vocabulary and a training image that matches the training symbol input;
• processing the training image using the image encoder neural network in accordance with the image encoder parameters to generate a training image encoder output for the training image;
• processing the training symbol input using the symbol encoder neural network in accordance with current values of the symbol encoder parameters to generate a training symbol encoder output for the training symbol input;
• sampling, from a distribution parameterized by the training symbol encoder output, a respective value for each of the plurality of visual factors;
• processing a training symbol decoder input comprising the respective values for the visual factors using the symbol decoder neural network in accordance with current values of the symbol decoder parameters to generate a training symbol output;
• determining a gradient with respect to the symbol encoder parameters and the symbol decoder parameters of an objective function that includes (i) a variational auto encoder (VAE) objective and (ii) a term that encourages alignment between the training symbol encoder output and the training image encoder output;
• and
• determining an update to the current values of the symbol encoder parameters and the symbol decoder parameters using the gradient.

A method performed by one or more computers and for training a symbol encoder neural network and a symbol decoder neural network using a trained image encoder neural network having a plurality of image encoder parameters, wherein the image encoder neural network is configured to:

• receive an input image;
• and
• process the input image in accordance with the image encoder parameters to generate an image encoder output that parameterizes a distribution over possible values for each of a plurality of generative visual factors of variation;
• wherein the symbol encoder neural network has a plurality of symbol encoder parameters and is configured to:
• receive a symbol input comprising one or more symbols from a vocabulary of symbols;
• and
• process the symbol input in accordance with the symbol encoder parameters to generate a symbol encoder output that parameterizes a distribution over possible values for each of the plurality of generative visual factors of variation;
• wherein the symbol decoder neural network has a plurality of symbol decoder parameters and is configured to:
• receive a symbol decoder input comprising a respective value for each of the plurality of generative visual factors;
• and
• process the symbol decoder input in accordance with the symbol decoder parameters to generate a symbol output that includes one or more symbols from the vocabulary of symbols;
• and
• wherein the method comprises determining trained values of the symbol encoder parameters and the symbol decoder parameters by repeatedly:
• receiving a training symbol input comprising one or more symbols from the vocabulary and a training image that matches the training symbol input;
• processing the training image using the image encoder neural network in accordance with the image encoder parameters to generate a training image encoder output for the training image;
• processing the training symbol input using the symbol encoder neural network in accordance with current values of the symbol encoder parameters to generate a training symbol encoder output for the training symbol input;
• sampling, from a distribution parameterized by the training symbol encoder output, a respective value for each of the plurality of visual factors;
• processing a training symbol decoder input comprising the respective values for the visual factors using the symbol decoder neural network in accordance with current values of the symbol decoder parameters to generate a training symbol output;
• determining a gradient with respect to the symbol encoder parameters and the symbol decoder parameters of an objective function that includes (i) a variational auto encoder (VAE) objective and (ii) a term that encourages alignment between the training symbol encoder output and the training image encoder output;
• and
• determining an update to the current values of the symbol encoder parameters and the symbol decoder parameters using the gradient.