Back to results
US 2023/0401835 A1Published application
Training a speaker neural network using one or more listener neural networks
AssigneeDeepMind Technologies Limited
InventorsAaditya K. Singh, Fengning Ding, Felix George Hill, Andrew Kyle Lampinen
Life of this patent
Estimating the term…
Abstract
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a speaker neural network using one or more listener neural networks.
Claims22
A method performed by one or more computers, the method comprising:
- obtaining a set of one or more training images;
- for each training image in the set:
- processing the training image using a speaker neural network to generate a text caption for the training image;
- generating a plurality of listener inputs, each listener input comprising (i) the text caption for the training image and (ii) a respective set of images, wherein the respective set of images includes a corresponding version of the training image and a corresponding set of one or more distractor images that are each different from the training image;
- for each listener input, processing the listener input using a respective listener neural network from a set of one or more listener neural networks to generate a respective match score for each image in the respective set of images in the listener input;
- and
- generating a reward for the training image based at least in part on the respective match scores for the training image generated by processing each of the listener inputs;
- and
- training the speaker neural network using the rewards for the training images.
6 more elements
A system comprising:
- one or more computers;
- and
- one or more storage devices storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
- obtaining a set of one or more training images;
- for each training image in the set:
- processing the training image using a speaker neural network to generate a text caption for the training image;
- generating a plurality of listener inputs, each listener input comprising (i) the text caption for the training image and (ii) a respective set of images, wherein the respective set of images includes a corresponding version of the training image and a corresponding set of one or more distractor images that are each different from the training image;
- for each listener input, processing the listener input using a respective listener neural network from a set of one or more listener neural networks to generate a respective match score for each image in the respective set of images in the listener input;
- and
- generating a reward for the training image based at least in part on the respective match scores for the training image generated by processing each of the listener inputs;
- and
- training the speaker neural network using the rewards for the training images.
9 more elements
One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
- obtaining a set of one or more training images;
- for each training image in the set:
- processing the training image using a speaker neural network to generate a text caption for the training image;
- generating a plurality of listener inputs, each listener input comprising (i) the text caption for the training image and (ii) a respective set of images, wherein the respective set of images includes a corresponding version of the training image and a corresponding set of one or more distractor images that are each different from the training image;
- for each listener input, processing the listener input using a respective listener neural network from a set of one or more listener neural networks to generate a respective match score for each image in the respective set of images in the listener input;
- and
- generating a reward for the training image based at least in part on the respective match scores for the training image generated by processing each of the listener inputs;
- and
- training the speaker neural network using the rewards for the training images.
6 more elements
Dependencies are inferred from the claim text.