# US 2019/0377987 A1 · Discriminative Caption Generation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0377987 A1 - Document ID: US20190377987A1 - Status: Published application - Kind: A1 - Application number: 16/004,395 - Filed: 2018-06-10 - Published: 2019-12-12 - Earliest claimed: 2018-06-10 - Assignees: Adobe Inc. - Applicants: Adobe Inc. - Inventors: Brian Lynn Price; Ruotian Luo; Scott David Cohen - Web page: https://patentgrep.com/patent/US20190377987A1 ## CPC - G06F 40/56: Natural language generation - G06F 16/583: using metadata automatically derived from the content - G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} - G06F 18/241: relating to the classification model, e.g. parametric or non-parametric approaches - G06F 40/216: using statistical methods - G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) - G06F 40/44: Statistical methods, e.g. probability models - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06N 20/00: Machine learning - G06V 10/7788: {the supervisor being a human, e.g. interactive learning with a human teacher} - G06V 20/30: in albums, collections or shared content, e.g. social network photos or video - G06V 20/35: {Categorising the entire scene, e.g. birthday party or wedding scene} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A discriminative captioning system generates captions for digital images that can be used to tell two digital images apart. The discriminative captioning system includes a machine learning system that is trained by a discriminative captioning training system that includes a retrieval machine learning system. For training, a digital image is input to the caption generation machine learning system, which generates a caption for the digital image. The digital image and the generated caption, as well as a set of additional images, are input to the retrieval machine learning system. The retrieval machine learning system generates a discriminability loss that indicates how well the retrieval machine learning system is able to use the caption to discriminate between the digital image and each image in the set of additional digital images. This discriminability loss is used to train the caption generation machine learning system. ## Claims ### Claim 1 In a digital medium environment to generate a discriminative caption for a digital image, a method implemented by a computing device, the method comprising: obtaining, by the computing device, a first digital image; generating, by the computing device using a caption generation machine learning system, a caption for the first digital image; providing, by the computing device, the caption, the first digital image, and a set of additional digital images to a retrieval machine learning system; generating, by the retrieval machine learning system of the computing device and based on the caption, the first digital image, and the set of additional digital images, a discriminability loss that indicates how well the retrieval machine learning system is able to use the caption to discriminate between the first digital image and each image in the set of additional digital images; using, by the computing device, the discriminability loss to train the caption generation machine learning system; and using the trained caption generation machine learning system to generate a caption for a second digital image. ### Claim 2 (depends on 1) The method as described in claim 1, further comprising training the retrieval machine learning system using a contrastive loss technique that has a desired caption for a target image be more similar to a target image than to each digital image in a set of distractor digital images. ### Claim 3 (depends on 2) The method as described in claim 2, the contrastive loss technique further having the desired caption for the target image be more similar to the target image than other captions corresponding to the set of distractor digital images are to the target image. ### Claim 4 (depends on 1) The method as described in claim 1, further comprising training the retrieval machine learning system using a pairwise loss technique that forces a target image and a desired caption for the image to have similarity higher than a distractor image by a margin. ### Claim 5 (depends on 1) The method as described in claim 1, the using the discriminability loss to train the caption generation machine learning system comprising training the caption generation machine learning system to minimize the discriminability loss. ### Claim 6 (depends on 1) The method as described in claim 1, wherein the discriminability loss is differentiable. ### Claim 7 (depends on 1) The method as described in claim 1, further comprising: providing, by the computing device, the caption and the first digital image to a caption likelihood module; generating, by the caption likelihood module, feedback that indicates a likelihood estimation that the caption describes the first digital image; and using, by the computing device, the feedback to train the caption generation machine learning system. ### Claim 8 In a digital medium environment to generate a discriminative caption for a digital image, a method implemented by a computing device, the method comprising: training, by the computing device, a retrieval machine learning system to generate a discriminability loss that indicates how well the retrieval machine learning system is able to use a caption to discriminate between a digital image for which the caption was generated and each image in a set of additional digital images; training, by the computing device, a caption generation machine learning system to generate captions for digital images using the discriminability loss from the retrieval machine learning system; and deploying the caption generation machine learning system to generate captions for digital images without the retrieval machine learning system. ### Claim 9 (depends on 8) The method as described in claim 8, the training the retrieval machine learning system comprising using a contrastive loss technique that has a desired caption for a target image be more similar to the target image than other captions corresponding to a set of distractor digital images is to the target image, and that has the desired caption for the target image be more similar to the target image than to each digital image in the set of distractor digital images. ### Claim 10 (depends on 8) The method as described in claim 8, the training the retrieval machine learning system comprising using a pairwise loss technique that forces a target image and a desired caption for the image to have similarity higher than a distractor image by a margin. ### Claim 11 (depends on 8) The method as described in claim 8, the training the caption generation machine learning system comprising training the caption generation machine learning system to minimize the discriminability loss. ### Claim 12 (depends on 8) The method as described in claim 8, wherein the discriminability loss is differentiable. ### Claim 13 (depends on 8) The method as described in claim 8, further comprising using a caption likelihood module to generate feedback that indicates a likelihood estimation that the caption describes the digital image, and the training the caption generation machine learning system further comprising training the caption generation machine learning system to generated captions for digital images using the feedback form the caption likelihood module. ### Claim 14 In a digital medium environment to generate a discriminative caption for a digital image, a system comprising: a caption generation machine learning system implemented at least partially in hardware of a computing device to generate a caption for a first digital image; a retrieval machine learning system implemented at least partially in hardware of the computing device to generate, based on the caption, the first digital image, and a set of additional digital images, a discriminability loss that indicates how well the retrieval machine learning system is able to use the caption to discriminate between the first digital image and each image in the set of additional digital images; and the caption generation machine learning system further to use the discriminability loss to train the caption generation machine learning system. ### Claim 15 (depends on 14) The system as described in claim 14, the retrieval machine learning system having been trained using a contrastive loss technique that has a desired caption for a target image be more similar to the target image than other captions corresponding to a set of distractor digital images is to the target image. ### Claim 16 (depends on 15) The system as described in claim 15, the contrastive loss technique further having the desired caption for the target image be more similar to the target image than to each digital image in the set of distractor digital images. ### Claim 17 (depends on 14) The system as described in claim 14, the retrieval machine learning system having been trained using a pairwise loss technique that forces a target image and a desired caption for the image to have similarity higher than a distractor image by a margin. ### Claim 18 (depends on 14) The system as described in claim 14, the caption generation machine learning system being further to use the discriminability loss to train the caption generation machine learning system by minimizing the discriminability loss. ### Claim 19 (depends on 14) The system as described in claim 14, further comprising a discriminability loss conversion module, implemented at least partially in hardware, to convert a discriminability loss generated by the retrieval machine learning system to a differentiable discriminability loss. ### Claim 20 (depends on 14) The system as described in claim 14, further comprising a caption likelihood module, implemented at least partially in hardware, to generate feedback that indicates a likelihood estimation that the caption describes the digital image, and the caption generation machine learning system being further to use the feedback from the caption likelihood module to train the caption generation machine learning system.