# US 11,907,850 B2 · Training image-to-image translation neural networks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,850 B2 - Document ID: US11907850B2 - Status: Granted - Kind: B2 - Application number: 17/454,516 - Filed: 2021-11-11 - Published: 2022-03-03 - Granted: 2024-02-20 - Earliest claimed: 2018-11-19 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Rui Zhang; Jia Li; Tomas Jon Pfister - Web page: https://patentgrep.com/patent/US11907850B2 ## CPC - G06N 3/084: Backpropagation, e.g. using gradient descent - G06V 10/778: Active pattern-learning, e.g. online learning of image or video features - G06F 18/22: Matching criteria, e.g. proximity measures - G06F 18/2413: based on distances to training or reference patterns - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/094: Adversarial learning - G06V 10/761: {Proximity, similarity or dissimilarity measures} - G06V 10/764: using classification, e.g. of video objects - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/82: using neural networks - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method includes obtaining a source training dataset that includes a plurality of source training images and obtaining a target training dataset that includes a plurality of target training images. For each source training image, the method includes translating, using the forward generator neural network G, the source training image to a respective translated target image according to current values of forward generator parameters. For each target training image, the method includes translating, using a backward generator neural network F, the target training image to a respective translated source image according to current values of backward generator parameters. The method also includes training the forward generator neural network G jointly with the backward generator neural network F by adjusting the current values of the forward generator parameters and the backward generator parameters to optimize an objective function. ## Claims ### Claim 1 A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising: obtaining a source training image in a source domain; selecting, from the source training image, a first source patch and a second source patch each representative of respective portions of the source training image; translating, using a forward generator neural network, the source training image to a translated target image in a target domain according to parameters of the forward generator neural network, the translated target image comprising a third source patch and a fourth source patch each representative of respective portions of the translated target image, the third source patch and the fourth source patch of the translated target image corresponding to the first source patch and the second source patch of the source training image; determining a first energy distance between the first source patch and the second source patch of the source training image; determining a second energy distance between the third source patch and the fourth source patch of the translated target image; generating a harmonic loss based on the first energy distance and the second energy distance; and updating, using the harmonic loss, the parameters of the forward generator neural network. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the operations further comprise determining, by a target discriminator neural network, a probability that the translated target image belongs to the target domain. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the operations further comprise: obtaining a target training image in the target domain; selecting, from the target training image, a first target patch and a second target patch each representative of respective portions of the target training image; translating, using a backward generator neural network, the target training image to a translated source image in the source domain according to parameters of the backward generator neural network, the translated source image comprising a third target patch and a fourth target patch each representative of respective portions of the translated source image, the third target patch and the fourth target patch of the translated source image corresponding to the first target patch and the second target patch of the target training image; determining a third energy distance between the first target patch and the second target patch of the target training image; determining a fourth energy distance between the third target patch and the fourth target patch of the translated source image; generating a second harmonic loss based on the third energy distance and the fourth energy distance; and updating, using the second harmonic loss, the parameters of the backward generator neural network. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, wherein the operations further comprise determining, by a source discriminator neural network, a probability that the translated source image belongs to the source domain. ### Claim 5 (depends on 3) The computer-implemented method of claim 3, wherein updating the parameters of the forward generator neural network comprises optimizing an objective function. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein the objective function comprises an adversarial loss component that ensures translation accuracy of the forward generator neural network and the backward generator neural network. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein the objective function further comprises a circularity loss component that ensures that the forward generator neural network and the backward generator neural network comprise inverse translation operations. ### Claim 8 (depends on 7) The computer-implemented method of claim 7, wherein the objective function comprises a weighted combination adversarial loss terms and the circularity loss component. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein selecting the first source patch and the second source patch of the source training image comprises determining whether the first energy distance between the first source patch and the second source patch of the source training image satisfies a threshold value. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the operations further comprise jointly updating the parameters of the forward generator neural network and parameters of a backward generator neural network. ### Claim 11 A system comprising: data processing hardware; and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising: obtaining a source training image in a source domain; selecting, from the source training image, a first source patch and a second source patch each representative of respective portions of the source training image; translating, using a forward generator neural network, the source training image to a translated target image in a target domain according to parameters of the forward generator neural network, the translated target image comprising a third source patch and a fourth source patch each representative of respective portions of the translated target image, the third source patch and the fourth source patch of the translated target image corresponding to the first source patch and the second source patch of the source training image; determining a first energy distance between the first source patch and the second source patch of the source training image; determining a second energy distance between the third source patch and the fourth source patch of the translated target image; generating a harmonic loss based on the first energy distance and the second energy distance; and updating, using the harmonic loss, the parameters of the forward generator neural network. ### Claim 12 (depends on 11) The system of claim 11, wherein the operations further comprise determining, by a target discriminator neural network, a probability that the translated target image belongs to the target domain. ### Claim 13 (depends on 11) The system of claim 11, wherein the operations further comprise: obtaining a target training image in the target domain; selecting, from the target training image, a first target patch and a second target patch each representative of respective portions of the target training image; translating, using a backward generator neural network, the target training image to a translated source image in the source domain according to parameters of the backward generator neural network, the translated source image comprising a third target patch and a fourth target patch each representative of respective portions of the translated source image, the third target patch and the fourth target patch of the translated source image corresponding to the first target patch and the second target patch of the target training image; determining a third energy distance between the first target patch and the second target patch of the target training image; determining a fourth energy distance between the third target patch and the fourth target patch of the translated source image; generating a second harmonic loss based on the third energy distance and the fourth energy distance; and updating, using the second harmonic loss, the parameters of the backward generator neural network. ### Claim 14 (depends on 13) The system of claim 13, wherein the operations further comprise determining, by a source discriminator neural network, a probability that the translated source image belongs to the source domain. ### Claim 15 (depends on 13) The system of claim 13, wherein updating the parameters of the forward generator neural network comprises optimizing an objective function. ### Claim 16 (depends on 15) The system of claim 15, wherein the objective function comprises an adversarial loss component that ensures translation accuracy of the forward generator neural network and the backward generator neural network. ### Claim 17 (depends on 16) The system of claim 16, wherein the objective function further comprises a circularity loss component that ensures that the forward generator neural network and the backward generator neural network comprise inverse translation operations. ### Claim 18 (depends on 17) The system of claim 17, wherein the objective function comprises a weighted combination adversarial loss terms and the circularity loss component. ### Claim 19 (depends on 11) The system of claim 11, wherein selecting the first source patch and the second source patch of the source training image comprises determining whether the first energy distance between the first source patch and the second source patch of the source training image satisfies a threshold value. ### Claim 20 (depends on 11) The system of claim 11, wherein the operations further comprise jointly updating the parameters of the forward generator neural network and parameters of a backward generator neural network.