# US 12,223,706 B2 · Training energy-based models from a single image for internal learning and inference using trained models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,706 B2 - Document ID: US12223706B2 - Status: Granted - Kind: B2 - Application number: 17/824,694 - Filed: 2022-05-25 - Published: 2022-12-15 - Granted: 2025-02-11 - Earliest claimed: 2021-06-09 - Assignees: Baidu USA LLC - Applicants: Baidu USA, LLC - Inventors: Zilong Zheng; Jianwen Xie; Ping Li - Web page: https://patentgrep.com/patent/US12223706B2 ## CPC - G06V 10/52: Scale-space analysis, e.g. wavelet analysis (multi-scale boundary representations G06V10/42) - G06V 10/85: {Markov-related models; Markov random fields} - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/778: Active pattern-learning, e.g. online learning of image or video features - G06V 10/82: using neural networks - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Different from prior works that model the internal distribution of patches within an image implicitly with a top-down latent variable model (e.g., generator), embodiments explicitly represent the statistical distribution within a single image by using an energy-based generative framework, where a pyramid of energy functions, each parameterized by a bottom-up deep neural network, are used to capture the distributions of patches at different resolutions. Also, embodiments of a coarse-to-fine sequential training and sampling strategy are presented to train the model efficiently. Besides learning to generate random samples from white noise, embodiments can learn in parallel with a self-supervised task (e.g., recover an input image from its corrupted version), which can further improve the descriptive power of the learned model. Embodiments does not require an auxiliary model (e.g., discriminator) to assist the training, and embodiments also unify internal statistics learning and image generation in a single framework. ## Claims ### Claim 1 A computer-implemented method comprising: given a set of energy-based neural network models (EBMs) in which each EBM is configured to synthesize an image at a scale and the scales of the set of EBMs range from a minimal scale to a maximum scale, and given a multi-scale set of versions of a training image that similarly range from the minimal scale to the maximum scale so that each EBM has a corresponding version of the training image at the same scale as the EBM, performing steps comprising: training each of the EBMs in the set of EBMs to synthesize an image at the scale of the EBM based on a patch distribution learned from the version of the training image at the scale of the EBM given an initial input image at the scale of the EBM and the version of the training image at the scale of the EBM, in which for at least some of the EBMs the initial input image is a synthesized image from a lower scaled EBM that has been upsampled to the scale of the EBM, comprising: for each EBM: using the initial input image for the EBM and the EBM in a Markov chain Monte Carlo (MCMC) sampling process to produce the synthesized image at the scale of the EBM; and updating parameters of the EBM by using a comparison comprising one or more the synthesized image generated using the MCMC sampling process and the version of the training image at the scale of the EBM; and responsive to a training stop condition being met, output the trained set of EBMs. ### Claim 2 (depends on 1) The computer-implemented method of claim 1 wherein the MCMC process is an iterative process with a finite number of Langevin dynamics steps. ### Claim 3 (depends on 2) The computer-implemented method of claim 2 wherein the finite number of Langevin dynamics steps for the MCMC sample process is set for each EBM and is a same or a different finite number than one or more for other EBMs. ### Claim 4 (depends on 2) The computer-implemented method of claim 2 wherein the synthesized image from a lower scaled EBM that has been upsampled to the scale of the EBM is the synthesized image generated at a last step of the finite number of Langevin dynamics steps. ### Claim 5 (depends on 1) The computer-implemented method of claim 1 wherein, for the EBM with the minimal scale, the initial input image is an image generated by randomly sampling from a uniform distribution. ### Claim 6 (depends on 1) The computer-implemented method of claim 1 wherein, the comparison comprising one or more the synthesized image generated by the EBM using the MCMC sampling process and the version of the training image at the scale of the EBM comprises: determining a loss gradient that at least approximates difference between internal statistics of the training image as determined the EBM versus an average of internal statistics of one or more synthesized images as determined by the EBM. ### Claim 7 (depends on 1) The computer-implemented method of claim 1 further comprising, given an input condition image: additionally training the set of EBMs for a conditional task using the input condition image as an input to the EBM with the minimum scale, in which training loss for the set of EBMs represents a combination of unconditional generation loss and conditional generation loss. ### Claim 8 (depends on 7) The computer-implemented method of claim 7 wherein: the input condition image is a low-resolution image and the conditional task is super-resolution; the input condition image is a noisy image and the conditional task is denoising; and the input condition image is an image from another domain and the conditional task is cross-domain image translation. ### Claim 9 (depends on 1) A computer-implemented method comprising: obtaining the trained set of (EBMs) trained according to the method of claim 1; starting with a trained EBM with the minimum scale and moving in order of increasing scale until the trained EBM with the maximum scale has been processed, performing steps comprising: responsive to the trained EBM being the trained EBM with the minimum scale, using an initial input image having the minimum scale and the trained EBM in a Markov chain Monte Carlo (MCMC) sampling process to produce a final synthesized image at the scale of the trained EBM; and responsive to the trained EBM not being the trained EBM with the minimum scale, using an initial image having the scale of the trained EBM and the trained EBM in a Markov chain Monte Carlo (MCMC) sampling process to produce a final synthesized image at the scale of the trained EBM, in which the initial image is obtained from the final synthesized image from the trained EBM with a lower scale; and outputting at least one of the final synthesized images. ### Claim 10 (depends on 9) The computer-implemented method of claim 9 the MCMC process is an iterative process with a finite number of Langevin dynamics steps. ### Claim 11 (depends on 9) The computer-implemented method of claim 9 wherein, for the trained EBM with the minimal scale, the initial input image is one of: an image generated by randomly sampling from a uniform distribution; a low-resolution image; a noisy image; or an image from domain different from that used to train the trained set of EBMs. ### Claim 12 A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by at least one processor, causes steps to be performed comprising: given a set of energy-based neural network models (EBMs) in which each EBM is configured to synthesize an image at a scale and the scales of the set of EBMs range from a minimal scale to a maximum scale, and given a multi-scale set of versions of a training image that similarly range from the minimal scale to the maximum scale so that each EBM has a corresponding version of the training image at the same scale as the EBM, performing steps comprising: training each of the EBMs in the set of EBMs to synthesize an image at the scale of the EBM based on a patch distribution learned from the version of the training image at the scale of the EBM given an initial input image at the scale of the EBM and the version of the training image at the scale of the EBM, in which for at least some of the EBMs the initial input image is a synthesized image from a lower scaled EBM that has been upsampled to the scale of the EBM, comprising: for each EBM: using the initial input image for the EBM and the EBM in a Markov chain Monte Carlo (MCMC) sampling process to produce the synthesized image at the scale of the EBM; and updating parameters of the EBM by using a comparison comprising one or more the synthesized image generated using the MCMC sampling process and the version of the training image at the scale of the EBM; and responsive to a training stop condition being met, output the trained set of EBMs. ### Claim 13 (depends on 12) The non-transitory computer-readable medium or media of claim 12 wherein the MCMC process is an iterative process with a finite number of Langevin dynamics steps and wherein the finite number of Langevin dynamics steps for the MCMC sample process is a same or a different finite number than for one or more other EBMs. ### Claim 14 (depends on 13) The non-transitory computer-readable medium or media of claim 13 wherein the synthesized image from a lower scaled EBM that has been upsampled to the scale of the EBM is the synthesized image generated at a last step of the finite number of Langevin dynamics steps. ### Claim 15 (depends on 12) The non-transitory computer-readable medium or media of claim 12 wherein, for the EBM with the minimal scale, the initial input image is an image generated by randomly sampling from a uniform distribution. ### Claim 16 (depends on 13) The non-transitory computer-readable medium or media of claim 13 wherein, the comparison comprising one or more the synthesized image generated by the EBM using the MCMC sampling process and the version of the training image at the scale of the EBM comprises: determining a loss gradient that at least approximates difference between internal statistics of the training image as determined the EBM versus an average of internal statistics of one or more synthesized images as determined by the EBM. ### Claim 17 (depends on 12) The non-transitory computer-readable medium or media of claim 12 further comprising one or more sequences of instructions which, when executed by at least one processor, causes steps to be performed comprising, given an input condition image: additionally training the set of EBMs for a conditional task using the input condition image as an input to the EBM with the minimum scale, in which training loss for the set of EBMs represents a combination of unconditional generation loss and conditional generation loss. ### Claim 18 (depends on 12) The non-transitory computer-readable medium or media of claim 12 wherein each of the EBMs is a patchwise generative convolution neural network.