# US 11,887,269 B2 · Computing method and apparatus with image generation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,887,269 B2 - Document ID: US11887269B2 - Status: Granted - Kind: B2 - Application number: 17/202,899 - Filed: 2021-03-16 - Published: 2022-05-12 - Granted: 2024-01-30 - Earliest claimed: 2020-11-11 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Minjung Son; Hyun Sung Chang - Web page: https://patentgrep.com/patent/US11887269B2 ## CPC - G06T 3/04: Context-preserving transformations, e.g. by using an importance map (panospheric to cylindrical image transformations G06T3/12) - G06T 11/00: Two-dimensional [2D] image generation - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0475: Generative networks - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G06T 7/0002: {Inspection of images, e.g. flaw detection} - G06T 7/50: Depth or shape recovery - G06T 7/60: Analysis of geometric attributes - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A method and apparatus for generating an image and for training an artificial neural network to generate an image are provided. The method of generating an image, including receiving input data comprising conditional information and image information, generating a synthesized image by applying the input data to an image generation neural network configured to maintain geometric information of the image information and to transform the remaining image information based on the conditional information, and outputting the synthesized image. ## Claims ### Claim 1 A processor-implemented method, the method comprising: receiving input data comprising conditional information and image information, wherein the image information includes geometric information; receiving depth information corresponding to the conditional information; extracting feature information from the depth information; generating a synthesized image based on the conditional information by applying the input data and the feature information to an image generation neural network, the image generation neural network is configured to: maintain the geometric information of the image information; and transform, based on the conditional information, first remaining image information of the image information to second remaining image information; and outputting the synthesized image. ### Claim 2 (depends on 1) The method of claim 1, wherein the image generation neural network is trained by a first loss generated based on a comparison between a first synthesized image that corresponds to the conditional information and is transformed based on a transformation relationship between the conditional information and a second synthesized image, the second synthesized image corresponds to the conditional information and is transformed based on depth information corresponding to the conditional information. ### Claim 3 (depends on 1) The method of claim 1, wherein the conditional information comprises structure information of any one or any combination of semantic information, edge information, and skeleton information. ### Claim 4 (depends on 1) The method of claim 1, further comprising: receiving depth information corresponding to the conditional information; generating a first synthesized image corresponding to the conditional information by the image generation neural network based on the image information; transforming the conditional information based on the depth information; generating a second synthesized image corresponding to the transformed conditional information by the image generation neural network based on the image information; and training the image generation neural network by comparing the first synthesized image and the second synthesized image based on a transformation relationship of the conditional information. ### Claim 5 A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to execute a method of generating an image, the method comprising: receiving input data comprising conditional information and image information, wherein the image information includes geometric information; receiving depth information corresponding to the conditional information; extracting feature information from the depth information; generating a synthesized image based on the conditional information by applying the input data and the feature information to an image generation neural network, the image generation neural network is configured to: maintain the geometric information of the image information; and transform, based on the conditional information, first remaining image information of the image information to second remaining image information; and outputting the synthesized image. ### Claim 6 A computing apparatus, the apparatus comprising: a communication circuit configured to: receive input data including conditional information and image information, wherein the image information includes geometric information; and receive depth information corresponding to the conditional information; a processor configured to: extract feature information from the depth information; and generate a synthesized image based on the conditional information by applying the input data and the feature information to an image generation neural network, wherein the image generation neural network is configured to: maintain the geometric information of the image information; and transform, based on the conditional information, first remaining image information of the image information to second remaining image information; and an output circuit configured to output the synthesized image. ### Claim 7 A processor-implemented method, the method comprising: receiving training data comprising conditional information and image information; receiving depth information corresponding to the conditional information; generating a first synthesized image corresponding to the conditional information by an image generation neural network based on the image information; transforming the conditional information based on the depth information; generating a second synthesized image corresponding to the transformed conditional information by the image generation neural network based on the image information; and training the image generation neural network by comparing the first synthesized image and the second synthesized image based on a transformation relationship of the conditional information. ### Claim 8 (depends on 7) The method of claim 7, wherein the training of the image generation neural network comprises: transforming the first synthesized image based on the transformation relationship of the conditional information; and generating a first loss based on a first difference between the transformed first synthesized image and the second synthesized image. ### Claim 9 (depends on 7) The method of claim 7, wherein the generating of the first loss comprises generating the first loss to maintain a geometry consistency between the transformed first synthesized image and the second synthesized image. ### Claim 10 (depends on 7) The method of claim 7, wherein the training of the image generation neural network comprises: inversely transforming the second synthesized image based on the transformation relationship of the conditional information; and generating a second loss based on a second difference between the inversely transformed second synthesized image and the first synthesized image. ### Claim 11 (depends on 10) The method of claim 10, wherein the generating of the second loss comprises generating the second loss to maintain a geometry consistency between the inversely transformed second synthesized image and the first synthesized image. ### Claim 12 (depends on 7) The method of claim 7, wherein the transforming of the conditional information comprises: unprojecting the training data to a three-dimensional (3D) space by a transformation relationship based on the depth information; and transforming the conditional information by projecting the unprojected training data to a viewpoint. ### Claim 13 (depends on 7) The method of claim 7, wherein the conditional information comprises structure information of any one or any combination of semantic information, edge information, and skeleton information. ### Claim 14 (depends on 7) The method of claim 7, further comprising: extracting a first geometric feature from the conditional information; and extracting a second geometric feature from the first synthesized image, wherein the training of the image generation neural network further comprises generating a third loss for training the image generation neural network, based on a difference between the first geometric feature and the second geometric feature. ### Claim 15 (depends on 7) The method of claim 7, further comprising: extracting a third geometric feature from the transformed conditional information; and extracting a fourth geometric feature from the second synthesized image, wherein the training of the image generation neural network further comprises generating a fourth loss for training the image generation neural network, based on a difference between the third geometric feature and the fourth geometric feature. ### Claim 16 (depends on 15) The method of claim 15, further comprising: transforming a second geometric feature extracted from the first synthesized image based on the depth information, wherein the training of the image generation neural network comprises generating a fifth loss for training the image generation neural network, based on a difference between the transformed second geometric feature and a fourth geometric feature extracted from the second synthesized image. ### Claim 17 (depends on 7) The method of claim 7, further comprising: extracting depth feature information from the depth information, wherein the generating of the first synthesized image further comprises generating the first synthesized image corresponding to the depth feature information and the conditional information by the image generation neural network based on the image information. ### Claim 18 (depends on 17) The method of claim 17, wherein the transforming of the conditional information comprises transforming the conditional information and the depth feature information based on the depth information. ### Claim 19 (depends on 18) The method of claim 18, wherein the generating of the second synthesized image further comprises generating the second synthesized image corresponding to the transformed conditional information and the transformed depth feature information by the image generation neural network based on the image information. ### Claim 20 (depends on 7) The method of claim 7, further comprising any one or any combination of: training an image discrimination neural network to estimate the first synthesized image as a fake image; training the image discrimination neural network to estimate the second synthesized image as the fake image; and training the image discrimination neural network to estimate the image information as a real image. ### Claim 21 (depends on 7) The method of claim 7, further comprising: generating a synthesized image by applying the input data to the trained image generation neural network; and outputting the synthesized image. ### Claim 22 A processor-implemented method, the method comprising: receiving training data comprising image information, conditional information, and depth information corresponding to the conditional information; extracting depth feature information from the depth information; generating a first synthesized image corresponding to the depth feature information and the conditional information by the image generation neural network based on the image information; transforming the conditional information and the depth feature information based on the depth information; generating a second synthesized image corresponding to the transformed conditional information and the transformed depth feature information by the image generation neural network based on the image information; and training the image generation neural network by comparing the first synthesized image and the second synthesized image based on a transformation relationship of the conditional information and the depth information. ### Claim 23 (depends on 22) The method of claim 22, wherein the training of the image generation neural network comprises: transforming the first synthesized image based on the transformation relationship; and training the image generation neural network to minimize a difference between the transformed first synthesized image and the second synthesized image. ### Claim 24 (depends on 22) The method of claim 22, wherein the depth feature information is based on extracting a discontinuity portion of the depth information and a discontinuity portion of the normal information.