# US 2022/0044352 A1 · Cross-domain image translation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0044352 A1 - Document ID: US20220044352A1 - Status: Published application - Kind: A1 - Application number: 17/278,652 - Filed: 2019-09-05 - Published: 2022-02-10 - Earliest claimed: 2018-10-31 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Jing LIAO; Lu YUAN; Kaidi Cao - Web page: https://patentgrep.com/patent/US20220044352A1 ## CPC - G06T 3/18: Image warping, e.g. rearranging pixels individually - G06T 11/00: Two-dimensional [2D] image generation - G06F 18/2135: based on approximation criteria, e.g. principal component analysis - G06N 20/20: Ensemble learning - G06T 3/04: Context-preserving transformations, e.g. by using an importance map (panospheric to cylindrical image transformations G06T3/12) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Implementations of the present disclosure provide a solution for cross-domain image translation. In this solution, a first learning network for geometric deformation from a first to a second image domain is determined based on first and second images in the first and second domains, images in the two domains having different styles and objects in the images having geometric deformation with respect to each other. Geometric deformation from the second to the first domains is performed on the second image or geometric deformation from the first to the second domains is performed on the first image, to generate an intermediate image. A second learning network for style transfer from the first to the second domains is determined based on the first and intermediate images or based on the second and intermediate images generated. Accordingly, processing accuracy of leaning networks for cross-domain image translation can be improved and complexity is lowered. ## Claims ### Claim 1 A computer-implemented method, comprising: determining a first learning network for geometric deformation from a first image domain to a second image domain based on a first image in the first image domain and a second image in the second image domain, images in the first and second image domains having different styles and objects in the images having geometric deformation with respect to each other; performing the geometric deformation from the second image domain to the first image domain on the second image to generate an intermediate image, the intermediate image inheriting a same style as the second image; and determining a second learning network for style transfer from the first image domain to the second image domain based on the first image and the intermediate image. ### Claim 2 (depends on 1) The method of claim 1, wherein determining the first learning network comprises: extracting first landmark points of a geometry of an object in the first image and second landmark points of a geometry of an object in the second image; determining a first principal component analysis (PCA) representation of the first landmark points and a second PCA representation of the second landmark points; and determining the first learning network based on the first and second PCA representations. ### Claim 3 (depends on 1) The method of claim 1, wherein the first learning network is based on a first generative adversarial network (GAN), and wherein determining the first learning network comprises: performing, by using a first sub-network of the first GAN, geometric deformation from the first image domain to the second image domain on the first image, to deform a first geometry of an object in the first image to a first deformed geometry; determining a first difference between the first geometry and a first average geometry and a second difference between the first deformed geometry and a second average geometry, the first average geometry being an average of geometries of objects in a plurality of images in the first image domain, and the second average geometry being an average of geometries of objects in a plurality of images in the second image domain; and updating the first sub-network of the first GAN such that a difference between the first and second differences is decreased, wherein the updated first sub-network is determined as the first learning network. ### Claim 4 (depends on 3) The method of claim 3, wherein updating the first sub-network of the first GAN further comprises: performing, by using a second sub-network of the first GAN, the geometric deformation from the second image domain to the first image domain on the second image, to deform a second geometry of an object in the second image to a second deformed geometry; determining a third difference between the second geometry and the second average geometry and a fourth difference between the second deformed geometry and the first average geometry; and jointly updating the first and second sub-networks of the first GAN such that a difference between the third and fourth differences is also decreased. ### Claim 5 (depends on 4) The method of claim 4, wherein performing the geometric deformation from the second image domain to the first image domain comprises: performing the geometric deformation from the second image domain to the first image domain by using the updated second sub-network. ### Claim 6 (depends on 1) The method of claim 1, wherein the second learning network is based on a second generative adversarial network (GAN), and wherein determining the second learning network comprises: performing, by using a first sub-network of the second GAN, style transfer from the first image domain to the second image domain on the first image, to transfer the first image with a first style to a first predicted image with a second style, the second style being randomly selected from a plurality of styles in the second image domain; and updating the first sub-network of the second GAN such that a difference in content between the first predicted image and the first image is decreased, wherein the updated first sub-network is determined as the second learning network. ### Claim 7 (depends on 6) The method of claim 6, wherein updating the first sub-network of the second GAN further comprises: performing, by using a second sub-network of the second GAN, style transfer from the second image domain to the first image domain on the intermediate image, to transfer the intermediate image with a third style to a second predicted image with a fourth style, the fourth style being randomly selected from a plurality of styles in the first image domain; and jointly updating the first and second sub-networks of the second GAN such that a difference in content between the second predicted image and the intermediate image is decreased. ### Claim 8 A computer-implemented method, comprising: performing geometric deformation on a source image to deform a first geometry of an object in the source image to a second geometry; performing style transfer on the source image to transfer the source image with a first style to an intermediate image with a second style; and transferring the object in the intermediate image based on the second geometry to generate a target image, the target image inheriting a same second style as the intermediate image. ### Claim 9 A computer-implemented method, comprising: performing geometric deformation on a source image to deform a first geometry of an object in the source image to a second geometry; transferring the object in the source image based on the second geometry to generate an intermediate image, the intermediate image inheriting a same first style as the source image; and performing style transfer on the intermediate image to transfer the intermediate image with the first style to a target image with a second style. ### Claim 10 (depends on 8) The method of claim 8, wherein performing geometric deformation comprises: determining landmark points in the source image that represent the first geometry; generating a principal component analysis (PCA) representation of the landmark points; performing the geometric deformation on the PCA representation; and determining deformed landmark points representing the second geometry based on the deformed PCA representation. ### Claim 11 (depends on 8) The method of claim 8, wherein performing the geometric deformation comprises: obtaining an indication of a deformation degree of the object; and transferring the first geometry to the second geometry based on the deformation degree. ### Claim 12 (depends on 8) The method of claim 8, wherein performing style transfer on the source image comprises: extracting a content-related feature representation of the source image; determining an appearance-related feature representation associated with the second style; and generating the intermediate image based on the content-related feature representation and the appearance-related feature representation. ### Claim 13 (depends on 12) The method of claim 12, wherein determining the appearance-related feature representation comprises: randomly selecting the appearance-related feature representation from a plurality of predetermined appearance-related feature representations; or determining the appearance-related feature representation based on an obtained indication related to the second style or a reference image with the second style. ### Claim 14 (depends on 8) The method of claim 8, wherein performing the geometric deformation comprises performing the geometric deformation by using a first learning network, the first learning network being determined based on a first generative adversarial network (GAN); or wherein performing the style transfer comprises performing the style transfer by using a second learning network, the second learning network being determined based on a second GAN network. ### Claim 15 (canceled) ### Claim 16 (depends on 9) The method of claim 9, wherein performing the geometric deformation comprises: determining landmark points in the source image that represent the first geometry; generating a principal component analysis (PCA) representation of the landmark points; performing the geometric deformation on the PCA representation; and determining deformed landmark points representing the second geometry based on the deformed PCA representation. ### Claim 17 (depends on 9) The method of claim 9, wherein performing the geometric deformation comprises: obtaining an indication of a deformation degree of the object; and transferring the first geometry to the second geometry based on the deformation degree. ### Claim 18 (depends on 9) The method of claim 9, wherein performing style transfer on the source image comprises: extracting a content-related feature representation of the source image; determining an appearance-related feature representation associated with the second style; and generating the intermediate image based on the content-related feature representation and the appearance-related feature representation. ### Claim 19 (depends on 18) The method of claim 18, wherein determining the appearance-related feature representation comprises: randomly selecting the appearance-related feature representation from a plurality of predetermined appearance-related feature representations; or determining the appearance-related feature representation based on an obtained indication related to the second style or a reference image with the second style. ### Claim 20 (depends on 9) The method of claim 9, wherein performing the geometric deformation comprises performing the geometric deformation by using a first learning network, the first learning network being determined based on a first generative adversarial network (GAN); or wherein performing the style transfer comprises performing the style transfer by using a second learning network, the second learning network being determined based on a second GAN network.