# US 12,223,623 B2 · Harmonizing composite images utilizing a semantic-guided transformer neural network > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,623 B2 - Document ID: US12223623B2 - Status: Granted - Kind: B2 - Application number: 18/053,027 - Filed: 2022-11-07 - Published: 2024-05-16 - Granted: 2025-02-11 - Earliest claimed: 2022-11-07 - Assignees: Adobe Inc. - Applicants: Adobe Inc. - Inventors: He Zhang; Hyun Joon Jung - Web page: https://patentgrep.com/patent/US12223623B2 ## CPC - G06T 11/60: Creating or editing images; Combining images with text - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 7/11: Region-based segmentation - G06T 7/194: involving foreground-background segmentation - G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion - G06V 10/267: {by performing operations on regions, e.g. growing, shrinking or watersheds} - G06V 10/42: Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation - G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components - G06V 10/82: using neural networks - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - G06T 2200/24: involving graphical user interfaces [GUIs] - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20092: Interactive image processing based on input by user - G06T 2207/20132: Image cropping - G06T 2207/20212: Image combination - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The present disclosure relates to systems, non-transitory computer-readable media, and methods that implement a multi-branch harmonization neural network architecture to harmonize composite images. For example, in one or more implementations, the semantic-guided transformer-based harmonization system uses a convolutional branch, a transformer branch, and a semantic branch to generate a harmonized composite image based on an input composite image and a corresponding segmentation mask. More particularly, the convolutional branch comprises a series of convolutional neural network layers followed by a style normalization layer to extract localized information from the input composite image. Further, the transformer branch comprises a series of transformer neural network layers to extract global information based on different resolutions of the input composite image. The semantic branch includes a visual neural network that generates semantic features that inform the harmonization of the composite images. ## Claims ### Claim 1 A computer-implemented method comprising: receiving a composite image and a segmentation mask of a foreground object portrayed against a background of the composite image; utilizing a global neural network branch to extract global information of the background; utilizing a semantic neural network branch to extract semantic information from the composite image; and generating a harmonized composite image comprising the foreground object harmonized with the background by decoding the global information and the semantic information. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein utilizing the global neural network branch to extract the global information of the background comprises extracting the global information utilizing a transformer neural network. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein utilizing the semantic neural network branch to extract semantic information from the composite image comprises extracting semantic features utilizing a visual neural network. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, further comprising utilizing a local neural network branch to extract local information from the composite image. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein generating the harmonized composite image comprises decoding the local information, the global information, and the semantic information. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein utilizing the local neural network branch to extract local information comprises extracting local information of a background portion adjacent to the foreground object utilizing a convolutional neural network. ### Claim 7 A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising: receiving a composite image and a segmentation mask of a foreground object portrayed against a background of the composite image; utilizing a local neural network branch to extract local information from the composite image; utilizing a global neural network branch to extract global information of the background; utilizing a semantic neural network branch to extract semantic information from the composite image; and generating a harmonized composite image comprising the foreground object harmonized with the background by decoding the local information, global information, and the semantic information. ### Claim 8 (depends on 7) The non-transitory computer-readable medium of claim 7, wherein the operations further comprise: generating a semantic aware image crop from the composite image; generating a semantic aware segmentation mask crop from the segmentation mask; and concatenating the semantic aware image crop and the semantic aware segmentation mask crop to generate a cropped four-channel semantic-aware input. ### Claim 9 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein utilizing the semantic neural network branch comprises extracting semantic features from the cropped four-channel semantic-aware input utilizing a visual neural network. ### Claim 10 (depends on 9) The non-transitory computer-readable medium of claim 9, wherein extracting semantic features from the cropped four-channel semantic-aware input utilizing the visual neural network comprises extracting the semantic features from a convolutional layer just prior to average pooling and a classification layer of the visual neural network. ### Claim 11 (depends on 9) The non-transitory computer-readable medium of claim 9, wherein utilizing the global neural network branch comprises utilizing one or more transformer neural network layers to extract the global information of the background from the cropped four-channel semantic-aware input. ### Claim 12 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein utilizing the one or more transformer neural network layers to extract the global information of the background comprises utilizing one or more layers comprising a self-attention neural network layer, a mix feed-forward neural network layer, and an overlap patch merging layer. ### Claim 13 (depends on 9) The non-transitory computer-readable medium of claim 9, wherein utilizing the local neural network branch comprises utilizing one or more convolutional neural network layers to extract the local information of a background portion adjacent to the foreground object from the cropped four-channel semantic-aware input. ### Claim 14 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein utilizing the local neural network branch comprises utilizing one or more style normalization layers to extract style information from the background portion adjacent to the foreground object by determining a mean and standard deviation of pixel color values between pixels corresponding to the background portion adjacent to the foreground object. ### Claim 15 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein the operations further comprise utilizing the one or more style normalization layers to apply the style information to the foreground object. ### Claim 16 (depends on 7) The non-transitory computer-readable medium of claim 7, wherein receiving the composite image and the segmentation mask comprises: identifying, via a graphical user interface, a user selection of a background image for combining with the foreground object as the composite image; identifying, via the graphical user interface, an additional user selection corresponding to a harmonization filter for harmonizing the foreground object and the background image; and utilizing a segmentation neural network to generate the segmentation mask of the foreground object portrayed against the background of the composite image. ### Claim 17 A system comprising: a composite image comprising a foreground object portrayed against a background of the composite image; a segmentation mask of the foreground object; a local neural network branch to extract local information of a background portion adjacent to the foreground object; a global neural network branch to extract global information of the background; a semantic neural network branch to extract semantic information from the composite image; a decoder to decode the local information, the global information, and the semantic information; and one or more processors configured to cause the system to generate a harmonized composite image comprising the foreground object with modified pixel color values based on decoding of the local information, the global information, and the semantic information. ### Claim 18 (depends on 17) The system of claim 17, wherein: the local neural network branch comprises one or more convolutional neural network layers; the global neural network branch comprises a transformer neural network; and the semantic neural network branch comprises a visual neural network. ### Claim 19 (depends on 18) The system of claim 18, wherein the global neural network branch comprises a series of transformer neural networks composed of at least one of a self-attention neural network layer, a mix feed-forward neural network layer, or an overlap patch merging layer. ### Claim 20 (depends on 18) The system of claim 18, wherein the local neural network branch comprises a style normalization layer to generate style-normalized foreground feature vectors for the foreground object based on style information for the background portion adjacent to the foreground object.