# US 11,308,628 B2 · Patch-based image matting using deep learning > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,628 B2 - Document ID: US11308628B2 - Status: Granted - Kind: B2 - Application number: 16/847,819 - Filed: 2020-04-14 - Published: 2021-10-14 - Granted: 2022-04-19 - Earliest claimed: 2020-04-14 - Assignees: Adobe Inc. - Applicants: Adobe Inc. - Inventors: Ning Xu - Web page: https://patentgrep.com/patent/US11308628B2 ## CPC - G06T 7/11: Region-based segmentation - G06T 7/194: involving foreground-background segmentation - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 5/60: using machine learning, e.g. neural networks - G06N 3/045: Combinations of networks - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20221: Image fusion; Image merging - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Methods and systems are provided for generating mattes for input images. A neural network system is trained to generate a matte for an input image utilizing contextual information within the image. Patches from the image and a corresponding trimap are extracted, and alpha values for each individual image patch are predicted based on correlations of features in different regions within the image patch. Predicting alpha values for an image patch may also be based on contextual information from other patches extracted from the same image. This contextual information may be determined by determining correlations between features in the query patch and context patches. The predicted alpha values for an image patch form a matte patch, and all matte patches generated for the patches are stitched together to form an overall matte for the input image. ## Claims ### Claim 1 A computer-implemented method for generating mattes for images, the method comprising: receiving, at a receiving module, an image and a trimap corresponding to the image; creating, at a patch module, a plurality of image patches each corresponding to part of the image and a plurality of trimap patches each corresponding to part of the trimap; generating, at a matting module, a plurality of matte patches utilizing a neural network, each matte patch indicating alpha values for each pixel within an image patch based on correlations between features in different regions of the image patch, the different regions including an unknown region, a foreground region, and a background region determined from a trimap patch corresponding to the image patch; and generating, at a stitching module, a matte for the image by stitching together the matte patches generated for the plurality of image patches. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein correlations between features in different regions of the image patch include a correlation between features in an unknown region and features in the foreground region, a correlation between features in the unknown region and features in the background region, and a correlation between features in the unknown region and a different unknown region within the image patch. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein each image patch is partially overlapping with at least one adjacent image patch within the image and wherein stitching together the matte patches comprises determining an alpha value for each pixel within an overlapping region based on a weighted average of alpha values for pixels within the overlapping region. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the image is a high resolution image. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the method further includes, for each image patch, at the patch module, identifying a subset of additional image patches from the plurality of images patches to provide contextual information in determining alpha values for the image patch. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein the subset of additional image patches comprises three image patches. ### Claim 7 (depends on 5) The computer-implemented method of claim 5, wherein each image patch within the subset of additional image patches corresponds to a trimap patch that includes a background region and a foreground region. ### Claim 8 A computerized system for generating mattes for input images, the system comprising: receiving, at a receiving module, an image and a trimap corresponding to the image; creating, at a patch module, a plurality of image patches each corresponding to part of the image and a plurality of trimap patches each corresponding to part of the trimap; for each image patch, selecting, at the patch module, a subset of additional image patches from the plurality of image patches to use as context patches; generating, at a matting module, a plurality of matte patches utilizing a neural network, each matte patch indicating alpha values for pixels within an image patch determined using the subset of additional image patches; and generating, at a stitching module, a matte for the image by stitching together the plurality of matte patches generated for the plurality of image patches. ### Claim 9 (depends on 8) The computerized system of claim 8, wherein the subset of additional image patches comprises three image patches. ### Claim 10 (depends on 8) The computerized system of claim 8, wherein selecting the subset of additional image patches comprises randomly selecting the subset of additional image patches to the extent that each additional image patch corresponds to a trimap patch that includes a background region and a foreground region. ### Claim 11 (depends on 8) The computerized system of claim 8, wherein alpha values for pixels within the image patch are determined at the matting module by: determining correlations between features in different regions of the image patch, the different regions including an unknown region, a foreground region, and a background region; and determining correlations between features in the image patch and the context patches corresponding to the image patch. ### Claim 12 (depends on 8) The computerized system of claim 8, wherein each image patch is partially overlapping with at least one adjacent image patch within the image. ### Claim 13 (depends on 12) The computerized system of claim 12, wherein stitching together the plurality of matte patches comprises determining an alpha value for each pixel within an overlapping region based on a weighted average of alpha values for pixels within the overlapping region. ### Claim 14 (depends on 8) The computerized system of claim 8, wherein the instructions further cause the processor to extract, at an extracting module, an object from the image using the matte. ### Claim 15 One or more non-transitory computer storage media having instructions embodied thereon that, when executed by one or more processors, perform operations comprising: receiving an image and a trimap corresponding to the image; creating a plurality of image patches each corresponding to part of the image and a plurality of trimap patches each corresponding to part of the trimap; generating a plurality of matte patches utilizing a neural network, each matte patch indicating alpha values for each pixel within an image patch based on correlations between features in different regions of the image patch, the different regions including an unknown region, a foreground region, and a background region determined from a trimap patch corresponding to the image patch; and generating a matte for the image by stitching together the matte patches generated for the plurality of image patches. ### Claim 16 (depends on 15) The one or more non-transitory computer storage media of claim 15, wherein correlations between features in different regions of the image patch include a correlation between features in an unknown region and features in the foreground region, a correlation between features in the unknown region and features in the background region, and a correlation between features in the unknown region and a different unknown region within the image patch. ### Claim 17 (depends on 15) The one or more non-transitory computer storage media of claim 15, wherein each image patch is partially overlapping with at least one adjacent image patch within the image and wherein stitching together the matte patches comprises determining an alpha value for each pixel within an overlapping region based on a weighted average of alpha values for pixels within the overlapping region. ### Claim 18 (depends on 15) The one or more non-transitory computer storage media of claim 15, wherein the image is a high resolution image. ### Claim 19 (depends on 15) The one or more non-transitory computer storage media of claim 15, wherein the operations further includes, for each image patch, at the patch module, identifying a subset of additional image patches from the plurality of images patches to provide contextual information in determining alpha values for the image patch. ### Claim 20 (depends on 19) The one or more non-transitory computer storage media of claim 19, wherein each image patch within the subset of additional image patches corresponds to a trimap patch that includes a background region and a foreground region.