# US 12,602,790 B2 · Machine-learning models for distractor segmentation with reduced user interactions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,790 B2 - Document ID: US12602790B2 - Status: Granted - Kind: B2 - Application number: 18/307,353 - Filed: 2023-04-26 - Published: 2024-10-31 - Granted: 2026-04-14 - Earliest claimed: 2023-04-26 - Assignees: Adobe Inc. - Applicants: Adobe Inc. - Inventors: Yuqian Zhou; Chuong Huynh; Connelly Barnes; Elya Shechtman; Sohrab Amirghodsi; Zhe Lin - Web page: https://patentgrep.com/patent/US12602790B2 ## CPC - G06T 7/12: Edge-based segmentation - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06T 7/11: Region-based segmentation - G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components - G06V 10/761: {Proximity, similarity or dissimilarity measures} - G06V 10/806: {of extracted features} - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20101: Interactive definition of point of interest, landmark or seed - G06V 2201/07: Target detection - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The present disclosure relates to systems, non-transitory computer-readable media, and methods for utilizing machine learning to generate a mask for an object portrayed in a digital image. For example, in some embodiments, the disclosed systems utilize a neural network to generate an image feature representation from the digital image. The disclosed systems can receive a selection input identifying one or more pixels corresponding to the object. In addition, in some implementations, the disclosed systems generate a modified feature representation by integrating the selection input into the image feature representation. Moreover, in one or more embodiments, the disclosed systems utilize an additional neural network to generate a plurality of masking proposals for the object from the modified feature representation. Furthermore, in some embodiments, the disclosed systems utilize a further neural network to generate the mask for the object from the modified feature representation and/or the masking proposals. ## Claims ### Claim 1 A computer-implemented method comprising: generating, utilizing a feature extraction model, an image feature representation by encoding feature information from a digital image at a plurality of resolutions to generate a feature vector representation; receiving, via a graphical user interface of a client device, a selection input identifying a position of one or more pixels corresponding to an object portrayed in the digital image; generating, utilizing a click integration model, a modified feature representation by integrating an encoded spatial location for the selection input into the image feature representation; generating, utilizing an object detection model, a plurality of masking proposals from the digital image based on the modified feature representation; and generating, utilizing a machine-learning mask model and based on the plurality of masking proposals, a mask for the object portrayed in the digital image from the modified feature representation and the digital image. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein integrating the selection input into the image feature representation comprises: generating an interaction map from the selection input; and combining the interaction map and the image feature representation to generate the modified feature representation. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein receiving the selection input comprises receiving, via a user interface of a client device, a scribble input comprising a locus of pixels, and wherein integrating the selection input into the image feature representation comprises generating the interaction map from the locus of pixels. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, further comprising: receiving an additional selection input identifying an additional one or more pixels corresponding to the object portrayed in the digital image; and generating an additional modified feature representation by integrating the selection input and the additional selection input into the image feature representation. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, further comprising: generating, utilizing the machine-learning mask model, a refined mask for the object portrayed in the digital image from the additional modified feature representation. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein generating the image feature representation comprises: generating, utilizing the feature extraction model, a first image feature representation at a first resolution; and generating a second image feature representation at a second resolution different from the first resolution. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein generating the modified feature representation comprises: generating a first interaction map at the first resolution from the selection input; generating a second interaction map at the second resolution from the selection input; and combining the first interaction map with the first image feature representation and the second interaction map with the second image feature representation. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein generating the mask comprises: generating, utilizing an object detection model, a plurality of masking proposals from the digital image based on the modified feature representation; identifying a subset of masking proposals by filtering the plurality of masking proposals based on the selection input; and generating the mask based on the subset of masking proposals. ### Claim 9 (depends on 8) The computer-implemented method of claim 8, wherein generating the mask based on the subset of masking proposals comprises: generating, utilizing the machine-learning mask model, a plurality of masks and a plurality of confidence scores for the object portrayed in the digital image from the modified feature representation and the subset of masking proposals; and selecting the mask from the plurality of masks based on the plurality of confidence scores. ### Claim 10 A system comprising: one or more memory devices; and one or more processors coupled to the one or more memory devices that cause the system to perform operations comprising: generating, utilizing a feature extraction model, an image feature representation by encoding feature information from a digital image at a plurality of resolutions to generate a feature representation; generating, utilizing a click integration model, a modified feature representation by integrating an encoded spatial location for a selection input, received via a user interface of a client device, corresponding to an object portrayed in the digital image into the image feature representation; generating, utilizing an object detection model, a plurality of masking proposals for the object portrayed in the digital image from the modified feature representation; and generating, utilizing a machine-learning mask model, a mask for the object portrayed in the digital image from the modified feature representation, the digital image, and the plurality of masking proposals. ### Claim 11 (depends on 10) The system of claim 10, wherein integrating the selection input into the image feature representation comprises: generating an interaction map from the selection input, wherein the interaction map comprises an encoded pixel location corresponding to the selection input; and combining the interaction map and the image feature representation to generate the modified feature representation. ### Claim 12 (depends on 10) The system of claim 10, wherein the operations further comprise: generating an additional modified feature representation by integrating the selection input and an additional selection input corresponding to the object portrayed in the digital image into the image feature representation; and generating, utilizing the machine-learning mask model, a refined mask for the object portrayed in the digital image from the additional modified feature representation. ### Claim 13 (depends on 10) The system of claim 10, wherein generating the modified feature representation comprises: generating a first interaction map from the selection input; generating a second interaction map from the selection input, the second interaction map having a resolution different from the first interaction map; and combining the first interaction map and the second interaction map with the image feature representation. ### Claim 14 (depends on 10) The system of claim 10, wherein generating the mask for the object portrayed in the digital image comprises: identifying a subset of masking proposals by filtering the plurality of masking proposals based on the selection input; generating, utilizing the machine-learning mask model, a plurality of masks and a plurality of confidence scores for the object portrayed in the digital image from the modified feature representation and the subset of masking proposals; and selecting the mask from the plurality of masks based on the plurality of confidence scores. ### Claim 15 (depends on 10) The system of claim 10, wherein the operations further comprise training the machine-learning mask model by: determining a center of the object portrayed in the digital image; receiving the selection input by generating a simulated selection input based on the center of the object portrayed in the digital image; and modifying parameters of the machine-learning mask model based on comparing the mask for the object portrayed in the digital image with a ground truth mask for the object portrayed in the digital image. ### Claim 16 A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to perform operations comprising: generating, utilizing a feature extraction model, an image feature representation by encoding feature information from a digital image at a plurality of resolutions to generate a feature vector representation; receiving, via a graphical user interface of a client device, a selection input identifying a position of one or more pixels corresponding to an object portrayed in the digital image; generating, utilizing a click integration model, a modified feature representation by integrating an encoded spatial location for the selection input into the image feature representation; generating, utilizing an object detection model, a plurality of masking proposals from the digital image based on the modified feature representation; and generating, utilizing a machine-learning mask model and based on the plurality of masking proposals, a mask for the object portrayed in the digital image from the modified feature representation and the digital image. ### Claim 17 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein integrating the selection input into the image feature representation comprises: generating an interaction map from a locus of pixels of the selection input; and combining the interaction map and the image feature representation to generate the modified feature representation. ### Claim 18 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein the operations further comprise: receiving an additional selection input identifying an additional one or more pixels corresponding to the object portrayed in the digital image; generating an additional modified feature representation by integrating the selection input and the additional selection input into the image feature representation; and generating, utilizing the machine-learning mask model, a refined mask for the object portrayed in the digital image from the additional modified feature representation. ### Claim 19 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein generating the image feature representation comprises: generating, utilizing the feature extraction model, a first image feature representation at a first resolution and a second image feature representation at a second resolution different from the first resolution; and generating a first interaction map at the first resolution from the selection input and a second interaction map at the second resolution from the selection input. ### Claim 20 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein generating the mask comprises: generating, utilizing an object detection model, a plurality of masking proposals from the digital image based on the modified feature representation; identifying a subset of masking proposals by filtering the plurality of masking proposals based on the selection input; generating, utilizing the machine-learning mask model, a plurality of masks and a plurality of confidence scores for the object portrayed in the digital image from the modified feature representation and the subset of masking proposals; and selecting the mask from the plurality of masks based on the plurality of confidence scores.