# US 12,008,698 B2 · Image segmentation using text embedding > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,698 B2 - Document ID: US12008698B2 - Status: Granted - Kind: B2 - Application number: 18/117,155 - Filed: 2023-03-03 - Published: 2023-06-29 - Granted: 2024-06-11 - Earliest claimed: 2020-11-18 - Assignees: Adobe Inc. - Applicants: Adobe Inc. - Inventors: Midhun Harikumar; Pranav Aggarwal; Baldo Faieta; Ajinkya Kale; Zhe Lin - Web page: https://patentgrep.com/patent/US12008698B2 ## CPC - G06T 7/11: Region-based segmentation - G06T 7/162: involving graph-based methods - G06T 11/60: Creating or editing images; Combining images with text - G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/82: using neural networks - G06V 20/60: Type of objects - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A non-transitory computer-readable medium includes program code that is stored thereon. The program code is executable by one or more processing devices for performing operations including generating, using a model, a learned image representation of a target image. The operations further include generating, using a text embedding model, a text embedding of a text query. The text embedding and the learned image representation of the target image are in a same embedding space. Additionally, the operations include convolving the learned image representation of the target image with the text embedding of the text query. Moreover, the operations include generating an object-segmented image based on the convolving of the learned image representation of the target image with the text embedding. ## Claims ### Claim 1 A non-transitory computer-readable medium having program code that is stored thereon, the program code executable by one or more processing devices for performing operations comprising: generating, using a model, a learned image representation of a target image; generating, using a text embedding model, a text embedding of a text query, wherein the text embedding and the learned image representation of the target image are in a same embedding space; convolving the learned image representation of the target image with the text embedding of the text query; and generating an object-segmented image based on the convolving of the learned image representation with the text embedding. ### Claim 2 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein generating the object-segmented image comprises: generating a fine-attention mask by applying a mask refinement network to (i) a class activation map of the target image generated via the convolving of the learned image representation with the text embedding and (ii) the target image; and segmenting the target image into the object-segmented image by, at least, applying the fine-attention mask to the target image. ### Claim 3 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein generating, by the model that includes trainable components, the learned image representation of the target image comprises applying a convolutional neural network to the target image to generate the learned image representation of the target image. ### Claim 4 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein generating the learned image representation of the target image comprises generating the learned image representation of the target image in a three-dimensional feature space. ### Claim 5 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein generating the text embedding of the text query comprises applying a cross-lingual, multi-modal model to the text query. ### Claim 6 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein generating the text embedding of the text query comprises applying the text embedding model to a multi-word phrase to generate the text embedding of the multi-word phrase in the embedding space. ### Claim 7 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein generating the text embedding of the text query comprises: applying a universal sentence encoder to the text query to generate a cross-lingual representation of the text query; and applying the text embedding model to the cross-lingual representation of the text query to generate the text embedding in the embedding space. ### Claim 8 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the object-segmented image is generated using a class activation map of the target image generated via the convolving of the learned image representation with the text embedding and wherein generating the object-segmented image further comprises highlighting at least one component of the target image that is relevant to the text query. ### Claim 9 A method comprising: receiving a target image and a text query; a step for segmenting the target image based on convolving a learned image representation of the target image with a text embedding of the text query; and outputting an object-segmented image generated by the step for segmenting the target image. ### Claim 10 (depends on 9) The method of claim 9, wherein the step for segmenting the target image comprises: generating a fine-attention mask by applying a mask refinement network to (i) a class activation map of the target image determined via the convolving of the learned image representation of the target image with the text embedding of the text query, and to (ii) the target image; and segmenting the target image into the object-segmented image by, at least, applying the fine-attention mask to the target image. ### Claim 11 (depends on 9) The method of claim 9, further comprising: applying a convolutional neural network to the target image to generate the learned image representation of the target image. ### Claim 12 (depends on 11) The method of claim 11, wherein the learned image representation of the target image comprises a three-dimensional feature space. ### Claim 13 (depends on 9) The method of claim 9, further comprising: applying a cross-lingual, multi-modal model to the text query to generate the text embedding. ### Claim 14 (depends on 9) The method of claim 9, further comprising: applying a text embedding model to a multi-word phrase of the text query to generate the text embedding of the multi-word phrase, wherein the text embedding and the learned image representation of the target image are in a same embedding space. ### Claim 15 (depends on 9) The method of claim 9, further comprising: applying a universal sentence encoder to the text query to generate a cross-lingual representation of the text query; and applying a text embedding model to the cross-lingual representation of the text query to generate the text embedding. ### Claim 16 A system comprising: a processing device; and a non-transitory computer-readable medium communicatively coupled to the processing device, wherein the processing device is configured to execute program code stored in the non-transitory computer-readable medium and thereby perform operations comprising: accessing a learned image representation of a target image and a text embedding of a text query, wherein the text embedding and the learned image representation of the target image are in a same embedding space; convolving the learned image representation of the target image with the text embedding of the text query; and generating an object-segmented image based on the convolving. ### Claim 17 (depends on 16) The system of claim 16, wherein the operation of generating the object-segmented image further comprises: generating a fine-attention mask by applying a mask refinement network to (i) a class activation map of the target image generated via the convolving and (ii) the target image; and segmenting the target image into the object-segmented image by, at least, applying the fine-attention mask to the target image. ### Claim 18 (depends on 16) The system of claim 16, the operations further comprising: generating, by a convolutional neural network, the learned image representation of the target image comprises. ### Claim 19 (depends on 16) The system of claim 16, the operations further comprising: generating the learned image representation of the target image in a three-dimensional feature space. ### Claim 20 (depends on 16) The system of claim 16, wherein generating the object-segmented image based on the convolving further comprises highlighting at least one component of the target image that is relevant to the text query.