# US 12,602,842 B2 · Texture generation using multimodal embeddings > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,842 B2 - Document ID: US12602842B2 - Status: Granted - Kind: B2 - Application number: 18/529,550 - Filed: 2023-12-05 - Published: 2024-10-24 - Granted: 2026-04-14 - Earliest claimed: 2023-04-18 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: Bohdan Ahafonov; Matthew Hallberg; Sergei Korolev; William Miles Miller; Daria Skrypnyk; Aleksei Stoliar - Web page: https://patentgrep.com/patent/US12602842B2 ## CPC - G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) - G06T 7/11: Region-based segmentation - G06T 11/40: Filling planar surfaces by adding surface attributes, e.g. adding colours or textures - G06T 11/60: Creating or editing images; Combining images with text - G06V 20/20: in augmented reality scenes - G10L 15/08: Speech classification or search - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G06T 2210/16: Cloth - G10L 2015/088: {Word spotting} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Methods and systems are disclosed for generating an extended reality (XR) try-on experience. The methods and systems store, in a multimodal memory, interaction data representing use of one or more interaction functions including data in different modalities. The methods and systems detect an object depicted in an image captured by an interaction client and generate, by a machine learning model, a prompt based on the object depicted in the image and the interaction data in the multimodal memory. The methods and systems generate an artificial texture based on the prompt and modify a texture of the object depicted in the image using the artificial texture that has been generated based on the prompt. ## Claims ### Claim 1 A method comprising: storing, in a multimodal memory, interaction data representing use of one or more interaction functions including data in different modalities; detecting an object depicted in an image captured by an interaction client; generating, by a machine learning model, a prompt based on the object depicted in the image and the interaction data in the multimodal memory; generating an artificial texture based on the prompt; modifying a texture of the object depicted in the image using the artificial texture that has been generated based on the prompt; accessing a fashion item template based on the prompt; providing the fashion item template to the machine learning model to populate the fashion item template with the artificial texture; and overlaying the fashion item template populated with the artificial texture on the object depicted in the image. ### Claim 2 (depends on 1) The method of claim 1, wherein the prompt comprises a textual description of a fashion item worn by the object depicted in the image. ### Claim 3 (depends on 1) The method of claim 1, further comprising: processing the prompt using a generative machine learning model to generate the artificial texture. ### Claim 4 (depends on 1) The method of claim 1, further comprising: receiving input from the interaction client that specifies one or more parameters of the prompt. ### Claim 5 (depends on 1) The method of claim 1, further comprising: processing the image to generate a segmentation of a fashion item worn by the object depicted in the image. ### Claim 6 (depends on 5) The method of claim 5, wherein the fashion item worn by the object is replaced with the artificial texture based on the segmentation of the fashion item. ### Claim 7 (depends on 1) The method of claim 1, further comprising: continuously monitoring speech input received from a user of the interaction client; and generating the prompt in response to detecting one or more keywords in the speech input. ### Claim 8 (depends on 7) The method of claim 7, wherein the one or more keywords correspond to a fashion item depicted in the image and one or more attributes. ### Claim 9 (depends on 8) The method of claim 8, wherein the one or more attributes comprise a fashion item style, further comprising: adding to the prompt an identification of the fashion item and a target style corresponding to the fashion item style, wherein the artificial texture comprises an artificial fashion item in the target style. ### Claim 10 (depends on 9) The method of claim 9, further comprising: identifying one or more preferences of the user based on the interaction data stored in the multimodal memory; adding to the prompt an indication of the one or more preferences, wherein the artificial texture comprises a representation of the one or more preferences. ### Claim 11 (depends on 10) The method of claim 10, wherein the one or more preferences correspond to a sporting event associated with the user, wherein the fashion item style corresponds to a jersey associated with the sporting event, and wherein the one or more preferences comprise a team associated with the user and the representation comprises a logo of the team. ### Claim 12 (depends on 1) The method of claim 1, further comprising: generating, by the machine learning model, an annotation representing the prompt; and overlaying the annotation on the image that comprises the object with the modified texture. ### Claim 13 (depends on 1) The method of claim 1, further comprising: detecting ambiguity in speech input received from a user of the interaction client; and resolving the ambiguity based on the multimodal memory, wherein the prompt is generated in response to resolving the ambiguity in the speech input. ### Claim 14 (depends on 1) The method of claim 1, wherein the fashion item template is generated using a segmentation machine learning model that is applied to the object depicted in the image. ### Claim 15 (depends on 14) The method of claim 14, wherein the segmentation machine learning model is selected from a plurality of segmentation machine learning models based on keywords in the prompt. ### Claim 16 (depends on 1) The method of claim 1, wherein the fashion item template comprises a combination of multiple fashion item templates for different parts of a body, further comprising providing an option for a user to purchase a real-world fashion item corresponding to the fashion item template populated with the artificial texture. ### Claim 17 (depends on 16) The method of claim 16, further comprising: providing a first of the multiple fashion item templates to the machine learning model to populate the first fashion item template with a first artificial texture; providing a second of the multiple fashion item templates to the machine learning model to populate the second fashion item template with a second artificial texture; and overlaying the first and second fashion item templates populated with the first and second artificial textures on the object depicted in the image. ### Claim 18 A system comprising: at least one processor; and at least one memory component having instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to perform operations comprising: storing, in a multimodal memory, interaction data representing use of one or more interaction functions including data in different modalities; detecting an object depicted in an image captured by an interaction client; generating, by a machine learning model, a prompt based on the object depicted in the image and the interaction data in the multimodal memory; generating an artificial texture based on the prompt; modifying a texture of the object depicted in the image using the artificial texture that has been generated based on the prompt; accessing a fashion item template based on the prompt; providing the fashion item template to the machine learning model to populate the fashion item template with the artificial texture; and overlaying the fashion item template populated with the artificial texture on the object depicted in the image. ### Claim 19 A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising: storing, in a multimodal memory, interaction data representing use of one or more interaction functions including data in different modalities; detecting an object depicted in an image captured by an interaction client; generating, by a machine learning model, a prompt based on the object depicted in the image and the interaction data in the multimodal memory; generating an artificial texture based on the prompt; modifying a texture of the object depicted in the image using the artificial texture that has been generated based on the prompt; accessing a fashion item template based on the prompt; providing the fashion item template to the machine learning model to populate the fashion item template with the artificial texture; and overlaying the fashion item template populated with the artificial texture on the object depicted in the image.