# US 12,602,942 B2 · Model fine-tuning for automated augmented reality descriptions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,942 B2 - Document ID: US12602942B2 - Status: Granted - Kind: B2 - Application number: 18/502,868 - Filed: 2023-11-06 - Published: 2025-05-08 - Granted: 2026-04-14 - Earliest claimed: 2023-11-06 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: Maksim Gusarov; Kwot Sin Lee; Patrick Poirson; Chen Wang - Web page: https://patentgrep.com/patent/US12602942B2 ## CPC - G06V 10/82: using neural networks - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06T 11/00: Two-dimensional [2D] image generation - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 20/20: in augmented reality scenes - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A second input image is generated by applying a target augmented reality (AR) effect to a first input image. The first input image and the second input image are provided to a first visual-semantic machine learning model to obtain output describing at least one feature of the target AR effect. The first visual-semantic machine learning model is fine-tuned from a second visual-semantic machine learning model by using training samples. Each training sample comprises a first training image, a second training image, and a training description of a given AR effect. The second training image is generated by applying the given AR effect to the first training image. A description of the target AR effect is selected based on the output of the visual-semantic machine learning model. The description of the target AR effect is stored in association with an identifier of the target AR effect. ## Claims ### Claim 1 A system comprising: at least one processor; and at least one memory component storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising: accessing a first input image and a second input image, the second input image generated by applying a target augmented reality (AR) effect to the first input image; providing the first input image and the second input image to a first visual-semantic machine learning model to obtain output describing at least one feature of the target AR effect, the first visual-semantic machine learning model being fine-tuned from a second visual-semantic machine learning model on training data comprising a plurality of training samples, each training sample comprising a first training image, a second training image, and a training description of a given AR effect, and the second training image being generated by applying the given AR effect to the first training image; selecting, based on the output of the first visual-semantic machine learning model, a description of the target AR effect; storing the description of the target AR effect in association with an identifier of the target AR effect, the storing comprising indexing the target AR effect based on the description of the target AR effect; receiving from a user device of a user, a search query; matching, based on the indexing of the target AR effect, the target AR effect with the search query; detecting user selection of the target AR effect; and based on the matching of the target AR effect with the search query and the user selection of the target AR effect, causing presentation of image data at the user device, the target AR effect being applied to the image data. ### Claim 2 (depends on 1) The system of claim 1, wherein fine-tuning of the second visual-semantic machine learning model comprises using the training samples to adapt a transformer of the second visual-semantic machine learning model to obtain the first visual-semantic machine learning model. ### Claim 3 (depends on 2) The system of claim 2, wherein the transformer is between an image encoder and a large language model of the first visual-semantic machine learning model, the first visual-semantic machine learning model being fine-tuned by adapting attention parameters of the transformer while keeping the image encoder and the large language model fixed. ### Claim 4 (depends on 2) The system of claim 2, wherein the second visual-semantic machine learning model is fine-tuned by Low-Rank Adaptation (LoRA). ### Claim 5 (depends on 1) The system of claim 1, wherein the at least one feature comprises at least one visual feature, the first visual-semantic machine learning model being fine-tuned to describe the at least one visual feature based on a visual transition from the first input image to the second input image, as caused by the target AR effect. ### Claim 6 (depends on 1) The system of claim 1, wherein the first visual-semantic machine learning model uses the first input image and the second input image to perform a captioning operation that is AR effect-specific. ### Claim 7 (depends on 1) The system of claim 1, the operations further comprising: applying the target AR effect to the first input image to obtain the second input image. ### Claim 8 (depends on 1) The system of claim 1, wherein the first input image and the second input image are provided to the first visual-semantic machine learning model as a concatenated image. ### Claim 9 (depends on 8) The system of claim 8, the operations further comprising: generating the concatenated image by concatenating the first input image and the second input image. ### Claim 10 (depends on 9) The system of claim 9, wherein the concatenating of the first input image and the second input image comprises positioning the first input image and the second input image relative to each other in a predetermined spatial arrangement to obtain the concatenated image. ### Claim 11 (depends on 1) The system of claim 1, wherein, for each training sample, the first training image and the second training image are concatenated into a concatenated training image. ### Claim 12 (depends on 1) The system of claim 1, wherein the first input image is a frame of a base video and the second input image is a frame of an augmented video, the augmented video generated by rendering the target AR effect on the base video. ### Claim 13 (depends on 12) The system of claim 12, the operations further comprising: identifying the frame of the augmented video as temporally corresponding to the frame of the base video; extracting the frame of the base video to obtain the first input image; and extracting the frame of the augmented video to obtain the second input image. ### Claim 14 (depends on 1) The system of claim 1, wherein the description of the target AR effect is a natural language text description. ### Claim 15 (depends on 1) The system of claim 1, the operations further comprising: determining, based on the description of the target AR effect, a category of the target AR effect; and storing the category of the target AR effect in association with the identifier of the target AR effect. ### Claim 16 (depends on 1) The system of claim 1, wherein the target AR effect is one of a plurality of AR effects supported by an interaction system, and the first visual-semantic machine learning model is used to generate descriptions of each of the plurality of AR effects. ### Claim 17 (depends on 16) The system of claim 16, wherein the storing of the description of the target AR effect in association with the identifier of the target AR effect comprises storing the description of the target AR effect in a database that maps each of the plurality of AR effects to a respective description. ### Claim 18 (depends on 1) The system of claim 1, wherein the search query comprises one or more of a keyword, a tag, or the description of the target AR effect. ### Claim 19 A method comprising: accessing a first input image and a second input image, the second input image generated by applying a target augmented reality (AR) effect to the first input image; providing the first input image and the second input image to a first visual-semantic machine learning model to obtain output describing at least one feature of the target AR effect, the first visual-semantic machine learning model being fine-tuned from a second visual-semantic machine learning model on training data comprising a plurality of training samples, each training sample comprising a first training image, a second training image, and a training description of a given AR effect, and the second training image being generated by applying the given AR effect to the first training image; selecting, based on the output of the first visual-semantic machine learning model, a description of the target AR effect; storing the description of the target AR effect in association with an identifier of the target AR effect, the storing comprising indexing the target AR effect based on the description of the target AR effect; receiving, from a user device of a user, a search query; matching, based on the indexing of the target AR effect, the target AR effect with the search query; detecting user selection of the target AR effect; and based on the matching of the target AR effect with the search query and the user selection of the target AR effect, causing presentation of image data at the user device, the target AR effect being applied to the image data. ### Claim 20 A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising: accessing a first input image and a second input image, the second input image generated by applying a target augmented reality (AR) effect to the first input image; providing the first input image and the second input image to a first visual-semantic machine learning model to obtain output describing at least one feature of the target AR effect, the first visual-semantic machine learning model being fine-tuned from a second visual-semantic machine learning model on training data comprising a plurality of training samples, each training sample comprising a first training image, a second training image, and a training description of a given AR effect, and the second training image being generated by applying the given AR effect to the first training image; selecting, based on the output of the first visual-semantic machine learning model, a description of the target AR effect; storing the description of the target AR effect in association with an identifier of the target AR effect, the storing comprising indexing the target AR effect based on the description of the target AR effect; receiving, from a user device of a user, a search query; matching, based on the indexing of the target AR effect, the target AR effect with the search query; detecting user selection of the target AR effect; and based on the matching of the target AR effect with the search query and the user selection of the target AR effect, causing presentation of image data at the user device, the target AR effect being applied to the image data.