Model fine-tuning for automated augmented reality descriptions

A method comprising:

• accessing a first input image and a second input image, the second input image generated by applying a target augmented reality (AR) effect to the first input image;
• providing the first input image and the second input image to a first visual-semantic machine learning model to obtain output describing at least one feature of the target AR effect, the first visual-semantic machine learning model being fine-tuned from a second visual-semantic machine learning model on training data comprising a plurality of training samples, each training sample comprising a first training image, a second training image, and a training description of a given AR effect, and the second training image being generated by applying the given AR effect to the first training image;
• selecting, based on the output of the first visual-semantic machine learning model, a description of the target AR effect;
• storing the description of the target AR effect in association with an identifier of the target AR effect, the storing comprising indexing the target AR effect based on the description of the target AR effect;
• receiving, from a user device of a user, a search query;
• matching, based on the indexing of the target AR effect, the target AR effect with the search query;
• detecting user selection of the target AR effect;
• and
• based on the matching of the target AR effect with the search query and the user selection of the target AR effect, causing presentation of image data at the user device, the target AR effect being applied to the image data.

A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

• accessing a first input image and a second input image, the second input image generated by applying a target augmented reality (AR) effect to the first input image;
• providing the first input image and the second input image to a first visual-semantic machine learning model to obtain output describing at least one feature of the target AR effect, the first visual-semantic machine learning model being fine-tuned from a second visual-semantic machine learning model on training data comprising a plurality of training samples, each training sample comprising a first training image, a second training image, and a training description of a given AR effect, and the second training image being generated by applying the given AR effect to the first training image;
• selecting, based on the output of the first visual-semantic machine learning model, a description of the target AR effect;
• storing the description of the target AR effect in association with an identifier of the target AR effect, the storing comprising indexing the target AR effect based on the description of the target AR effect;
• receiving, from a user device of a user, a search query;
• matching, based on the indexing of the target AR effect, the target AR effect with the search query;
• detecting user selection of the target AR effect;
• and
• based on the matching of the target AR effect with the search query and the user selection of the target AR effect, causing presentation of image data at the user device, the target AR effect being applied to the image data.