Efficient augmentation for multimodal machine learning
AssigneeAdobe Inc.
InventorsHandong Zhao, Yue Bai, Zhe Lin, Ajinkya Gorakhnath Kale, Jiuxiang Gu, Tong Yu, Sungchul Kim
Life of this patent
Estimating the term…
Abstract
Systems and methods for multimodal machine learning are provided. According to one aspect, a method for multimodal machine learning includes obtaining a prompt; encoding the prompt using a multimodal encoder to obtain a prompt embedding, wherein the encoding comprises generating a plurality of multi-head attention (MHA) outputs corresponding to a plurality of different scales, respectively, and combining the plurality of MHA outputs using a multi-scale aggregator; and generating a response to the prompt based on the prompt embedding.
Claims20
A method for multimodal machine learning, comprising:
- obtaining a prompt;
- encoding the prompt using a multimodal encoder to obtain a prompt embedding, wherein the encoding comprises generating a plurality of multi-head attention (MHA) outputs corresponding to a plurality of different scales, respectively, and combining the plurality of MHA outputs using a multi-scale aggregator;
- and
- generating a response to the prompt based on the prompt embedding.
A method for multimodal machine learning, comprising:
- obtaining training data comprising an image and text describing the image;
- encoding the text using a multimodal encoder to obtain a predicted text embedding, wherein encoding the text comprises generating a plurality of multi-head attention (MHA) text outputs corresponding to a plurality of different text scales, respectively, and combining the plurality of MHA text outputs using a text multi-scale aggregator;
- encoding the image using the multimodal encoder to obtain a predicted image embedding, wherein encoding the image comprises generating a plurality of MHA image outputs corresponding to a plurality of different image scales, respectively, and combining the plurality of MHA image outputs using an image multi-scale aggregator;
- and
- training the multimodal encoder based on the predicted image embedding and the predicted text embedding.
An apparatus for multimodal machine learning, comprising:
- at least one processor;
- at least one memory storing instructions executable by the processor;
- and
- the apparatus further comprising a multimodal encoder comprising parameters stored in the at least one memory, wherein the multimodal encoder comprises a multi-scale aggregator and is configured to encode a prompt to obtain a prompt embedding by generating a plurality of multi-head attention (MHA) outputs corresponding to a plurality of different scales, respectively, and combining the plurality of MHA outputs using the multi-scale aggregator.
Dependencies are inferred from the claim text.