US 12,608,852 B1Granted

Utilizing a multimodal large language model to generate synthesized responses corresponding to multimodal content items

AssigneeDropbox, Inc.

InventorsDongjie Chen, Dhruvil Gala

Life of this patent

Estimating the term…

Loading patent…

Abstract

The present disclosure relates to systems, non-transitory computer-readable media, and methods for processing multimodal content to generate summaries or responses using a multimodal large language model. In one or more embodiments, the disclosed systems the disclosed systems utilize the multimodal large language model to generate various types of synthesized responses corresponding to multimodal content items that contain data and information within images. For example, in some embodiments, in response to receiving a request to generate a synthesized response corresponding to a multimodal content item, the disclosed systems employ preprocessing pipelines that generate thumbnail images from the multimodal content item and use the thumbnail images to generate a data structure for a prompt for the multimodal large language model.

Claims20

Dependent claims

A computer-implemented method comprising:

receiving, from a client device, a request to generate a synthesized response corresponding to a multimodal content item;
generating, in response to receiving the request and from the multimodal content item, one or more thumbnail images representing content of the multimodal content item;
selecting a preprocessing pipeline corresponding to a synthesized response type for the request to generate the synthesized response by selecting an answer preprocessing pipeline or a summary preprocessing pipeline;
generating a prompt data structure corresponding to the synthesized response type by:
selecting, utilizing the summary preprocessing pipeline, one or more sets of thumbnail images from the one or more thumbnail images by selecting one or more sliding windows, each sliding window comprising two or more thumbnail images;
or
selecting, in response to utilizing a multimodal embedding-generating machine-learning model of the answer preprocessing pipeline to generate embedding distances between image embeddings corresponding to the one or more thumbnail images and one or more text embeddings corresponding to a user input of the request to generate the synthesized response, a set of thumbnail images based on the embedding distances;
generating, utilizing a multimodal large language model, the synthesized response by providing the prompt data structure within a prompt to the multimodal large language model;
and
sending the synthesized response for display on the client device.

7 more elements

A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause a computer system to:

receive, from a client device, a request to generate a synthesized response comprising a request for an answer to a user input corresponding to a multimodal content item;
generate, in response to the request, one or more thumbnail images representing content of the multimodal content item;
based on determining that an answer preprocessing pipeline corresponds to the request to generate the synthesized response comprising the request for the answer to the user input, generate, utilizing a multimodal embedding-generating machine-learning model of the answer preprocessing pipeline, one or more image embeddings corresponding to the one or more thumbnail images and one or more text embeddings corresponding to the user input corresponding to the multimodal content item;
generate a prompt data structure corresponding to the request to generate the synthesized response comprising the request for the answer to the user input by selecting of a set of thumbnail images from the one or more thumbnail images based on embedding distances between the one or more image embeddings corresponding to the one or more thumbnail images and the one or more text embeddings corresponding to the user input;
generate, utilizing a multimodal large language model, the synthesized response by providing the set of thumbnail images and instructions to generate the synthesized response using the set of thumbnail images within a prompt to the multimodal large language model;
and
send the synthesized response to the client device.

4 more elements

A system comprising:

at least one processor;
and
at least one non-transitory computer-readable storage medium storing instructions that, when executed by the at least one processor, cause the system to:
receive, from a client device, a request to generate a synthesized summary corresponding to a multimodal content item;
generate, in response to the request, one or more thumbnail images representing content of the multimodal content item;
determine that a summary preprocessing pipeline corresponds to the request to generate the synthesized summary;
generate, utilizing the summary preprocessing pipeline, a prompt data structure corresponding to the request to generate the synthesized summary by selecting one or more sets of thumbnail images from the one or more thumbnail images;
generate, utilizing a multimodal large language model, one or more thumbnail summaries for the one or more sets of thumbnail images of the prompt data structure;
generate, utilizing an additional large language model, the synthesized summary by providing the one or more thumbnail summaries within a prompt to the additional large language model;
and
send the synthesized summary for display on the client device.

8 more elements

Dependencies are inferred from the claim text.