US 12,591,609 B1Granted

Multimodal search and retrieval system

AssigneeTypeface Inc.

InventorsSiddharth Srivastava, Aditya Dhaulakhandi, Harshvardhan Pratap Singh, Neeraj Matiyali, Xinyu Zhao, Gaurav Sharma

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

A Multimodal Search System (MSS) is designed to enable accurate search and retrieval across different modalities of data, including images, videos, and text. The MSS leverages multimodal and domain-specific models to extract and represent features and information from multimodal data, enabling seamless querying using text-based search queries. The MSS integrates multiple pipelines, including feature vector-based and tag-based approaches, to ensure robust and scalable search capabilities. The MSS addresses current multimodal-search challenges by leveraging a hybrid approach that combines multimodal embeddings, text embeddings, and semantic tagging to enable cross-modal searching. Additionally, for visual data, both tagging and embedding techniques are integrated within a single modality, ensuring a comprehensive and accurate representation. By assembling multiple retrieval strategies, the solution harnesses the strengths of each, resulting in a balanced and effective search system.

Claims20

Dependent claims

A computer-implemented method comprising:

processing multimodal data for a plurality of multimodal items in a plurality of modes to generate multimodal feature vectors and text feature vectors, the plurality of modes comprising text snippets, images, videos, and audios, wherein the multimodal feature vectors represent semantic characteristics in a unified vector space to capture cross-modal relationships for the text snippets, images, videos, and audios, the text feature vectors optimized for textual semantics and mapped into a separate latent space from the multimodal feature vectors;
generating, by a large language model, data tags based on semantic information from the plurality of multimodal items, each data tag representing an attribute associated with a corresponding multimodal item;
storing the multimodal feature vectors, the text feature vectors, and the data tags in a database;
receiving a user query in any mode from the plurality of modes;
generating a query multimodal vector, a query text vector, and one or more query tags based on the user query;
matching the query multimodal vector, the query text vector, and the one or more query tags to the multimodal feature vectors, the text feature vectors, and the data tags in the database to obtain initial results:
refining the initial results to obtain refined results, wherein refining the initial results comprises combining scores from multimodal vector matching and scores from text feature vector matching, and reranking the initial results based on the combination of scores;
and
presenting the refined results on a user device, the refined results comprising data from any modality that matches the user query.

6 more elements

A system comprising:

a memory comprising instructions;
and
one or more computer processors, the instructions, when executed by the one or more computer processors, causing the system to perform operations comprising:
processing multimodal data for a plurality of multimodal items in a plurality of modes to generate multimodal feature vectors and text feature vectors, the plurality of modes comprising text snippets, images, videos, and audios, wherein the multimodal feature vectors represent semantic characteristics in a unified vector space to capture cross-modal relationships for the text snippets, images, videos, and audios, the text feature vectors optimized for textual semantics and mapped into a separate latent space from the multimodal feature vectors;
generating, by a large language model, data tags based on semantic information from the plurality of multimodal items, each data tag representing an attribute associated with a corresponding multimodal item;
storing the multimodal feature vectors, the text feature vectors, and the data tags in a database;
receiving a user query in any mode from the plurality of modes;
generating a query multimodal vector, a query text vector, and one or more query tags based on the user query;
matching the query multimodal vector, the query text vector, and the one or more query tags to the multimodal feature vectors, the text feature vectors, and the data tags in the database to obtain initial results;
refining the initial results to obtain refined results, wherein refining the initial results comprises combining scores from multimodal vector matching and scores from text feature vector matching, and reranking the initial results based on the combination of scores;
and
presenting the refined results on a user device, the refined results comprising data from any modality that matches the user query.

9 more elements

A non-transitory machine-readable storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:

processing multimodal data for a plurality of multimodal items in a plurality of modes to generate multimodal feature vectors and text feature vectors, the plurality of modes comprising text snippets, images, videos, and audios, wherein the multimodal feature vectors represent semantic characteristics in a unified vector space to capture cross-modal relationships for the text snippets, images, videos, and audios, the text feature vectors optimized for textual semantics and mapped into a separate latent space from the multimodal feature vectors;
generating, by a large language model, data tags based on semantic information from the plurality of multimodal items, each data tag representing an attribute associated with a corresponding multimodal item;
storing the multimodal feature vectors, the text feature vectors, and the data tags in a database;
receiving a user query in any mode from the plurality of modes;
generating a query multimodal vector, a query text vector, and one or more query tags based on the user query;
matching the query multimodal vector, the query text vector, and the one or more query tags to the multimodal feature vectors, the text feature vectors, and the data tags in the database to obtain initial results;
refining the initial results to obtain refined results, wherein refining the initial results comprises combining scores from multimodal vector matching and scores from text feature vector matching, and reranking the initial results based on the combination of scores;
and
presenting the refined results on a user device, the refined results comprising data from any modality that matches the user query.

6 more elements

Dependencies are inferred from the claim text.