System and method for product search by embedding visual representation into text sequences

A system for searching a product corresponding to a query from a customer, wherein the system comprises a computing device, the computing device comprises a processor and a storage device storing computer executable code, and the computer executable code, when executed at the processor, is configured to:

• embed the query to obtain a query embedding;
• retrieve product information comprising a product text and a product image;
• embed the product text to obtain a product text embedding, embed the product image to obtain a product image embedding, and combine the product text embedding and the product image embedding to obtain a product embedding, wherein the product image embedding has a same format as the product text embedding,
• wherein the computer executable code is configured to embed the product image by:
• normalizing the product image to obtain a normalized product image;
• splitting the normalized product image into a plurality of grids;
• concatenating the plurality of grids into a sequence of grids;
• and extracting product image feature from the sequence of grids to obtain a grid token for each of the sequence of grids,
• wherein the computer executable code is further configured to embed the product image by:
• adding a position vector to each of the grid tokens, the position vectors representing positions of the grids in the sequence of grids;
• adding a segment vector to each of the grid tokens, the segment vectors representing identification of the product image;
• adding a mask vector to each of the grid tokens, wherein values of the mask vectors are 0 or 1, and value of one of the grid tokens is converted to 0 when the value of corresponding one of the mask vectors is 0;
• and defining a category identification, the category identification representing category of the product in an e-commerce platform, wherein the product embedding comprises the grid tokens, the position vectors, the segment vectors, the mask vectors, and the category identification;
• subject the query embedding and the product embedding to a transformer to determine whether the query and the product are relevant;
• and
• provide the product as a search result of the query when the query and the product are relevant.

A non-transitory computer readable medium storing computer executable code, wherein the computer executable code, when executed at a processor of a computing device, is configured to:

• embed the query to obtain a query embedding;
• retrieve product information comprising a product text and a product image;
• embed the product text to obtain a product text embedding, embed the product image to obtain a product image embedding, and combine the product text embedding and the product image embedding to obtain a product embedding, wherein the product image embedding has a same format as the product text embedding,
• wherein the computer executable code is configured to embed the product image by:
• normalizing the product image to obtain a normalized product image;
• splitting the normalized product image into a plurality of grids;
• concatenating the plurality of grids into a sequence of grids;
• and extracting product image feature from the sequence of grids to obtain a grid token for each of the sequence of grids,
• wherein the computer executable code is further configured to embed the product image by:
• adding a position vector to each of the grid tokens, the position vectors representing positions of the grids in the sequence of grids;
• adding a segment vector to each of the grid tokens, the segment vectors representing identification of the product image;
• adding a mask vector to each of the grid tokens, wherein values of the mask vectors are 0 or 1, and value of one of the grid tokens is converted to 0 when the value of corresponding one of the mask vectors is 0;
• and defining a category identification, the category identification representing category of the product in an e-commerce platform, wherein the product embedding comprises the grid tokens, the position vectors, the segment vectors, the mask vectors, and the category identification;
• subject the query embedding and the product embedding to a transformer to determine whether the query and the product are relevant;
• and
• provide the product as a search result of the query when the query and the product are relevant.