# US 12,183,053 B2 · System and method for product search by embedding visual representation into text sequences > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,183,053 B2 - Document ID: US12183053B2 - Status: Granted - Kind: B2 - Application number: 17/590,259 - Filed: 2022-02-01 - Published: 2023-08-03 - Granted: 2024-12-31 - Earliest claimed: 2022-02-01 - Assignees: Beijing Wodong Tianjun Information Technology Co., Ltd. - Applicants: Beijing Wodong Tianjun Information Technology Co., Ltd. - Inventors: Chi Zhang; Xueying Zhang; Yunjiang Jiang; Xiaochuan Fan; Yue Shang; Yun Xiao - Web page: https://patentgrep.com/patent/US12183053B2 ## CPC - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06F 16/532: Query formulation, e.g. graphical querying - G06F 16/954: Navigation, e.g. using categorised browsing - G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion - G06V 10/32: Normalisation of the pattern dimensions - G06V 10/467: {Encoded features or binary features, e.g. local binary patterns [LBP]} - G06V 10/82: using neural networks - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A computer-implemented method for searching a product corresponding to a query from a customer. The method includes: embedding the query to obtain a query embedding; retrieving product information having a product text and a product image; embedding the product text to obtain a product text embedding, embedding the product image to obtain a product image embedding, and combining the product text embedding and the product image embedding to obtain a product embedding, where the product image embedding has a same format as the product text embedding; subjecting the query embedding and the product embedding to a transformer to determine whether the query and the product are relevant; and providing the product as a search result of the query when the query and the product are relevant. ## Claims ### Claim 1 A computer-implemented method for searching a product corresponding to a query from a customer, comprising: embedding the query to obtain a query embedding; retrieving product information comprising a product text and a product image; embedding the product text to obtain a product text embedding, embedding the product image to obtain a product image embedding, and combining the product text embedding and the product image embedding to obtain a product embedding, wherein the product image embedding has a same format as the product text embedding, wherein embedding the product image comprises: normalizing the product image to obtain a normalized product image; splitting the normalized product image into a plurality of grids: concatenating the plurality of grids into a sequence of grids; and extracting product image feature from the sequence of grids to obtain a grid token for each of the sequence of grids, wherein embedding the product image further comprises: adding a position vector to each of the grid tokens, the position vectors representing positions of the grids in the sequence of grids; adding a segment vector to each of the grid tokens, the segment vectors representing identification of the product image: adding a mask vector to each of the grid tokens, wherein values of the mask vectors are 0 or 1, and value of one of the grid tokens is converted to 0 when the value of corresponding one of the mask vectors is 0; and defining a category identification, the category identification representing category of the product in an e-commerce platform, wherein the product embedding comprises the grid tokens, the position vectors, the segment vectors, the mask vectors, and the category identification; subjecting the query embedding and the product embedding to a transformer to determine whether the query and the product are relevant; and providing the product as a search result of the query when the query and the product are relevant. ### Claim 2 (depends on 1) The method of claim 1, wherein the transformer comprises a query transformer for processing the query embedding and a product transformer for processing the product embedding. ### Claim 3 (depends on 1) The method of claim 1, wherein the query comprises a query text and a query image, and the query embedding comprises a query text embedding corresponding to the query text and a query image embedding corresponding to the query image. ### Claim 4 (depends on 1) The method of claim 1, wherein about 10-15% of the mask vectors have a value of 0. ### Claim 5 (depends on 1) The method of claim 1, wherein the step of extracting the product image feature from the sequence of grids comprises performing a convolutional neural network (CNN) on each of the grids, and the transformer is a pre-trained bidirectional encoder representations from transformers (BERT). ### Claim 6 A system for searching a product corresponding to a query from a customer, wherein the system comprises a computing device, the computing device comprises a processor and a storage device storing computer executable code, and the computer executable code, when executed at the processor, is configured to: embed the query to obtain a query embedding; retrieve product information comprising a product text and a product image; embed the product text to obtain a product text embedding, embed the product image to obtain a product image embedding, and combine the product text embedding and the product image embedding to obtain a product embedding, wherein the product image embedding has a same format as the product text embedding, wherein the computer executable code is configured to embed the product image by: normalizing the product image to obtain a normalized product image; splitting the normalized product image into a plurality of grids; concatenating the plurality of grids into a sequence of grids; and extracting product image feature from the sequence of grids to obtain a grid token for each of the sequence of grids, wherein the computer executable code is further configured to embed the product image by: adding a position vector to each of the grid tokens, the position vectors representing positions of the grids in the sequence of grids; adding a segment vector to each of the grid tokens, the segment vectors representing identification of the product image; adding a mask vector to each of the grid tokens, wherein values of the mask vectors are 0 or 1, and value of one of the grid tokens is converted to 0 when the value of corresponding one of the mask vectors is 0; and defining a category identification, the category identification representing category of the product in an e-commerce platform, wherein the product embedding comprises the grid tokens, the position vectors, the segment vectors, the mask vectors, and the category identification; subject the query embedding and the product embedding to a transformer to determine whether the query and the product are relevant; and provide the product as a search result of the query when the query and the product are relevant. ### Claim 7 (depends on 6) The system of claim 6, wherein the transformer comprises a query transformer for processing the query embedding and a product transformer for processing the product embedding. ### Claim 8 (depends on 6) The system of claim 6, wherein the query comprises a query text and a query image, and the query embedding comprises a query text embedding corresponding to the query text and a query image embedding corresponding to the query image. ### Claim 9 (depends on 6) The system of claim 6, wherein about 10-15% of the mask vectors have a value of 0. ### Claim 10 (depends on 6) The system of claim 6, wherein the computer executable code is configured extract the product image feature from the sequence of grids by performing a convolutional neural network (CNN) on each of the grids, and the transformer is a pre-trained bidirectional encoder representations from transformers (BERT). ### Claim 11 A non-transitory computer readable medium storing computer executable code, wherein the computer executable code, when executed at a processor of a computing device, is configured to: embed the query to obtain a query embedding; retrieve product information comprising a product text and a product image; embed the product text to obtain a product text embedding, embed the product image to obtain a product image embedding, and combine the product text embedding and the product image embedding to obtain a product embedding, wherein the product image embedding has a same format as the product text embedding, wherein the computer executable code is configured to embed the product image by: normalizing the product image to obtain a normalized product image; splitting the normalized product image into a plurality of grids; concatenating the plurality of grids into a sequence of grids; and extracting product image feature from the sequence of grids to obtain a grid token for each of the sequence of grids, wherein the computer executable code is further configured to embed the product image by: adding a position vector to each of the grid tokens, the position vectors representing positions of the grids in the sequence of grids; adding a segment vector to each of the grid tokens, the segment vectors representing identification of the product image; adding a mask vector to each of the grid tokens, wherein values of the mask vectors are 0 or 1, and value of one of the grid tokens is converted to 0 when the value of corresponding one of the mask vectors is 0; and defining a category identification, the category identification representing category of the product in an e-commerce platform, wherein the product embedding comprises the grid tokens, the position vectors, the segment vectors, the mask vectors, and the category identification; subject the query embedding and the product embedding to a transformer to determine whether the query and the product are relevant; and provide the product as a search result of the query when the query and the product are relevant.