# US 12,223,533 B2 · Method, medium, and system for intelligent online personal assistant with image text localization > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,533 B2 - Document ID: US12223533B2 - Status: Granted - Kind: B2 - Application number: 17/222,251 - Filed: 2021-04-05 - Published: 2021-07-22 - Granted: 2025-02-11 - Earliest claimed: 2016-11-11 - Assignees: eBay Inc. - Applicants: eBay Inc. - Inventors: Shuai Zheng; Robinson Piramuthu - Web page: https://patentgrep.com/patent/US12223533B2 ## CPC - G06Q 30/0625: {by formulating product or service queries, e.g. using keywords or predefined options} - G06F 16/532: Query formulation, e.g. graphical querying - G06F 16/583: using metadata automatically derived from the content - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/22: Matching criteria, e.g. proximity measures - G06F 18/24: Classification techniques - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 5/02: Knowledge representation; Symbolic representation - G06Q 20/12: specially adapted for electronic shopping systems - G06Q 20/40: Authorisation, e.g. identification of payer or payee, verification of customer or shop credentials; Review and approval of payers, e.g. check credit lines or negative lists - G06T 7/0004: {Industrial image inspection} - G06V 20/63: {Scene text, e.g. street names} - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06T 2207/30108: Industrial image inspection - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Systems, methods, and computer program products for identifying a candidate product in an electronic marketplace based on a visual comparison between candidate product image visual text content and input query image visual text content. Unlike conventional optical character recognition (OCR) based systems, embodiments automatically localize and isolate portions of a candidate product image and an input query image that each contain visual text content, and calculate a visual similarity measure between the respective portions. A trained neural network may be re-trained to more effectively find visual text content by using the localized and isolated visual text content portions as additional ground truths. The visual similarity measure serves as a visual search result score for the candidate product. Any number of images of any number of candidate products may be compared to an input query image to enable text-in-image based product searching without resorting to conventional OCR techniques. ## Claims ### Claim 1 A method comprising: localizing and isolating a candidate portion of a candidate product image with candidate visual text content; retraining a neural network using the candidate portion of the candidate product image; localizing and isolating, via the retrained neural network, an input query portion of an input query image with input query visual text content; calculating a visual similarity measure between the candidate portion and the input query portion based on a distance value between image signatures associated with the candidate product image and the input query image; ranking a candidate product in a product list based on the visual similarity measure; and causing presentation of the product list on a graphical user interface of a client device. ### Claim 2 (depends on 1) The method of claim 1, wherein the input query image comprises one of: a photograph; a video frame; a sketch; or a diagram. ### Claim 3 (depends on 1) The method of claim 1, wherein the candidate product image is associated with an electronic marketplace. ### Claim 4 A non-transitory computer-readable storage medium having embedded therein a set of instructions which, when executed by one or more processors of a computer, causes the computer to execute operations comprising: localizing and isolating a candidate portion of a candidate product image with candidate visual text content; retraining a neural network using the candidate portion of the candidate product image; localizing and isolating, via the retrained neural network, an input query portion of an input query image with input query visual text content; calculating a visual similarity measure between the respective portions of the candidate product image and the input query image based on a distance value between image signatures associated with the candidate product image and the input query image; ranking a candidate product in a product list based on the visual similarity measure; and causing presentation of the product list on a graphical user interface of a client device. ### Claim 5 A system comprising: at least one processor; and memory encoding computer-executable instructions that, when executed by the at least one processor, cause the system to perform operations comprising: localizing and isolating a candidate portion of a candidate product image with candidate visual text content; retraining a neural network using the candidate portion of the candidate product image; localizing and isolating, via the retrained neural network, an input query portion of an input query image with input query visual text content; calculating a visual similarity measure between the candidate portion and the input query portion based on a distance value between image signatures associated with the candidate product image and the input query image; ranking a candidate product in a product list based on the visual similarity measure; and causing presentation of the product list on a graphical user interface of a client device. ### Claim 6 (depends on 5) The system of claim 5, wherein the input query image comprises one of: a photograph; a video frame; a sketch; or a diagram. ### Claim 7 (depends on 1) The method of claim 1, wherein localizing and isolating the respective portions of the candidate product image and the input query image comprises localizing and isolating, using a machine learning model, the respective portions of the candidate product image and the input query image. ### Claim 8 (depends on 7) The method of claim 7, wherein the machine learning model comprises a deep neural network. ### Claim 9 (depends on 7) The method of claim 7, further comprising retraining the machine learning model using a new image of a new product provided to an electronic marketplace. ### Claim 10 (depends on 1) The method of claim 1, wherein the image signatures comprise binary vectors, and the distance value comprises one or more bits that are different in the binary vectors. ### Claim 11 (depends on 4) The non-transitory computer-readable storage medium of claim 4, wherein the input query image comprises one of: a photograph; a video frame; a sketch; or a diagram. ### Claim 12 (depends on 4) The non-transitory computer-readable storage medium of claim 4, wherein the candidate product image is associated with an electronic marketplace. ### Claim 13 (depends on 4) The non-transitory computer-readable storage medium of claim 4, wherein localizing and isolating the respective portions of the candidate product image and the input query image comprises localizing and isolating, using a machine learning model, the respective portions of the candidate product image and the input query image. ### Claim 14 (depends on 13) The non-transitory computer-readable storage medium of claim 13, wherein the machine learning model comprises a deep neural network. ### Claim 15 (depends on 13) The non-transitory computer-readable storage medium of claim 13, wherein the operations further comprise retraining the machine learning model using a new image of a new product provided to an electronic marketplace. ### Claim 16 (depends on 4) The non-transitory computer-readable storage medium of claim 4, wherein the image signatures comprise binary vectors, and the distance value comprises one or more bits that are different in the binary vectors. ### Claim 17 (depends on 5) The system of claim 5, wherein the candidate product image is associated with an electronic marketplace. ### Claim 18 (depends on 5) The system of claim 5, wherein localizing and isolating the respective portions of the candidate product image and the input query image comprises localizing and isolating, using a machine learning model, the respective portions of the candidate product image and the input query image. ### Claim 19 (depends on 18) The system of claim 18, wherein the operations further comprise retraining the machine learning model using a new image of a new product provided to an electronic marketplace. ### Claim 20 (depends on 5) The system of claim 5, wherein the image signatures comprise binary vectors, and the distance value comprises one or more bits that are different in the binary vectors.