# US 10,319,019 B2 · Method, medium, and system for detecting cross-lingual comparable listings for machine translation using image similarity > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,319,019 B2 - Document ID: US10319019B2 - Status: Granted - Kind: B2 - Application number: 15/264,873 - Filed: 2016-09-14 - Published: 2018-03-15 - Granted: 2019-06-11 - Earliest claimed: 2016-09-14 - Assignees: eBay Inc. - Applicants: eBay Inc. - Inventors: Sanjika Hewavitharana; Evgeny Matusov; Robinson Piramuthu; Hassan Sawaf - Web page: https://patentgrep.com/patent/US10319019B2 ## CPC - G06Q 30/0623: {by investigating goods or services} - G06F 16/50: of still image data - G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation - G06N 20/00: Machine learning - G06Q 30/0601: Electronic shopping [e-shopping] - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract In various example embodiments, a system and method for a Listing Engine that translates a first listing from a first language to a second language. The first listing includes an image(s) of a first item. The Listing Engine provides as input to an encoded neural network model a portion(s) of a translated first listing and a portions(s) of a second listing in the second language. The second listing includes an image(s) of a second item. The Listing Engine receives from the encoded neural network model a first feature vector for the translated first listing and a second feature vector for the second listing. The first and the second feature vectors both include at least one type of image signature feature and at least one type of listing text-based feature. Based on a similarity score of the first and second feature vectors at least meeting a similarity score threshold, the Listing Engine generates a pairing of the first listing in the first language with the second listing in the second language for inclusion in training data of a machine translation system. ## Claims ### Claim 1 A computer system comprising: a processor; a memory device holding an instruction set executable on the processor to cause the computer system to perform operations comprising: translating a first listing from a first language to a second language, the first listing including at least one image of a first item; providing as input to an encoded neural network model at least one image portion and at least one text portion of the translated first listing and at least one image portion and at least one text portion of a second listing in the second language, the second listing including at least one image of a second item; receiving from the encoded neural network model a first feature vector for the translated first listing and a second feature vector for the second listing, the first and the second feature vectors both include at least one type of image signature feature and at least one type of listing text-based feature; calculating a similarity score based on the first feature vector and the second feature vector, and based on the similarity score at least meeting a similarity score threshold, generating a pairing of the first listing in the first language with the second listing in the second language for inclusion in a database of training data of a machine translation system, wherein the pairing is based on a degree of likelihood that the first item and the second item are the same. ### Claim 2 (depends on 1) The computer system of claim 1, wherein a first type of image signature feature is based on at least one pixel color of an image in a respective listing and at least one pixel location in the respective listing. ### Claim 3 (depends on 2) The computer system of claim 2, wherein a first type of listing text-based feature is based on an edit distance between respective portions of two listings. ### Claim 4 (depends on 3) The computer system of claim 3, wherein the edit distance comprises an edit distance between a title text portion of the translated first listing and a title text portion of the second listing. ### Claim 5 (depends on 3) The computer system of claim 3, wherein a second type of listing text-based feature is based on a length of a respective listing. ### Claim 6 (depends on 5) The computer system of claim 5, wherein a third type of listing text-based feature is based on a lexical translation overlap between two respective listings. ### Claim 7 (depends on 6) The computer system of claim 6, wherein a fourth type of listing text-based feature is based on a product category of the respective listing. ### Claim 8 (depends on 7) The computer system of claim 7, wherein a fifth type of listing text-based feature is based on structural information of an item comprising at least one of: item color and item brand. ### Claim 9 A computer-implemented method comprising: translating a first listing from a first language to a second language, the first listing including at least one image of a first item; providing as input to a encoded neural network model at least one image portion and at least one text portion of the translated first listing and at least one image portion and at least one text portion of a second listing in the second language, the second listing including at least one image of a second item; receiving from the encoded neural network model a first feature vector for the translated first listing and a second feature vector for the second listing, the first and the second feature vectors both include at least one type of image signature feature and at least one type of listing text-based feature; calculating, via at least one processor, a similarity score based on the first feature vector and the second feature vector, and based on the similarity score at least meeting a similarity score threshold, generating a pairing of the first listing in the first language with the second listing in the second language for inclusion in a database of training data of a machine translation system, wherein the pairing is based on a degree of likelihood that the first item and the second item are the same. ### Claim 10 (depends on 9) The computer-implemented method of claim 9, wherein a first type of image signature feature is based on at least one pixel color of an image in a respective listing and at least one pixel location in the respective listing. ### Claim 11 (depends on 10) The computer-implemented method of claim 10, wherein a first type of text-based feature is based on an edit distance between respective portions of two listings. ### Claim 12 (depends on 11) The computer-implemented method of claim 11, wherein the edit distance comprises an edit distance between a title text portion of the translated first listing and a title text portion of the second listing. ### Claim 13 (depends on 11) The computer-implemented method of claim 11, wherein a second type of listing text-based feature is based on a length of a respective listing. ### Claim 14 (depends on 13) The computer-implemented method of claim 13, wherein a third type of listing text-based feature is based on a lexical translation overlap between two respective listings. ### Claim 15 (depends on 14) The computer-implemented method of claim 14, wherein a fourth type of listing text-based feature is based on a product category of the respective listing. ### Claim 16 (depends on 15) The computer-implemented method of claim 15, wherein a fifth type of listing text-based feature is based on structural information of an item comprising at least one of: item color and item brand. ### Claim 17 A non-transitory computer-readable medium storing executable instructions thereon that, when executed by a processor, cause the processor to perform operations including: translating a first listing from a first language to a second language, the first listing including at least one image of a first item; providing as input to an encoded neural network model at least one image portion and at least one text portion of the translated first listing and at least one image portion and at least one text portion of a second listing in the second language, the second listing including at least one image of a second item; receiving from the encoded neural network model a first feature vector for the translated first listing and a second feature vector for the second listing, the first and the second feature vectors both include at least one type of image signature feature and at least one type of listing text-based feature; calculating a similarity score based on the first feature vector and the second feature vector, and based on the similarity score at least meeting a similarity score threshold, generating a pairing of the first listing in the first language with the second listing in the second language for inclusion in a database training data of a machine translation system, wherein the pairing is based on a degree of likelihood that the first item and the second item are the same. ### Claim 18 (depends on 17) The non-transitory computer-readable medium of claim 17, wherein the at least one type of image signature feature is based on at least one pixel color of an image in a respective listing and at least one pixel location in the respective listing. ### Claim 19 (depends on 18) The non-transitory computer-readable medium of claim 18, wherein a first type of listing text-based feature is based on an edit distance between respective portions of two listings. ### Claim 20 (depends on 19) The non-transitory computer-readable medium of claim 19, wherein the edit distance comprises an edit distance between a title portion of the translated first listing and a title text portion of the second listing.