# US 12,216,705 B2 · System and method for attribute-based visual search over a computer communication network > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,216,705 B2 - Document ID: US12216705B2 - Status: Granted - Kind: B2 - Application number: 17/404,367 - Filed: 2021-08-17 - Published: 2021-12-09 - Granted: 2025-02-04 - Earliest claimed: 2018-05-21 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Li Huang; Meenaz Merchant; Houdong Hu; Arun Sacheti - Web page: https://patentgrep.com/patent/US12216705B2 ## CPC - G06F 16/583: using metadata automatically derived from the content - G06F 16/24578: {using ranking} - G06F 16/248: Presentation of query results - G06F 16/51: Indexing; Data structures therefor; Storage structures - G06F 16/532: Query formulation, e.g. graphical querying - G06F 16/56: having vectorial format - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06F 18/22: Matching criteria, e.g. proximity measures - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A visual search system includes a computing device, where the computing device includes an image processing engine for generating a feature vector representing a user-selected object in an image. The computing device also includes, an object detection engine for locating one or more objects in the image and for determining a category of a user-selected object from objects in the image, where the object detection engine uses the category to generate a plurality of attributes for the user-selected object. The computing device further includes a product data store for storing a plurality of tables storing one or more attributes associated with a category of the user-selected object. The computing device additionally includes an attribute generation engine for generating a plurality of attribute options and an attribute matching engine for comparing attributes and attribute options of the user-selected object with attributes and attribute options of visually similar products and images. ## Claims ### Claim 1 A computing system, comprising: a hardware processor; and at least one memory device storing instructions that, when executed by the hardware processor, cause the hardware processor to perform acts comprising: obtaining an image, wherein the image is displayed to a user; in response to receiving an indication that an object in the image has been selected by the user, generating a feature vector that represents the object; determining a category of the object based upon the feature vector; retrieving a plurality of attributes based upon the category of the object, wherein the plurality of attributes comprise one or more first attributes and one or more second attributes, wherein the one or more first attributes are indicative of a feature of the object and the one or more second attributes are indicative of a feature of at least one other object in the category but are not indicative of the object; generating a plurality of search results based upon the feature vector and the plurality of attributes; and causing the plurality of search results to be displayed within a graphical user interface (GUI). ### Claim 2 (depends on 1) The computing system of claim 1, wherein the plurality of search results comprise images that include objects belonging to the category of the object displayed in the image. ### Claim 3 (depends on 2) The computing system of claim 2, wherein the plurality of search results further comprise descriptions of each of the objects. ### Claim 4 (depends on 1) The computing system of claim 1, wherein the plurality of search results include a first image that includes a first object, wherein the first object has the at least one of the one or more second attributes in place of the one or more first attributes. ### Claim 5 (depends on 1) The computing system of claim 1, wherein the image comprises a plurality of objects, the acts further comprising: prior to generating the feature vector, identifying the object from amongst the plurality of objects in the image. ### Claim 6 (depends on 1) The computing system of claim 1, wherein a second object is in the image, wherein generating the feature vector that represents the object in the image comprises: cropping the image to generate a query image, wherein the query image includes the object, wherein the query image excludes the second object, wherein the feature vector is generated based upon the query image. ### Claim 7 (depends on 1) The computing system of claim 1, the acts further comprising: obtaining user preferences of the user; and subsequent to generating the plurality of search results and prior to causing the plurality of search results to be displayed within the GUI, assigning rankings to the plurality of search results based upon the user preferences, wherein the plurality of search results are displayed in accordance with the rankings. ### Claim 8 (depends on 1) The computing system of claim 1, wherein generating the plurality of search results based upon the feature vector and the plurality of attributes comprises: identifying a database from amongst a plurality of databases based upon the category of the object; and executing a search over the database based upon the feature vector and the plurality of attributes, wherein the plurality of search results are generated from the search. ### Claim 9 (depends on 1) The computing system of claim 1, wherein the image is obtained from a computing device operated by the user, wherein the GUI is presented on a display of the computing device. ### Claim 10 (depends on 1) The computing system of claim 1, wherein the image is obtained from a server computing device. ### Claim 11 (depends on 1) The computing system of claim 1, wherein retrieving the plurality of attributes comprises executing a rule assigned to the category. ### Claim 12 A method executed by a processor of a computing system, the method comprising: obtaining an image, wherein the image is displayed on a display of a computing device operated by a user; identifying a plurality of objects in the image; upon identifying the plurality of objects in the image, determining respective categories of each of the plurality of objects in the image; in response to receiving an indication that an object in the plurality of objects in the image has been selected by the user, generating a feature vector that represents the object; retrieving a plurality of attributes based upon a category of the object, wherein the plurality of attributes comprise one or more first attributes and one or more second attributes, wherein the one or more first attributes are indicative of a feature of the object and the one or more second attributes are indicative of a feature of at least one other object in the category but are not indicative of the object; generating a plurality of search results based upon the feature vector and the plurality of attributes; and causing the plurality of search results to be displayed within a graphical user interface (GUI) shown on the display. ### Claim 13 (depends on 12) The method of claim 12, wherein each of the plurality of objects in the image is marked with a respective icon, wherein the object is selected when an icon assigned to the object is selected by the user. ### Claim 14 (depends on 12) The method of claim 12, wherein generating the plurality of search results based upon the feature vector and the plurality of attributes comprises: generating a visual-word-quantized representation of the object based upon the feature vector; generating a product quantized representation of object based upon the visual-word-quantized representation of the object; and executing a search over at least one database based upon the product quantized representation of the object and the attribute set, wherein the plurality of search results are generated based upon the search. ### Claim 15 (depends on 12) The method of claim 12, further comprising: obtaining user preferences of the user; and subsequent to generating the plurality of search results and prior to causing the plurality of search results to be displayed within the GUI, assigning rankings to the plurality of search results based upon the user preferences, wherein the plurality of search results are displayed in accordance with the rankings. ### Claim 16 (depends on 12) The method of claim 12, wherein generating the feature vector that represents the object in the image comprises: cropping the image to generate a query image, wherein the query image includes the object, wherein the feature vector is generated based upon the query image. ### Claim 17 (depends on 12) The method of claim 12, wherein the plurality of search results comprise images that include objects that belong to the category of the object displayed in the image. ### Claim 18 A non-transitory computer-readable storage medium comprising instructions that, when executed by a processor of a computing system, cause the processor to perform acts comprising: obtaining an image, wherein the image is displayed within a graphical user interface (GUI) shown on a display of a computing device; in response to receiving an indication that an object in the image has been selected by a user of the computing device, generating a feature vector that represents the object; determining a category of the object based upon the feature vector; retrieving a plurality of attributes based upon the category of the object, wherein the plurality of attributes comprise one or more first attributes and one or more second attributes, wherein the one or more first attributes are indicative of a feature of the object and the one or more second attributes are indicative of at least one other object in the category but are not indicative of the object; generating a plurality of search results based upon the feature vector and the plurality of attributes; and transmitting the plurality of search results to the computing device, wherein the plurality of search results are displayed within the GUI. ### Claim 19 (depends on 18) The non-transitory computer-readable storage medium of claim 18, wherein the image is displayed at a first region of the GUI, wherein the plurality of search results are displayed at a second region of the GUI. ### Claim 20 (depends on 18) The non-transitory computer-readable storage medium of claim 18, wherein the at least one additional attribute belongs to the category.