# US 9,870,516 B2 · Hand-drawn sketch recognition > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,870,516 B2 - Document ID: US9870516B2 - Status: Granted - Kind: B2 - Application number: 14/847,834 - Filed: 2015-09-08 - Published: 2015-12-31 - Granted: 2018-01-16 - Earliest claimed: 2013-05-03 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Changhu Wang; Lei Zhang; Zhenbang Sun - Web page: https://patentgrep.com/patent/US9870516B2 ## CPC - G06F 16/5854: {using shape and object relationship} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Some examples of a sketch-based image recognition system may generate a model for identifying a subject of a sketch. The model is formed from a plurality of images having visual features similar to the visual features of the sketch. The model may include object topics representative of categories which may correspond to the subject of the sketch and shape topics representative of the visual features of the sketch. ## Claims ### Claim 1 A method comprising: receiving, from a computing device over a network, a sketch; searching a database of images using the sketch as a query to identify a set of images, an individual image of the set of images including a shape feature similar to at least one shape feature of the sketch; determining, based at least in part on the set of images, at least one object topic; and generating, based at least in part on the set of images, a model for identifying a subject of the sketch, the model including: the at least one object topic, the at least one object topic being representative of a category that is descriptive of the subject of the sketch; and at least one shape topic associated with the at least one object topic, the at least one shape topic representative of a shape feature of the sketch. ### Claim 2 (depends on 1) The method of claim 1, wherein the individual image of the set of images further includes at least one text-based label, and wherein the method further comprises: generating a word distribution for the at least one object topic from text-based labels of at least a portion of the set of images; and identifying a word descriptive of the subject of the sketch from the word distribution. ### Claim 3 (depends on 2) The method of claim 2, wherein the word distribution includes at least a first word and a second word, and wherein the method further comprises: determining a first rank for the first word based at least in part on a first probability that the first word is descriptive of the subject of the sketch; determining a second rank for the second word based at least in part on a second probability that the second word is descriptive of the subject of the sketch; and generating a ranking for the first word and the second word based at least in part on the first rank and the second rank, and wherein identifying the word descriptive of the subject of the sketch is based at least in part on the ranking. ### Claim 4 (depends on 1) The method of claim 1, wherein the at least one object topic includes a first object topic and a second object topic, and wherein the method further comprises: determining a first probability that a shape topic of the at least one shape topic belongs to the first object topic; determining a second probability that the shape topic belongs to the second object topic; and associating the shape topic with at least one of the first object topic or the second object topic based at least in part on the first probability and the second probability. ### Claim 5 (depends on 4) The method of claim 4, wherein: the first probability is greater than the second probability; and associating the shape topic with the at least one of the first object topic or the second object topic comprises associating the shape topic with the first object topic based at least in part on the first probability for the first shape topic being greater than the second probability for the second object topic. ### Claim 6 (depends on 1) The method of claim 1, further comprising: filtering the set of images into a set of filtered images by removing images that include more than one subject, and wherein generating the model based at least in part on the set of images comprises generating the model based at least in part on the set of filtered images. ### Claim 7 (depends on 1) The method of claim 1, further comprising: identifying, for at least an image of the set of images, a shape feature of the image; and associating the image with a shape topic of the at least one shape topic based at least in part on the shape feature of the image. ### Claim 8 A computing device comprising: one or more input interfaces for receiving a sketch; one or more processors; and a computer-readable storage media storing instructions, which when executed by the one or more processors, cause the one or more processors to perform operations comprising: performing an image-based search of a database of images using the sketch as a search query; identifying a set of images, an individual image of the set of images including a shape feature similar to a shape feature of the sketch; determining an object topic based at least in part on the set of images; and generating, based at least in part on the set of images, a model for identifying a subject associated with the sketch, the model including: the object topic, the object topic representative of a category for the subject that is associated with the sketch; and a shape topic associated with the object topic, the shape topic representative of the shape feature of the sketch. ### Claim 9 (depends on 8) The computing device of claim 8, wherein the individual image of the set of images is associated with at least one text-based label, and wherein the operations further comprise: generating a word distribution for the object topic from text-based labels associated with at least a portion of the set of images; and identifying a word descriptive of the subject associated with the sketch from the word distribution. ### Claim 10 (depends on 9) The computing device of claim 9, wherein the word distribution includes at least a first word and a second word, and wherein the operations further comprise: determining a first rank for the first word based at least in part on a first probability that the first word is descriptive of the subject associated with the sketch; determining a second rank for the second word based at least in part on a second probability that the second word is descriptive of the subject associated with the sketch; and generating a ranking for the first word and the second word based at least in part on the first rank and the second rank, and wherein identifying the word descriptive of the subject of the sketch is based at least in part on the ranking. ### Claim 11 (depends on 8) The computing device of claim 8, wherein the model further includes an additional object topic, and wherein the operations further comprise: determining a first probability that the shape topic is associated with the object topic; determining a second probability that the shape topic is associated with the additional object topic; and associating the shape topic with the object topic based at least in part on the first probability and the second probability. ### Claim 12 (depends on 8) The computing device of claim 8, the operations further comprising: filtering the set of images into a set of filtered images by removing images that include more than one subject, and wherein generating the model based at least in part on the set of images comprises generating the model based at least in part on the set of filtered images. ### Claim 13 (depends on 8) The computing device of claim 8, the operations further comprising: identifying, for at least an image of the set of images, a shape feature of the image; and associating the image with the shape topic based at least in part on the shape feature of the image. ### Claim 14 A computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: searching a database of images using a sketch as a query to identify a set of images, an individual image of the set of images including a shape feature similar to at least one shape feature of the sketch; determining, based at least in part on the set of images, at least one object topic; and generating, based at least in part on the set of images, a model for identifying a subject of the sketch, the model including: the at least one object topic, the at least one object topic being representative of a category that is descriptive of the subject of the sketch; and at least one shape topic associated with the at least one object topic, the at least one shape topic representative of a shape feature of the sketch. ### Claim 15 (depends on 14) The computer-readable storage media of claim 14, wherein the individual image of the set of images further includes at least one text-based label, and wherein the operations further comprise: generating a word distribution for the at least one object topic from text-based labels of at least a portion the set of images; and identifying a word descriptive of the subject of the sketch from the word distribution. ### Claim 16 (depends on 15) The computer-readable storage media of claim 15, wherein the word distribution includes at least a first word and a second word, and wherein the operations further comprise: determining a first rank for the first word based at least in part on a first probability that the first word is descriptive of the subject of the sketch; determining a second rank for the second word based at least in part on a second probability that the second word is descriptive of the subject of the sketch; and generating a ranking for the first word and the second word based at least in part on the first rank and the second rank, and wherein identifying the word descriptive of the subject of the sketch is based at least in part on the ranking. ### Claim 17 (depends on 14) The computer-readable storage media of claim 14, wherein the at least one object topic includes a first object topic and a second object topic, and wherein the operations further comprise: determining a first probability that a shape topic of the at least one shape topic belongs to the first object topic; determining a second probability that the shape topic belongs to the second object topic; and associating the shape topic with at least one of the first object topic or the second object topic based at least in part on the first probability and the second probability. ### Claim 18 (depends on 17) The computer-readable storage media of claim 17, wherein: the first probability is greater than the second probability; and associating the shape topic with the at least one of the first object topic or the second object topic comprises associating the shape topic with the first object topic based at least in part on the first probability for the first shape topic being greater than the second probability for the second object topic. ### Claim 19 (depends on 14) The computer-readable storage media of claim 14, the operations further comprising: identifying, for at least an image of the set of images, a shape feature of the image; and associating the image with a shape topic of the at least one shape topic based at least in part on the shape feature of the image. ### Claim 20 (depends on 1) The method of claim 1, further comprising identifying a word descriptive of the subject of the sketch using the model.