# US 8,788,529 B2 · Information sharing between images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,788,529 B2 - Document ID: US8788529B2 - Status: Granted - Kind: B2 - Application number: 11/679,080 - Filed: 2007-02-26 - Published: 2008-08-28 - Granted: 2014-07-22 - Earliest claimed: 2007-02-26 - Assignees: Microsoft Corporation - Applicants: Charles Lawrence Zitnick, III; Richard Lloyd Hughes - Inventors: Charles Lawrence Zitnick, III; Richard Lloyd Hughes - Web page: https://patentgrep.com/patent/US8788529B2 ## CPC - H04L 67/06: specially adapted for file transfer, e.g. file transfer protocol [FTP] - G06F 16/50: of still image data - G06F 16/53: Querying - G06F 16/95: Retrieval from the web - G06Q 10/107: Computer-aided management of electronic mailing [e-mailing] - H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP] - H04L 67/04: specially adapted for terminals or networks with limited capabilities; specially adapted for terminal portability - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR ## Abstract A content-based image retrieval technique for sharing image information between images. In the present image information sharing technique digital photographs containing relevant comments and links to web pages are supplied by a set of authors. A user of the image information sharing technique takes a digital picture of something that interests them, such as, for example, a poster, a product, a painting, and uploads it to an image information sharing server. The image information sharing technique then matches the photo to an image in an image information sharing database. The comments and web links associated with the matched photo are subsequently sent to the user. ## Claims ### Claim 1 A computer-implemented process for obtaining information about an object in an image, comprising: authoring images by appending information about an object in each image to obtain a database of authored images; acquiring an image containing an object; matching the object in the acquired image to an object in an authored image in the database using a content-based image retrieval technique wherein the matching matches the object in the acquired image to an object in an authored image in the database by using the contents of the acquired image in the database and the authored image themselves, rather than relying on image metadata, by using triplets of image features warped into equilateral triangles, comprising: finding features in the acquired image; grouping three of the features into a feature triplet; computing an affine transformation using two neighboring feature positions for each feature in the feature triplet to warp the feature triplet into an equilateral triangle in canonical space; and matching each of the features in the feature triplet of the acquired image with features in a feature triplet of the authored images in the database to identify the object in the authored images; and extracting the appended information about the object that was found in the matched images from the matched authored image; sending the extracted appended information, and not the image in the image database to a client for use in locating additional information from different sources for the object in the acquired image. ### Claim 2 (depends on 1) The computer-implemented process of claim 1 wherein the following actions are repeated to obtain information on one or more additional objects in images: acquiring an image containing an object at a client; matching the object in the acquired image to an object in an authored image in the database using a content-based image retrieval technique comprising using triplets of features warped into equilateral triangles to match the object in the image uploaded to the server to an object in an authored image; and extracting the appended information about the object that was found in the matched image from the matched authored image. ### Claim 3 (depends on 1) The computer-implemented process of claim 1 wherein the client is a cellular phone. ### Claim 4 (depends on 1) The computer-implemented process of claim 1 wherein the appended information comprises comments about an object in an authored image and a website address relevant to an object in the authored image. ### Claim 5 (depends on 1) The computer-implemented process of claim 1 wherein the content-based image retrieval technique is an object instance recognition technique. ### Claim 6 (depends on 1) The computer-implemented process of claim 1 wherein the database resides on a server. ### Claim 7 (depends on 1) The computer-implemented process of claim 1 wherein the database is accessed over a network. ### Claim 8 A computer-implemented process for obtaining information about an object in an image, comprising: acquiring an image containing an object; computing information used for matching the object in the acquired image using information in the image itself vice metadata; querying a database using the computed information used for matching; matching the object in the acquired image to an object in an image in the database using a content-based image retrieval technique with the computed information used for matching wherein the content-based image retrieval technique uses triplets of features wherein each feature triplet is warped into an equilateral triangle in order to match the object in the acquired image to an object in the image in the database, comprising: finding features in the acquired image; grouping three of the features into a feature triplet; computing an affine transformation using two neighboring feature positions for each feature in the feature triplet to warp the feature triplet into an equilateral triangle in canonical space; and matching each of the features in the feature triplet of the acquired image with features in a feature triplet of the images in the database to identify the object in the images in the database; extracting only information about the matched object from the image in the database; and sending only the extracted information, and not the image in the database, to a client. ### Claim 9 (depends on 8) The computer-implemented process of claim 8 wherein the images in the database are appended with information comprising comments about the object; and wherein the comments about the object are received at a client that queried the database. ### Claim 10 (depends on 8) The computer-implemented process of claim 8 wherein the images in the database are appended with a website address for obtaining additional information about the object; and wherein the website address for obtaining additional information about the object are received at a client that queried the database. ### Claim 11 (depends on 8) The computer-implemented process of claim 8 wherein the images in the database are appended with one of: location information about the object; or key words helpful in searching for other information about the image. ### Claim 12 A system for querying a database, comprising: a general purpose computing device; a computer program comprising program modules executable by the general purpose computing device, wherein the computing device is directed by the program modules of the computer program to, acquire an image of an object at a client; and query a database for the object using the acquired image; match the object in the acquired image to an object in an image in the database using an object instance recognition technique that automatically recognizes a particular object from a category of objects of the same type by using feature triplets, each comprising three image features, warped into equilateral triangles using an affine transformation, comprising: finding features in the acquired image; grouping three of the features in the image into a feature triplet; computing an affine transformation using two neighboring feature positions for each feature in the feature triplet to warp the feature triplet into an equilateral triangle in canonical space; and matching each of the features in the feature triplet of the acquired image with features in a feature triplet of the images in the database to identify the object in the images in the database; send information about the object in the matched image from the image in the database to a client. ### Claim 13 (depends on 12) The system of claim 12, wherein the images in the database are appended with a website link by an author with a certain level of privilege. ### Claim 14 (depends on 13) The system of claim 13 further comprising a module for the client to compute an image signature and wherein the image signature is used to query the database and match the acquired image to an image in the database. ### Claim 15 (depends on 14) The system of claim 14 wherein a user must subscribe to a service to use the system and wherein only users that subscribe to the service to use the system can upload images to the database to extract information about the object from the image in the database. ### Claim 16 (depends on 12) The system of claim 12 wherein the database contains images and wherein the images and information that can be extracted are uploaded in bulk using a bulk loading module. ### Claim 17 (depends on 12) The system of claim 12 wherein the client is a cellular phone that can query the database by pushing a single control device on the cellular phone to send an image with which to query the database.