US 5,852,823 AGranted

Image classification and retrieval system using a query-by-example paradigm

AssigneeMicrosoft Corporation

InventorJeremy S. De Bonet

Life of this patent

Estimating the term…

Loading patent…

Abstract

Apparatus and an accompanying method for generating a semantically based, linguistically searchable, numeric descriptor of a pre-defined group of input images and which is particularly useful in a system for automatically classifying individual images, on a numerical basis, in, e.g., an image database, and, through a query-by-example paradigm, retrieving a desired image(s) therefrom. Specifically, a signature is computed for each image in a set using multi-level iterative convolution filtering, with pixel values supplied as input to each filtering level being separately convolved with each one of a set of predefined Gaussian kernels. Average and variance vectors, as collective numeric descriptors of all the images in the set, are separately computed across corresponding elements in all the image signatures for the set. A linguistic term, semantically descriptive of all the images in the set, is associated with the numeric descriptors of this set and, with this set, is stored in a database. For image retrieval, the descriptor for any set is accessed by a textual search through the database using the appropriate linguistic term. The descriptor is then compared against accessed signatures for other images in the database in order to retrieve a image, among those stored in the database, that is the most similar to those in the set associated with the descriptor.

Claims96

Dependent claims

A system, responsive to a query image set, for retrieving, from a stored database, a desired image that is visually similar to the image set, said set containing at least one query image, said system comprising:

an image database having stored images therein;
a signature generator, responsive to an input image and having three successive convolution filters, for producing a signature for the input image, the signature containing a numeric measure of each of a plurality of pre-defined characteristics of the input image;
wherein the signature generator, in response to the query image set, generates a corresponding signature for each image in the query image set, and, in response to a test image accessed from the stored database, generates a corresponding test signature;
a statistics generator, responsive to the signature for each image in the query image set, for providing a separate pre-defined statistical measure, for each of the pre-defined characteristics, that represents variability of said each pre-defined characteristic across all images in the query image set;
a image database manager, operative in conjunction with the image database, for retrieving successive ones of a plurality of stored images from the database and for routing each one of the plurality of successive images as the test image to the signature generator so as to generate a plurality of corresponding test signatures;
a comparator for comparing, across all of the pre-defined characteristics, the statistical measure for each of the pre-defined characteristics against a corresponding numeric measure in each one of the test signatures so as to yield a similarity measure, the similarity measure for said one of the test images being reflective of a degree of visual similarity between the one test image and the query image set;
and
a manager, responsive to the similarity measure for each of the test images and operative in conjunction with the image database manager, for selecting at least one of the test images having a highest relative similarity measure as a retrieved image and providing the retrieved image as output.

5 more elements

A system, responsive to a term chosen by a user, for retrieving, from a stored database, a desired image that is visually similar to images in a set associated with the term, said set containing at least one query image, said system comprising:

a database having stored therein a plurality of images and a signature associated with each of said images, wherein each of the signatures contains a numeric measure of each of a plurality of pre-defined characteristics of an associated image stored in the database so as to form a plurality of numeric measures in the signature for the associated image;
a manager for accessing, in response to a term associated with the pre-defined set of images, from the database an associated set of signatures for a query image set, and for accessing individual ones of the signatures for a plurality of other images in the database so as to define a plurality of test images and a plurality of corresponding test signatures therefor;
a statistics generator, responsive to the set of signatures for the query image set, for providing separate statistical measures, for each of the pre-defined characteristics, that separately represent average and variability measures of said each pre-defined characteristic across all images in the query image set so as to form separate corresponding pluralities of average and variability measures for all of said characteristics;
a comparison engine for determining, for each of the test signatures, a similarity measure reflective of a degree of visual similarity between a corresponding one of the test images and the query image set, wherein the similarity measure is a function, taken across all of the pre-defined characteristics, of separate corresponding ones of the pluralities of the average and variance measures for the query image set and a separate corresponding one of the plurality of said numeric measures for said each test signature;
and
wherein the manager, in response to the similarity measure for each of the test images, selects at least one of the test images having a highest relative similarity measure as a retrieved image and provides the retrieved image as output.

3 more elements

a database having stored therein a plurality of images and a signature associated with each of said images, wherein each of the signatures contains a numeric measure of each of a plurality of pre-defined characteristics of an associated image stored in the database so as to form a plurality of numeric measures in the signature for the associated image;
a manager for accessing, in response to a term associated with the pre-defined set of query images, from the database an associated set of statistical measures descriptive of the query image set, and for accessing individual ones of the signatures for a plurality of other images in the database so as to define a plurality of test images and a plurality of corresponding test signatures therefor, wherein the statistical measures separately represent average and variability measures of a corresponding one of the pre-defined characteristics across all images in the query image set so as to form separate corresponding pluralities of average and variability measures for all of said characteristics;
a comparison engine for determining, for each of the test signatures, a similarity measure reflective of a degree of visual similarity between a corresponding one of the test images and the query image set, wherein the similarity measure is a function, taken across all of the pre-defined characteristics, of separate corresponding ones of the pluralities of the average and variance measures for the query image set and a separate corresponding one of the plurality of said numeric measures for said each test signature;
and
wherein the manager, in response to the similarity measure for each of the test images, selects at least one of the test images having a highest relative similarity measure as a retrieved image and provides the retrieved image as output.

2 more elements

A method, responsive to a query image set, for retrieving, from a stored image repository, a desired image that is visually similar to the image set, said set containing at least one query image, the method comprising the steps of:

generating, through three successive convolution filters responsive to an input image, a signature for the input image, the signature containing a numeric measure of each of a plurality of pre-defined characteristics of the input image;
wherein the signature generating step comprises the steps of:
generating, in response to the query image set, a corresponding signature for each image in the query image set, and
generating, in response to a test image accessed from the stored image repository, a corresponding test signature;
providing, in response to the signature for each image in the stored image repository, a separate pre-defined statistical measure, for each of the pre-defined characteristics, that represents variability of said each pre-defined characteristic across all images in the query image set;
retrieving successive ones of a plurality of stored images from the stored image repository and applying each one of the plurality of successive images as the test image to the signature generating step so as to generate a plurality of corresponding test signatures;
comparing, across all of the pre-defined characteristics, the statistical measure for each of the pre-defined characteristics against a corresponding numeric measure in each one of the test signatures so as to yield a similarity measure, the similarity measure for said one of the test images being reflective of a degree of visual similarity between the one test image and the query image set;
selecting, in response to the similarity measure for each of the test images, at least one of the test images having a highest relative similarity measure as a retrieved image;
and
providing the retrieved image as output.

7 more elements

A method, responsive to a term chosen by a user, for retrieving, from a stored database, a desired image that is visually similar to images in a set associated with the term, said set containing at least one query image, the method comprising the steps of:

accessing, in response to a term associated with the pre-defined set of images, from a database an associated set of signatures for a query image set, and accessing individual signatures from the database for a plurality of other images in the database so as to define a plurality of test images and a plurality of corresponding test signatures therefor, the database storing a plurality of images and a signature associated with each of said images, wherein each of the signatures contains a numeric measure of each of a plurality of pre-defined characteristics of an associated image stored in the database so as to form a plurality of numeric measures in the signature for the associated image;
providing, in response to the set of signatures for the query image set, separate statistical measures, for each of the pre-defined characteristics, that separately represent average and variability measures of said each pre-defined characteristic across all images in the query image set so as to form separate corresponding pluralities of average and variability measures for all of said characteristics;
determining, for each of the test signatures, a similarity measure reflective of a degree of visual similarity between a corresponding one of the test images and the query image set, wherein the similarity measure is a function, taken across all of the pre-defined characteristics, of separate corresponding ones of the pluralities of the average and variance measures for the query image set and a separate corresponding one of the plurality of said numeric measures for said each test signature;
selecting, in response to the similarity measure for each of the test images, at least one of the test images having a highest relative similarity measure as a retrieved image;
and
providing the retrieved image as output.

3 more elements

accessing, in response to a term associated with the pre-defined set of query images, from a database an associated set of statistical measures descriptive of a set of pre-defined characteristics of the query image set, and accessing individual ones of the signatures from the database for a plurality of other images in the database so as to define a plurality of test images and a plurality of corresponding test signatures therefor, the statistical measures separately representing average and variability measures of a corresponding one of said pre-defined characteristics across all images in the query image set so as to form separate corresponding pluralities of average and variability measures for all of said characteristics, wherein the database stores a plurality of images and a signature associated with each of said images, each of the signatures containing a numeric measure of each one of the pre-defined characteristics for an associated image stored in the database so as to form a plurality of numeric measures in the signature for the associated image;
determining, for each of the test signatures, a similarity measure reflective of a degree of visual similarity between a corresponding one of the test images and the query image set, wherein the similarity measure is a function, taken across all of the pre-defined characteristics, of separate corresponding ones of the pluralities of the average and variance measures for the query image set and a separate corresponding one of the plurality of said numeric measures for said each test signature;
selecting, in response to the similarity measure for each of the test images, at least one of the test images having a highest relative similarity measure as a retrieved image;
and
providing the retrieved image as output.

2 more elements

Dependencies are inferred from the claim text.