Multi-modal fusion in content-based retrieval

A method for constructing and executing a fusion search for searching a multimedia database, said method comprising the steps of:

• forming a plurality of searches that produce intermediate search results, wherein a user builds a fusion search constructed as a text string interactively by sequentially choosing among descriptors and data sources and by selecting from various combining and score aggregation functions to fuse results of each individual search technique, and wherein each of the plurality of searches utilizes a different search technique, and wherein said search techniques consist of:
• content-based retrieval which allows searching and matching based on perceptual similarity of multimedia content, wherein a user provides at least one example of multimedia content and result similarity is based on distance in feature space;
• and
• model-based retrieval which allows searching based on automatically extracted concept labels produced by statistical models, wherein a user provides desired label text and matches are ranked using D=1−C, with distance D derived from a confidence score C associated with each automatically assigned label;
• interactively integrating the plurality of searches to selectively direct each search to multiple individual search tools and provide desired matches from multiple disparate data sources based on the search technique and content type;
• selecting at least one fusion method to be used in combining a plurality of the intermediate search results of said searches;
• combining the plurality of intermediate search results of said searches interactively based on the intermediate results of the searches until a final results list is achieved;
• returning the plurality of intermediate search results to a user;
• and
• returning a fused result the final results list to a user;
• wherein the plurality of intermediate search results of said searches include multimedia data;
• and
• wherein combining the intermediate search results interactively further comprises executing a cascade of sequential operations, the cascade of sequential operations comprising conducting the plurality of searches that produce the intermediate search results, normalizing the intermediate search results, fusing the intermediate search results to produce fused results, and using the fused results as input for further normalization and fusion with subsequent intermediate results of a subsequent search until the final results list is achieved.

A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for constructing and executing a fusion search for searching a multimedia database, said method comprising the steps of:

• forming a plurality of searches that produce intermediate search results, wherein a user builds a fusion search constructed as a text string interactively by sequentially choosing among descriptors and data sources and by selecting from various combining and score aggregation functions to fuse results of each individual search technique, and wherein each of the plurality of searches utilizes a different search technique, and wherein said search techniques consist of:
• content-based retrieval which allows searching and matching based on perceptual similarity of multimedia content, wherein a user provides at least one example of multimedia content and result similarity is based on distance in feature space;
• and
• model-based retrieval which allows searching based on automatically extracted concept labels produced by statistical models, wherein a user provides desired label text and matches are ranked using D=1−C, with distance D derived from a confidence score C associated with each automatically assigned label;
• interactively integrating the plurality of searches to selectively direct each search to multiple individual search tools and provide desired matches from multiple disparate data sources based on the search technique and content type;
• selecting at least one fusion method to be used in combining a plurality of the intermediate search results of said searches;
• combining the plurality of intermediate search results of said searches interactively based on the intermediate results of the searches until the final results list is achieved;
• returning the plurality of intermediate search results to a user;
• and
• returning a fused result the final results list to a user;
• wherein the plurality of intermediate search results of said searches includes include multimedia data;
• and
• wherein combining the intermediate search results interactively further comprises executing a cascade of sequential operations, the cascade of sequential operations comprising conducting the plurality of searches that produce the intermediate search results, normalizing the intermediate search results, fusing the intermediate search results to produce fused results, and using the fused results as input for further normalization and fusion with subsequent intermediate results of a subsequent search until the final results list is achieved.