Dataset relevance estimation in storage systems

A computer system for managing datasets, wherein a subset of the datasets is labeled with respect to their relevance, so as to be associated with respective relevance values, the computer system comprising:

• one or more computer devices each having one or more processors and one or more tangible storage devices;
• and
• a program embodied on at least one of the one or more storage devices, the program having a plurality of program instructions for execution by the one or more processors, the program instructions comprising instructions for:
• defining supersets of the datasets, based on metadata available for the datasets;
• associating each of the datasets to at least one of the defined supersets by comparing metadata available for the each of the datasets with metadata used to define at least one of the supersets;
• determining, for each unlabeled dataset of the datasets, a respective probability distribution over a set of relevance values, to obtain a corresponding relevance value, wherein the respective probability distribution is computed based on distances, in terms of metadata values, between the each unlabeled dataset and the labeled datasets, and wherein the respective probability distribution is computed based on a sum of weighted initial probability distributions associated with other datasets of the datasets, and wherein an initial probability distribution of a given dataset is determined as a probability distribution over the relevance values, according to a relevance value associated with that given dataset, and wherein the sum of weighted initial probability distributions associated with the other datasets of the datasets is weighted according to inverse distances between the each unlabeled dataset and the other datasets, and wherein the respective probability distribution, as associated with the each unlabeled dataset, is determined based on a sum of weighted initial probability distributions associated with other datasets of the at least one of the supersets with which the each unlabeled dataset is associated, and wherein the respective probability distribution, as associated with the each unlabeled dataset, is determined based on several sums of weighted initial probability distributions associated with other datasets of the at least one of the supersets, as respectively obtained for several supersets of interest, with which the each unlabeled dataset is associated, and wherein distances between datasets of the each of the supersets of interest are computed according to a distance metric associated with each of the supersets of interest, respectively, and wherein the distance metric is determined based on metadata fields of the datasets of the each of the supersets of interest, and wherein the distance metric is determined based on common metadata fields shared by the datasets of the each of the supersets of interest, whereby a distance between two datasets f1, f2 is computed as a symmetric divergence measure between two conditional probability distributions P(R|k, vk(f1)) and P(R|k, vk(f2)), respectively pertaining to the two datasets f1, f2, and wherein a conditional probability distribution P(R|(k, vk(f)) is an empirical distribution of probability, obtained from the labeled datasets, of observing a given relevance value R when a file f has metadata values vk(f) for a set of metadata fields k;
• and
• managing the datasets in the computer system based on their associated relevance values.

A computer program product for managing datasets in a storage system, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions being executable by a computerized system to cause to:

• define supersets of the datasets, based on metadata available for the datasets;
• associate each of the datasets to at least one of the defined supersets by comparing metadata available for the each of the datasets with metadata used to define at least one of the supersets;
• determine, for each unlabeled dataset of the datasets, a respective probability distribution over a set of relevance values, to obtain a corresponding relevance value, wherein the respective probability distribution is computed based on distances, in terms of metadata values, between the each unlabeled dataset and the labeled datasets, and wherein the respective probability distribution is computed based on a sum of weighted initial probability distributions associated with other datasets of the datasets, and wherein an initial probability distribution of a given dataset is determined as a probability distribution over the relevance values, according to a relevance value associated with that given dataset, and wherein the sum of weighted initial probability distributions associated with the other datasets of the datasets is weighted according to inverse distances between the each unlabeled dataset and the other datasets, and wherein the respective probability distribution, as associated with the each unlabeled dataset, is determined based on a sum of weighted initial probability distributions associated with other datasets of the at least one of the supersets with which the each unlabeled dataset is associated, and wherein the respective probability distribution, as associated with the each unlabeled dataset, is determined based on several sums of weighted initial probability distributions associated with other datasets of the at least one of the supersets, as respectively obtained for several supersets of interest, with which the each unlabeled dataset is associated, and wherein distances between datasets of the each of the supersets of interest are computed according to a distance metric associated with each of the supersets of interest, respectively, and wherein the distance metric is determined based on metadata fields of the datasets of the each of the supersets of interest, and wherein the distance metric is determined based on common metadata fields shared by the datasets of the each of the supersets of interest, whereby a distance between two datasets f1, f2 is computed as a symmetric divergence measure between two conditional probability distributions P(R|k, vk(f1)) and P(R|k, vk(f2)), respectively pertaining to the two datasets f1, f2, and wherein a conditional probability distribution P(R|(k, vk(f)) is an empirical distribution of probability, obtained from the labeled datasets, of observing a given relevance value R when a file f has metadata values vk(f) for a set of metadata fields k;
• and
• manage labeled datasets in the storage system based on their determined relevance values.