System and method for handling the confounding effect of document length on vector-based similarity scores

A computer-implemented method of generating vector-based similarity scores in text document comparisons considering document length, comprising:

• computing a mean of a number of word types of two text documents to be compared;
• determining a similarity score with a vector-based similarity model, wherein the vector-based similarity model is a Content-Vector Analysis model and wherein a normalization slope parameter has a value of 5;
• performing pivoted document length normalization on the similarity score using the mean of the number of word types of the two text documents as a normalization affected by both text documents and using the normalization slope parameter;
• and
• outputting a normalized similarity score.

A computer system for generating vector-based similarity scores in text document comparisons considering document length, comprising:

• a computer programmed with instructions that, when executed, cause the computer to execute steps comprising:
• computing a mean of a number of word types of two text documents;
• determining a similarity score with a vector-based similarity model, wherein the vector-based similarity model is a Random Indexing model and wherein a normalization slope parameter has a value of 10;
• performing pivoted document length normalization on the similarity score using the mean of the number of word types of the two text documents as a normalization affected by both text documents and using the normalization slope parameter;
• and
• outputting a normalized similarity score.

A computer system for generating vector-based similarity scores in text document comparisons considering document length, comprising:

• a computer programmed with instructions that, when executed, cause the computer to execute steps comprising:
• computing a mean of a number of word types of two text documents to be compared;
• determining a similarity score with a vector-based similarity model, wherein the vector-based similarity model is a Content-Vector Analysis model and wherein a normalization slope parameter has a value of 5;
• performing pivoted document length normalization on the similarity score using the mean of the number of word types of the two text documents as a normalization affected by both text documents and using the normalization slope parameter;
• and
• outputting a normalized similarity score.

An article of manufacture comprising a non-transitory computer-readable storage medium for causing a computer to generate vector-based similarity scores in text document comparisons considering document length, said computer readable medium including programming instructions that, when executed, cause the computer to execute steps comprising:

• computing a mean of a number of word types of two text documents;
• determining a similarity score with a vector-based similarity model, wherein the vector-based similarity model is a Random Indexing model and wherein a normalization slope parameter has a value of 10;
• performing pivoted document length normalization on the similarity score using the mean of the number of word types of the two text documents as a normalization affected by both text documents and using the normalization slope parameter;
• and
• outputting a normalized similarity score.

An article of manufacture comprising a non-transitory computer-readable storage medium for causing a computer to generate vector-based similarity scores in text document comparisons considering document length, said computer readable medium including programming instructions that, when executed, cause the computer to execute steps comprising:

• computing a mean of a number of word types of two text documents to be compared;
• determining a similarity score with a vector-based similarity model, wherein the vector-based similarity model is a Content-Vector Analysis model and wherein a normalization slope parameter has a value of 5;
• performing pivoted document length normalization on the similarity score using the mean of the number of word types of the two text documents as a normalization affected by both text documents and using the normalization slope parameter;
• and
• outputting a normalized similarity score.