Determining similarity between documents

A data processing apparatus comprising a processor adapted to perform the steps of:

• identifying terms within each digital work of a plurality of digital works, wherein the terms are words and/or phrases;
• determining a number of times that the identified terms occur within each digital work of the plurality of digital works;
• generating a fingerprint for each digital work of the plurality of digital works, the generated fingerprint based on the identified terms and the number of times that the identified terms occur within each digital work;
• using a neural network to find an encoding function, g, that encodes a higher dimensionality space, x, of each fingerprint into a lower dimensionality space, y;
• applying the encoding function to each fingerprint of the plurality of digital works to reduce the dimensionality of each fingerprint;
• determining a similarity between a first fingerprint and one or more dimensionality reduced fingerprints;
• and
• generating an output of data indicating digital works having fingerprints with a similarity to a first dimensionality reduced fingerprint exceeding a threshold, wherein the output is generated in response to receiving a query,
• wherein the similarity between two or more dimensionality reduced fingerprints is determined based on a scoring function sim(.), as:
• sim
• ⁡
• (
• f
• ⁢
• 1
• ,
• f
• ⁢
• 2
• )
• =
• ∑
• ❘
• "\[LeftBracketingBar]"
• f
• ⁢
• 1
• ⋂
• f
• ⁢
• 2
• ❘
• "\[RightBracketingBar]"
• f
• ⁢
• 1
• i
• ∑
• k
• =
• 1
• N
• f
• ⁢
• 1
• k
• ⁢
• f
• ⁢
• 2
• j
• ∑
• k
• =
• 1
• N
• f
• ⁢
• 2
• k
• ⁢
• ∀
• i
• ≡
• j
• where f1 and f2 are dimensionality reduced fingerprints to compare and I and j are terms within the dimensionality reduced fingerprints.

A non-transitory computer-readable medium comprising instructions, which when executed by a computer, cause the computer to carry out the steps of:

• identifying terms within each digital work of a plurality of digital works, wherein the terms are words and/or phrases;
• determining a number of times that the identified terms occur within each digital work of the plurality of digital works;
• generating a fingerprint for each digital work of the plurality of digital works, the generated fingerprint based on the identified terms and the number of times that the identified terms occur within each digital work;
• using a neural network to find an encoding function, g, that encodes a higher dimensionality space, x, of each fingerprint into a lower dimensionality space, y;
• applying the encoding function to each fingerprint of the plurality of digital works to reduce the dimensionality of each fingerprint;
• determining a similarity between a first fingerprint and one or more dimensionality reduced fingerprints;
• and
• generating an output of data indicating digital works having fingerprints with a similarity to a first dimensionality reduced fingerprint exceeding a threshold, wherein the output is generated in response to receiving a query,
• wherein the similarity between two or more dimensionality reduced fingerprints is determined based on a scoring function sim(.), as:
• sim
• ⁡
• (
• f
• ⁢
• 1
• ,
• f
• ⁢
• 2
• )
• =
• ∑
• ❘
• "\[LeftBracketingBar]"
• f
• ⁢
• 1
• ⋂
• f
• ⁢
• 2
• ❘
• "\[RightBracketingBar]"
• f
• ⁢
• 1
• i
• ∑
• k
• =
• 1
• N
• f
• ⁢
• 1
• k
• ⁢
• f
• ⁢
• 2
• j
• ∑
• k
• =
• 1
• N
• f
• ⁢
• 2
• k
• ⁢
• ∀
• i
• ≡
• j
• where f1 and f2 are dimensionality reduced fingerprints to compare and I and j are terms within the dimensionality reduced fingerprints.