Composite code sparse autoencoders for approximate neighbor search

A method for creating an index for information retrieval, the method being implemented by a processor and memory, the method comprising:

• (a) receiving a set of input data, wherein each element of data in the set is a dense representation;
• (b) generating, using a trained encoder implemented using the processor and including one or more layers, a composite code from the dense representation, the composite code comprising a vector of dimension D decomposed into C clusters of dimension L, where C and L are at least two;
• (c) using an activation function to generate a sparse binary representation from the composite code, the sparse representation comprising C clusters of dimension L;
• (d) repeating steps (a)-(c) for each element of data in the set of input data;
• (e) creating, using the processor, an index based on each of the sparse representations generated at step (c) corresponding to each element of data in the set of input data;
• and
• (f) outputting the index for information retrieval;
• wherein the method further comprises:
• regularizing the sparse binary representation to enforce load-balancing between different dimensions.

An autoencoder for information retrieval implemented by a processor and a memory comprising:

• a neural network encoder implemented using the processor, the neural network encoder being trained to receive a dense representation and generate a composite code from the dense representation, the composite code comprising a vector of dimension D decomposed into C clusters of dimension L, where C and L are at least two, and
• an activation function implemented using the processor, the activation function being configured to generate a sparse composite code from the composite code, the sparse representation being a binary representation comprising C clusters of dimension L;
• wherein the sparse composite code is used to generate, supplement, or search an index for information retrieval;
• further comprising:
• a neural regularizer implemented using the processor and configured to regularize the sparse representation to enforce load-balancing between different dimensions;
• wherein in the composite code of each dense representation data elements are represented by values of each of Z subconcepts or classifications for each of C concepts.

An information retrieval architecture implemented by one or more processors in combination with a memory, the architecture comprising:

• an encoding phase implemented using the one or more processors and having one or more layers, the encoding phase being trained to receive a dense representation corresponding to a query, generate a composite code comprising a vector of dimension D decomposed into C clusters of dimension L from the dense representation and generate a sparse composite code from the composite code using an activation function, the sparse composite code comprising a binary representation comprising C clusters of dimension L, where each cluster sums to one, where C and L are at least two;
• a scoring phase implemented using the one or more processors and configured to search an index based on the sparse composite code and output scores for a plurality of documents based on the index;
• and
• a sorting phase implemented using the one or more processors and configured to rank a subset of the plurality of documents based on the scores and output one or more relevant documents from the ranked subset.

A method for information retrieval implemented by one or more processors and a memory, the method comprising:

• receiving a dense representation of a query;
• generating, using a trained encoder implemented using the one or more processors and including one or more layers, a composite code comprising a vector of dimension D decomposed into C clusters of dimension L from the dense representation, where C and L are at least two;
• generating a sparse composite code from the composite code model using an activation implemented using the one or more processors, the sparse composite code comprising a binary representation comprising C clusters of dimension L, where each cluster sums to one;
• generating scores for a plurality of documents in an index based on the sparse composite code;
• ranking a subset of the plurality of documents based on the scores;
• and
• outputting the ranked subset.

An apparatus for transforming one or more dense representations corresponding to one or more documents into a sparse composite code for information retrieval comprising:

• a non-transitory computer-readable medium having executable instructions stored thereon for causing a processor and a memory to:
• for each of the one or more dense representations:
• receive the dense representation;
• generate, using a trained encoder including one or more layers, a composite code from the dense representation, the composite code comprising a vector of dimension D decomposed into C clusters of dimension L, where C and L are at least two;
• and
• use an activation layer to generate the sparse composite code from the composite code, the sparse composite code comprising a binary representation comprising C clusters of dimension L, where each cluster sums to one;
• and
• outputting the sparse composite codes for the one or more documents for generating, supplementing, or searching an index for information retrieval.